Profiling Python Code
(Updated 10/04/2022 with
Python is an interpreted language, which typically makes it slower
than compiled languages like C/C++, Java, Rust, or Go. In order to optimize Python
code for speed, it’s best to know what parts to optimize. That’s
cProfile and other profiling tools come in.
Most of my code is serial (no multithreading, multiprocessing or
other distributed environments). For these situations, cProfile is
cProfile is a module profiling Python code running
in a CPython environment. It can be run as a command-line module, or
used in your source code to pinpoint a specific function.
Run your program as you normally would, but wrap it in
python -m cProfile -o profiling/whatever-v1.prof \ -m src.module.whatever arg1 arg2
cProfile can be used from inside your
source code like so:
cProfile.run("eval(arg1, arg2)", ='tottime', sort='eval.prof', filename)
The only downside of this is that the first argument is a string that
is evaluated by
cProfile instead of passing a function and
some arguments. Other than that, it works in exactly the same
SnakeViz is a small open-source tool that is very valuable for visualizing profiling data. It presents a graph that shows a breakdown by time per function, and allows you to drill down into what causes the length of each function:
From here, you can see what functions take the most time, and what causes each function to take so long. If you can further optimize from here (removing things from loops, using list comprehensions, etc., do so now).
If you profile your serial code, you might eventually add
cProfile sucks for multiprocessing code.
For example, here’s the SnakeViz visualization of a long-running python
program using multiprocessing:
To profile parallel code, I recommend py-spy. It requires sudo on macOS and some Linux systems, however.
py-spy record -o profiling/whatever-v1.svg -- python -m src.module.whatever arg1 arg2
py-spy generates svg files which can typically be opened in an
Please email me if you have any comments or want to discuss further.
[Relevant link] [Source]
Sam Stevens, 2022