Profiling Python Code

(Updated 10/04/2022 with py-spy)

Python is an interpreted language, which typically makes it slower than compiled languages like C/C++, Java, Rust, or Go. In order to optimize Python code for speed, it’s best to know what parts to optimize. That’s where cProfile and other profiling tools come in.

Serial Code

Most of my code is serial (no multithreading, multiprocessing or other distributed environments). For these situations, cProfile is great. cProfile is a module profiling Python code running in a CPython environment. It can be run as a command-line module, or used in your source code to pinpoint a specific function.

Run your program as you normally would, but wrap it in cProfile:

python -m cProfile -o profiling/whatever-v1.prof \
  -m src.module.whatever arg1 arg2

Additionally, cProfile can be used from inside your source code like so:

cProfile.run(
    "eval(arg1, arg2)", 
    sort='tottime', 
    filename='eval.prof',
)

The only downside of this is that the first argument is a string that is evaluated by cProfile instead of passing a function and some arguments. Other than that, it works in exactly the same manner.

SnakeViz is a small open-source tool that is very valuable for visualizing profiling data. It presents a graph that shows a breakdown by time per function, and allows you to drill down into what causes the length of each function:

From here, you can see what functions take the most time, and what causes each function to take so long. If you can further optimize from here (removing things from loops, using list comprehensions, etc., do so now).

Parallel Code

If you profile your serial code, you might eventually add multiprocessing. cProfile sucks for multiprocessing code.

To profile parallel code, I recommend py-spy. It requires sudo on macOS and some Linux systems, however.

py-spy record -o profiling/whatever-v1.svg -- python -m src.module.whatever arg1 arg2

py-spy generates svg files which can typically be opened in a browser like FireFox.

Please email me if you have any comments or want to discuss further.

[Relevant link] [Source]

Sam Stevens, 2024