Profiling Python Code with
Python is an interpreted language, which typically makes it slower than compiled languages like C/C++, Java, Rust, or Go. In order to optimize Python code for speed, it’s best to know what parts to optimize. That’s where
cProfile comes in.
cProfile is a module profiling Python code running in a CPython environment. It can be run as a command-line module, or used in your source code to pinpoint a specific function.
From the Command Line
Run your program as you normally would, but wrap it in
- Outputs all function calls sorted by internal time.
python -m cProfile -s tottime writing.py
- Outputs all function calls sorted by internal time and sends it to
writing.txtfor later inspection.
python -m cProfile -s tottime writing.py > writing.txt
- Outputs all function calls to
writing.profto be used by another tool later on.
python -m cProfile -o writing.prof writing.py
As a Module
cProfile can be used from inside your source code like so:
"eval(arg1, arg2)", sort='tottime', filename='eval.prof')cProfile.run(
The only downside of this is that the first argument is a string that is evaluated by
cProfile instead of passing a function and some arguments. Other than that, it works in exactly the same manner.
Regardless of how you use it, the output from these
cProfile tools is loooong. It can be read, but it’s often easier to use it in another tool. Python has support for this with the
Visualizing Profiling Data with SnakeViz
SnakeViz is a small open-source tool that was recently very valuable to visualizing profiling data. It presents a graph that shows a breakdown by time per function, and allows you to drill down into what causes the length of each function:
From here, you can see what functions take the most time, and what causes each function to take so long. If you can further optimize from here (removing things from loops, using list comprehensions, etc., do so now).
Please email me if you have any comments or want to discuss further.
[Relevant link] [Source]
Sam Stevens, 2020