insideout Get Pro

10 · Making it fast

Pro

Python for Market Data, lesson 10

Making it fast

Vectorise first, then measure where the time goes.

You can write a research pipeline in pandas: returns, windows, ranks, joins and a backtest.

Part 1. Where does the time go?

A research script over ten years of prices for 500 simulated stocks, timed step by step on one laptop. You can make one step ten times faster. Pick it.

Fix the right stepLive

Making a small step faster saves almost nothing, however clever the fix. The loop was most of the run, so replacing it with one pandas call did more than everything else together. Then the biggest step was something else, writing the file, and the only way to know was to measure again.

Profiling and Amdahl's law

Profiling is measuring where a program spends its time. Amdahl's law puts a number on what you saw: if a step takes a share s of the run, making it infinitely fast saves at most s. A step that's 5% of the run can save you 5%, never more.

Your turn

A step takes 20% of a run. You make it 10 times faster. How much faster is the whole run?

The step goes from 20 to 2, and the other 80 stay: 82 in total, or 100 / 82 ≈ 1.22 times faster. The 80% you didn't touch sets the limit.

Part 2. Measure first

Keep going with Pro

You've done the first part. Pro opens the rest of this lesson, and every other one.

Still to come
  1. Measure first
  2. When a loop can't vanish
  3. Worked: memory and copies
  4. The same thing, for real
See Pro plansI have Pro: sign in
The last lesson in Python for Market DataBack to the overviewThe tools every quant uses every day Back to Python for Market Data

Python for Market Data overviewSpotted a mistake? Tell us.