Lab · interactive model

Pipeline race: one row at a time against whole columns

Two pipelines run the same six steps on the same table. The front track carries each record on its own and writes every step's result to staging. The back track reads only the columns it needs, moves them as tall batches across several cores, and never stops between steps. Press Start, then orbit the scene.

Drag to orbit · scroll to zoom · right-drag to pan
Columnar advantages (click to switch off)
0row track: rows through step 1
0 srow track: simulated time
0columnar track: rows processed
0 scolumnar track: simulated time
×1finishing ratio in this model

What it shows. A row-at-a-time ETL job pays a fixed overhead for every record at every step, touches every column whether the query needs it or not, and writes each step's result to a staging table before the next step reads it back. A Polars-style columnar engine plans the whole query first. It reads only the needed columns and applies the filter while reading, so the unneeded columns drop away before any work starts. It then runs the steps as one fused pass over large column batches, spread across the CPU cores.

What to try. Switch off one advantage at a time and restart. Without pruning, wide batches carry every column. Without parallel cores, one lane does all the work. Without vectorised batches, the tall slabs break into single values and the per-row overhead comes back. With all three off, the only saving left is the missing staging. Then raise the column count while the query still needs six columns: the row track slows down and the columnar track does not.

Configuration at these settingsSimulated timeAgainst the row job

The clock in the corner shows simulated time. It runs slowly while the columnar track works so you can watch it, then speeds up until the row track finishes. The playback speed is shown next to it.

This is an illustrative model. The speeds come from a simplified formula with made-up constants, written out in one config block in the page's model script. They are not benchmarks and not measurements of any real system or project.