Tag

data-processing

Posts collected under this tag.

field note

Solving Memory Pressure in a Python Data Pipeline

After a vectorized rewrite cut processing time from 30 minutes to 10 seconds, concurrent load under limited server memory exposed a completely different constraint, and the solution required batching, job queues, and polling to solve without giving back the performance gains.

pythonpandasmemory-managementbatch-processingqueuespollingscalabilitydata-processing
story

How I Replaced For Loops with Vectorized Pandas

A hands-on account of replacing loop-based data processing with vectorized pandas operations across two financial calculation engines, and what that architectural shift actually looks like in code.

pythonpandasnumpyvectorizationdata-processingperformance
guide

5 Pandas Operations That Replace Most For Loops

A practical walkthrough of the merge, reduce, pivot, and column math operations that together replace almost every for loop in a production data processing pipeline, with worked examples and the reasoning behind each.

pythonpandasnumpyvectorizationmergepivotdata-processing