Retail · Data engineering
Automating a 7-billion-row partitioning job
Oracle · Aurora PostgreSQL · Python · SQL
A global retailer migrating from Oracle to Aurora PostgreSQL needed to partition eight tables across four environments. The largest held 7 billion rows and occupied more than 6 TB.
I took ownership of all eight tables, wrote a parameterized Python tool to generate the SQL, and executed partition loads and index creation in parallel. For the largest table, I broke the generated SQL into batches so I could parallelize inserts and then index creation. I watched database CPU usage to set a safe concurrency limit.
Result The automated process completed in about 12 hours per environment, down from an estimated two weeks of work per environment.