The problem
Keepa's API returns a limited window of results per query, so a single ingestion strategy saw only about 15% of each catalogue — and row-by-row inserts made every run slow.
What I built
- 01Built a pagination engine with six configurable sort strategies and a global de-duplication set, lifting coverage to 75%+ per run.
- 02Replaced row-by-row inserts with batched INSERT … UNION ALL statements for a 10–20x faster load.
- 03Modelled semi-structured API payloads as canonical entities and attributes, with token-based rate limiting.
- 04Made every procedure idempotent so runs can be repeated safely, and versioned the Snowflake schema in git.

