Skip to content
Muhammad Safyan

E-commerce intelligence platform

From 15% to 75% catalogue coverage on Snowflake

An idempotent Snowflake and Snowpark ETL enriching 50k–100k+ products per run from Keepa and ScaleSERP — 5x the catalogue coverage and 10–20x faster loads.

catalogue coverage, up from 15%
75%+catalogue coverage, up from 15%
faster database loads
10–20xfaster database loads
products per run
100k+products per run

The problem

Keepa's API returns a limited window of results per query, so a single ingestion strategy saw only about 15% of each catalogue — and row-by-row inserts made every run slow.

What I built

  1. 01Built a pagination engine with six configurable sort strategies and a global de-duplication set, lifting coverage to 75%+ per run.
  2. 02Replaced row-by-row inserts with batched INSERT … UNION ALL statements for a 10–20x faster load.
  3. 03Modelled semi-structured API payloads as canonical entities and attributes, with token-based rate limiting.
  4. 04Made every procedure idempotent so runs can be repeated safely, and versioned the Snowflake schema in git.

Want results like these on your project?

Tell me what you are building and where it hurts, and I will come back with questions or a plan.