Retail competitor price monitoring
An industrial supplies distributor approached me because, commercially, they had no reliable way to monitor competitor prices across their product portfolio - the targets are Akamai-protected, SAP-level ecommerce installations, some behind logins. This is a familiar shape of problem for anyone in distribution: decades of gut feel for the market, thousands of products, and the checking apparatus is a browser tab and a spare afternoon. Pricing calls were being made half-blind. Nobody could answer "where are they cheaper than us?", or "what do they stock in our categories that we don't?", and by the time anyone had dug the answer out of five separate retailer websites, the prices had moved anyway.
So we built them a price monitor: a private web app on Cloudflare, with a chat assistant and a spreadsheet export, fed by an overnight crawl and an AI matching engine that ran on our own hardware with a DeepSeek V4 eval. Here's how it works.
What their week looked like
Ad-hoc searches, one product at a time. A strong team with a genuinely good feel for the market and no systematic way to check it - which meant the instinct did all the work and the data did none. The range-gap question (the products competitors stock that you don't) simply never got answered, because answering it meant comparing thousands of listings by hand. Nobody's afternoon survives that more than once a quarter, and quarterly is no cadence for pricing.
How I built it
The build is a crawler, a database and a matching engine with an Astro UI.
Firecrawl collects the retailer listings overnight - the retailers are all bot-protected, and after trying the polite routes, Firecrawl's stealth crawling was the one approach that worked reliably across all five. Every page lands in a Cloudflare D1 database as it's collected, not held in memory and written at the end. That decision sounds like plumbing, but it's the difference between a crash costing one page and a crash costing the whole night's run. I learned that one the hard way, on this very build (more on that when the full write-up's done).
Then the matching: is this competitor's "100mm grey rubber swivel castor" the same product as yours, or just a similar one? There's no barcode to lean on - none of the five retailers expose one - so it's a judgement, and judgement at 12,000-candidate scale needs to be cheap. Two AI models running on our own hardware grade every candidate pair at no per-call cost, and a paid model is only brought in on the pairs where the two free ones disagree - which turned out to be 12% of them. Every kept match carries a short written reason, so the commercial team can see why the system paired two products and overrule it where it's wrong.
The same local models do a job nobody asked for but everyone needed: category normalisation. Five retailers' worth of messy category strings - 5,600-plus of them - mapped down to one clean set, so "castors" means the same thing whichever site the listing came from.
Where it stands
~41,800 listings tracked. 97% of the catalogue matched to at least one competitor. Refreshed by the overnight crawl instead of anybody's afternoon.
Firecrawl crawl → Cloudflare D1 + R2 → local-AI product matching and category normalisation → web app, CSV export and chat assistant.