Retail price and product data collection, with n8n
Houtini has a network of websites for testing and deployment - it's like having a publishing network on the side, but it's important to have this infrastructure so we can develop and test ideas for our clients before they ever go near client work. Anyway, this pipeline is an n8n system, on a self-hosted server, that scrapes, enriches and serves product data via API for affiliate publishers - simracingcockpit.gg, our sim-racing property, is the flagship consumer. An editor there drops a shortcode into an article and a product grid appears: current prices, ratings, stock, drawn from a catalogue of roughly 9,900 products collected automatically from more than twenty supplier sources.
All of it is meant to stay current on its own. And for a while, it quietly didn't.
When the automation rotted
That's the thing about broken automation: it fails silently. This system had been running for a good while before we discovered that one supplier network had blocked the server's IP address, so its scrapes were returning nothing at all. A separate enrichment pass had a bug that wiped product categories as it ran. And the nightly categoriser - the workflow that files each product into its category - had degraded to processing a single product per run. No alarms anywhere. Just stale and empty listings that the team caught by chance, weeks later. If you run any automated collection, some version of this is probably happening to you right now, which is a cheerier thought for me than it is for you.
How I rebuilt it
The rebuild had two jobs: make the pipeline trustworthy, and make its failures loud.
The collection layer is a set of scheduled n8n workflows on our own server - one master workflow per platform type, so a new supplier is a configuration, not a new scraper. Every supplier catalogue re-scrapes daily, and a full 3,400-product supplier network scrapes and stores in about six and a half minutes. From there, every product runs through an AI categoriser into one of 54 canonical categories (the categoriser runs on a local model, so the volume costs nothing per call), and then a multi-stage enrichment pass extracts structured specifications from the listing - torque, dimensions, encoder resolution, materials, dozens of fields - so the catalogue is filterable by real attributes rather than by whatever string a supplier happened to type. Everything lands in PostgreSQL and serves through a REST API that the content sites read.
And when something breaks now, it says so. The orchestrator's board reads 4,364 production runs in the last seven days, zero failed. I check that number most mornings - partly for reassurance, partly because I still don't quite believe it.
Where the data goes
This is the part that makes the collection worth doing. An editor writing a buyer's guide drops one shortcode into the article and gets a live product grid - prices, ratings, stock, sorted however the page needs. No copying prices into prose that go stale the week after publishing; the grid reads the API, and the API read the suppliers this morning.
Behind each row sits the augmented record: the supplier's raw listing enriched into dozens of structured fields - build intent, materials, dimensions, warranty, compatibility. And the same structured data goes out the other door as properly marked-up JSON for search engines and AI agents, because the next consumer of a product catalogue isn't only a human reader on the page. It's an agent asking "what cockpits under $1,000 are in stock?" - and this catalogue can answer.
Where it stands
Live, running daily, no human touch on the routine path. The failure modes that used to hide for weeks now surface the morning they happen.
Scheduled n8n workflows, self-hosted → PostgreSQL → local-AI auto-categorisation + multi-stage spec enrichment → REST API for publishers + structured JSON for search engines and agents.