Market Price Pipeline

What it is

A headless data pipeline with no public UI — and that's the point. It continuously scrapes Buff163 (the largest CS2 skin secondary market), normalizes the data, and lands it in PostgreSQL where the other lab builds consume it as pricing truth.

How it works

Playwright drives a pool of browser contexts against the marketplace endpoints, because the data is rendered client-side and rate-limited per fingerprint. Requests are scheduled around observed throttle windows, results are diffed against the last known state, and only movements get written — keeping the write volume and the downstream signal clean.

Storage is a partitioned Postgres schema: one table for item identity, append-only tables for price points, and Redis for the "current value" hot cache the platforms read on every request. If an item's price moves, subscribers find out through the same Socket.io layer IO Saga runs on.

Why it matters

Every instant buy/sell quote on my platforms traces back to this pipeline. It's the single most load-bearing piece of infrastructure in the lab — everything else is presentation; this is the source of truth. Failure here means stale prices, and stale prices mean real money lost on arbitrage, so the monitoring is louder than for anything else I run.