Market Price Pipeline

What it is

A headless data pipeline with no public UI, and that’s the point. It continuously scrapes Buff163, the largest CS2 skin secondary market, normalizes what it finds, and lands it in PostgreSQL. The other lab builds read their pricing from there.

How it works

Playwright drives a pool of browser contexts against the marketplace endpoints, because the data is rendered client-side and rate-limited per fingerprint. Requests are scheduled around the throttle windows I’ve observed. Results are diffed against the last known state and only the movements get written, which keeps the write volume and the downstream signal clean.

Storage is a partitioned Postgres schema: one table for item identity, append-only tables for price points, and Redis as the “current value” hot cache the platforms read on every request. When an item’s price moves, subscribers hear about it through the same Socket.io layer IO Saga runs on.

Why it matters

Every instant buy and sell quote on my platforms traces back to this pipeline. It’s the most load-bearing piece of infrastructure in the lab. Everything else is presentation, and this is the source of truth. If it fails, prices go stale, and stale prices mean real money lost to arbitrage, so the monitoring here is louder than for anything else I run.