The problem
Racing data is spread across the national results body, a form-data provider, broadcaster sectional feeds, state stewards' offices, exchange and bookmaker prices and the sale companies. Each has its own identifiers, spellings and gaps. Anyone who wants to ask a real question across all of it, such as how a sire's progeny run over distance given their stride profile, has to assemble the data themselves. This platform is that assembly, built for our own equine data venture and used by client systems through its API.
Sources and scale
- Results, form, horse pages, trials, forward acceptances and weights, owners and racing colours from the national body.
- 2.2 million form-provider runs from 2018, plus a post-race sectional collector backfilling about 130,000 races at a deliberately gentle rate.
- Stewards' reports from six states since 2016: 833,000 notes, 577,000 linked to individual runs.
- Broadcaster sectional and stride telemetry: 483,000 runs and 3.2 million race sections, with stride data for three states from mid-2021.
- Market prices from 2021: 1.17 million rows of opening, starting and exchange prices from official files and a live poller.
- Yearling and online sale records from the major sale companies.
- An append-only daily log of model outputs, 1.17 million rows, so no prediction can be rewritten after the result.
The platform holds roughly 489,000 races, 2.2 million results and 124,000 reconciled horses after a large identity clean-up. Coverage is dense from 2010 and thinner before, which the documentation says plainly.
Architecture
There is no always-on worker. Seventeen scheduled jobs on Render handle the daily form refresh, results, debutants, backfill, maintenance and alarms, reconcile and heal, enrichment (stewards, sectionals, broadcaster data), daily prices, exchange settlements, materialised-view refresh, weekly ratings, weekly dedupe, a weekly backup to Google Cloud Storage and the hourly sectional collector. Each job writes a heartbeat.
The web front end is FastAPI with Jinja templates behind a login, backed by materialised views so that horse, race, sire and trainer pages with years of history load in under a second. A read-only JSON API at /api serves search, horses, analytics, breeding, tracks, trainers, sales and per-run sections, and is consumed by a native SwiftUI iOS app with search, planner, breeding and results tabs.
Front-end features
- Horse pagesForm lines with gear, a stewards' report button and a speed-telemetry button, colours and sale history.
- Stride tabFor 28,000+ horses, each form line has a 200-metre speed, stride and cadence strip drawn from 1.4 million sectional rows, de-meaned by source and month to correct for feed drift.
- Race pagesIn-run position charts and a day index.
- Sires and breedingSire leaderboards, nicks, broodmare sires and stride "signatures" for 947 sires.
- System BuilderAsk a racing question in plain English. The model writes read-only SQL, returns the matching runners and backtests them. Tool chaining for follow-ups, a daily spend cap and a 106-question test harness.
- Data-quality monitorNineteen invariants recorded daily, freshness and heartbeat alarms, a nightly reconcile-and-heal job and a weekly dedupe census.
Lessons
Identity is the hard part of any multi-source platform. A horse with the same name in two countries, or two spellings of one horse, will quietly double your counts unless matching is exact-first with a fuzzy fallback and a dedupe routine that reports what it merged. Feeds drift: the same telemetry source can shift by half a standard deviation between seasons, so any trend has to be normalised by source and period before it is shown. And a monitor that counts cumulatively will stay red forever; alarms must be growth-based. All of this is built in, documented and reused on every data platform we deliver.