Skip to content

Options Data Pipeline — Architecture Reference

status: ACTIVE tier: 2 (design reference) last-verified: 2026-09-27 verified-against: e3ba5fd4 supersedes: Projects/_closed/Options Indicator Improvements/Options Data Concept and Architecture.md (historical source, Tier 3) governing-spec: Projects/_closed/Options Indicator Improvements/Simplify Options Flow Data Pipeline/Unified Implementation Plan v4.1.md + Decision Log (D1-D10)

2026-09-26 cutover (v4.1 — the state described below is TRANSITIONAL)

The storage root moved to ~/Dropbox/Apps/ATAS/Options (gex_lib.OPTIONS_ROOT single source; env TRADERJAN_OPTIONS_ROOT) and a Schema v4 day-file store is live: gex/{SYM}/{date}.json — one file per symbol-session (ascending UTC snapshots[] + session metadata: publish_ts/publish_detected/settled/activation). 2,629 day-files: NQ 1,692 (2020-01→), ES 937 (2023-01→), the other 5 instruments' live era backfilling. Raw layer live: raw/{SYM}/{date}/{HHMMSS}.json.gz byte-exact per poll (daemon + C#).

Binding (C# v2.10.0, single-source): ALL render/signal paths bind DayFileOptionsStore.PayloadAt (UTC-first, 4-rung ladder: carry-over → settled → morning baseline → prior-base; no day-file = honest blank). D10: static levels switch ONCE daily at the detected OI publication (publish_ts; the Morning Activation Floor = thin-day fallback only) — the session-roll switch and its ~12h lookahead retired. The legacy tiers (_ETH/_RTH/_HHMM/phase files) are write-once P-IT compat + fallback internals pending Phase 4 retirement; the readers' Tier-2 description below remains accurate for the LEGACY path only.

Purpose

How options-derived market structure (GEX levels, walls, profiles) flows from raw vendor data to the ATAS chart and the Python backtester — one canonical contract, three consumers.

CBOE delayed CDN (QQQ/SPY chains) ──┐
ThetaData EOD + 15-min parquets   ──┼──► gex_lib.py (canonical math, schema v3)
Yahoo futures closes + VIX/VXN    ──┘         │
                                              ├──► ~/Dropbox/Apps/ATAS/Options/
                                              │      gex/{NQ,ES}/{date}.json        (v4 day-files)
                                              │      nq_qqq_ratios.json / es_spy_ratios.json
                                              │
              C# GexEngine.cs (parity port) ◄─┤──► Options Flow indicator (chart/HUD)
                                              └──► manage_data.py ──► Master Parquets ──► backtest.py

Data sources

Source Content Cadence Notes
CBOE delayed CDN QQQ/SPY option chains (OI, IV, volume) ~15 min delayed Live path for collector daemon + C# in-indicator fetch
ThetaData EOD quotes+OI, 15-min intraday quotes Cached (thetadata_cache/) QQQ/SPY full history (2020–2026); IV via bisection inversion (no greeks subscription)
Yahoo Finance NQ=F/ES=F daily closes, ^VXN/^VIX On demand Ratio files + IV rank; fallback constants forbidden for archives
AlphaVantage (RETIRED) Historical EOD chains Retired 2026-09-27 Superseded by ThetaData full-history (2020–2026) EOD + intraday

Schema v3 contract (OptionsPayload)

Defined in TraderJan.Indicators/GexEngine.cs, mirrored by Backtesting/scripts/gex_lib.py. Key fields: schema_version: 3, spot_price (futures scale), etf_spot, ratio, key_levels (10 fields), term_structure (dte0/weekly/monthly tiers), regime, strike_profile (band-limited rows), source, timestamp, trade_date, chain_date, phase.

Archive tiers per session date in snapshots/{SYM}/:

File Semantics (point-in-time)
full_options_{sym}_{date}.json Base: day-T EOD chain (oracle)
…_{date}_ETH.json Overnight slot: T-1 chain
…_{date}_RTH.json RTH slot: T-1 chain (PIT); live-era = collector live chain pre-Phase 2, repaired after
…_{date}_RTH_settled.json Day-T settled EOD (ThetaData) — the oracle
…_{date}_{HHMM}.json 15-min intraday snapshot (HHMM = ET); EOD-frozen OI, IV inverted from quotes

PIT hard rule: chain_date < trade_date for _RTH/_ETH; enforced by audit_options_archives.py (0 violations = gate).

Session calendar

  • Options day rolls at 05:00 CT (OCC morning publication); RTH window 08:30–16:15 CT.
  • Weekend guard: no dated snapshots written Sat/Sun (collector + C# WriteArchive).
  • Live-era boundary: 2026-08-25 (collector v3 start); generator protects v3 collector files there.

Generators & consumers

Tool Role
Backtesting/scripts/options_collector.py 24/7 daemon (launchd, 900 s): live chains → active file + snapshots
Backtesting/scripts/generate_intraday_json.py Historical _HHMM backfill (band-limited ±4 %, D3)
Backtesting/scripts/build_gex_from_thetadata.py _RTH_settled regeneration (OI pre-check)
TraderJan.Indicators/OptionsFlow/download_all_historical.py AV offline rebuild + PIT _ETH/_RTH
Backtesting/engine/manage_data.py Master Parquet GEX columns (delegates to gex_lib since Phase 5)
Backtesting/scripts/audit_options_archives.py Quality gates: vt bounds, scale, PIT, provenance, weekend
Backtesting/scripts/verify_intraday.py / verify_engine_a_neutrality.py Parquet PIT + Engine A non-GEX byte-identity

Known limitations (documented, accepted)

  • Intraday snapshots freeze OI at day-T EOD (ThetaData carries none intraday) — historical DynamicRealTime replay is not strictly PIT for OI.
  • ES intraday history starts 2023-01 (SPY cache scope); NQ covers 2020-01-02 → present.
  • CBOE feed is ~15-min delayed; live levels lag fast markets by design.