Skip to content

Trade-Level Portfolio Ledger & Mixer — Architecture Reference

status: DRAFT (Round-2 review incorporated 2026-09-28) tier: 2 (design reference) last-verified: 2026-09-28 verified-against: 95a97b1b initiative: Projects/Trade-Level Portfolio Mixer/Trade-Level Portfolio Mixer - Revised Master Plan (2026-09-25).md supersedes: none

Purpose

This document defines the proposed data and execution contracts for the Trade-Level Portfolio Mixer. It is a design reference, not an implementation description. All decisions marked PROPOSED require Gate 0 approval before code is written.

The contract exists to keep the following three layers honest:

  1. the per-record trade ledger;
  2. the portfolio mixer/optimizer;
  3. the dashboard and targeted-evolution consumers.

The governing plan is Projects/Trade-Level Portfolio Mixer/Trade-Level Portfolio Mixer - Revised Master Plan (2026-09-25).md.

Scope boundary

The first release consumes replayed bar-resolution trades from the Python backtester. It does not claim exchange/tick execution, live order routing, or exact intra-bar fill timing. A later tick-data initiative may add higher-resolution fields without changing the immutable identity model.

1. Immutable record identity

1.1 Aliases versus records

genome_uid is an alias. It is retained for compatibility with existing champion JSONs, dashboards, family keys, and vault references.

record_id is the canonical key for one replay (replay identity). Proposed inputs to the identity hash are:

  • normalized genome UID;
  • chart/dataset identity — the exact parquet consumed, including research variants (*_DI_EVOLVE, *_DI_*);
  • engine (PA or OPT);
  • canonical CLI hash after neutral-parameter normalization — computed on a deep copy of the parsed params (never on the live replay dict; canonicalize_genome mutates in place — the documented v3.20.0 gotcha), with the _INTERNAL_CLI_KEYS denylist and the launcher token stripped;
  • backtest engine version;
  • dataset snapshot fingerprint;
  • effective data window (resolved date_from/date_to/last_sessions — identity-relevant even though the reporting denylist strips those keys from replay CLIs);
  • data-preparation parameter set (frozen prep config, e.g. atr_period, source_tz).

Schema-version exclusion (D-11): the ledger schema version and the generation id are not record_id inputs. They belong to the ledger-row identity (record_id × ledger schema version × generation id). Bumping the ledger schema must never re-key existing records.

A single UID may resolve to multiple record_id values. This is a first-class result, not an error to hide.

Identity is derived from CLI/JSON provenance, never from lossy CSV headers — the P4 case (header-only replay gives 5,301 vs 6,106 trades) is a mandatory regression fixture.

1.2 Collision rules

  • Never use genome_uid as a parquet overwrite key.
  • Never resolve duplicate records by highest Net Ticks alone.
  • Retain conflicting records and classify them in the catalog: generation-history duplicate, CLI collision, engine/version conflict, dataset conflict, DIVERGENT_ENGINE_CONTRACT (pre-v3.20.0 Engine-B rows — replaying under today's neutral-default engine yields different trades than their stored DB Net Ticks; the 11 EPOCH-B champion JSONs already carry the _entry_semantics_divergence annotation as precedent), or unresolved.
  • The catalog must expose the alias→record mapping and the reason for every conflict.
  • A portfolio candidate references record_id; genome_uid alone is insufficient.

1.3 Population classification

Every build manifest must list the population rule and census:

  • included records;
  • excluded records and reason;
  • charts covered;
  • engines covered;
  • duplicate/collision groups;
  • date and dataset windows;
  • per-record and aggregate trade counts.

The manifest — not a historical snapshot figure — is the authoritative census for a published generation. Reference census at Round-2 review (2026-09-28 ~20:30, live DB): 7,442 rows / 5,851 distinct UIDs with valid CLI / 805 champion-class UIDs; the "410 champions / 5,762 pool" figures of the Sept-20 snapshot are stale, the pool churns under continuous evolution, and by the same evening the parallel Genome Evaluation DB rebuild had already moved the census to 10,620 / 9,966 / 154 — tier sizes are measured at build time (F-15), never hard-coded.

Champion taxonomy pinning (D-03): a generation must name the champion definition it used. Three coexist: 410-institutional (Sept-20 artifact), live-DB institutional (Net Ticks > 30000 & Max Drawdown < 0), and multi-horizon status rows (Anchor_Net/Lifetime_Net/Net > 0). The proposed Generation-1 definition is live-DB institutional + family reps (family_reps.csv).

Anchor-coverage sparsity: only 163 champion rows carry anchor_net > 0 (pre-v3.24.1 runs lack the column). Facet-index filters must not assume anchor coverage; coverage flags are mandatory.

2. Ledger schema (proposed)

Field Requirement Semantics
record_id required immutable record key
genome_uid required compatibility alias
Chart required M1, M5, D200, D500, T900, V300, or explicitly classified extension
Engine required PA or OPT
Trade_ID required zero-based ordinal within the record
Signal_Bar required bar index where the setup was detected
Fill_Bar required bar index where the limit/entry was filled
Exit_Bar required bar index where the trade closed
Fill_Window_Bars required order fill window (CLI param; engine default 1) — needed to derive unfilled pending occupancy; the Python engine never exports expired orders
Signal_Time required signal-bar stamp
Fill_Bar_Time required fill-bar stamp
Exit_Bar_Time required exit-bar stamp
Timestamp_Precision required proposed initial value BAR
Direction required Long or Short
Volume required contracts
Entry_Price / Exit_Price required executed price at declared precision
Gross_Ticks required price PnL before friction, with declared rounding
Slippage_Ticks required explicit slippage component
Commission_Ticks required engine-canonical commission component, or explicit NA reason
Net_Ticks required canonical engine net value; mixer must not reapply friction
Tick_Size / Tick_Value required instrument contract specification values
Contract_Spec_Version required versioned instrument metadata
Net_PnL_USD required conversion from the declared contract spec
Exit_Reason required canonical engine reason
SessionDay required signal-bar CME session day
Weekday required signal-bar weekday
Intraday_Window required signal-bar session bucket
Duration_Bars derived exit minus fill bar under the declared convention
Duration_Seconds derived/optional bar-stamp difference only when precision is declared
MAE_BarRange_Ticks required (D-10, BarRange-first) worst intra-bar high/low excursion from entry; no same-bar ordering ambiguity (no intra-bar sequencing claimed); engine-computed via the existing unr_best/unr_worst block (engine/simulate.py:556-563)
MFE_BarRange_Ticks required (D-10, BarRange-first) best intra-bar high/low excursion from entry
MAE_Close_Ticks proposed, later worst bar-close mark-to-market excursion from entry
MFE_Close_Ticks proposed, later best bar-close mark-to-market excursion from entry
Excursion_Method required BAR_RANGE / CLOSED_BAR / TICK; documents the method per field
Source_* required CLI, engine version, DB/run id, dataset fingerprint, schema version

2.1 Precision rule

The first release declares Timestamp_Precision = BAR. The current backtester trade log provides signal-bar, fill-bar, and close-bar stamps; it does not prove an exact second within a bar. The ledger must never turn a bar stamp into a claim of exact execution.

A Duration_Seconds value derived from bar stamps is allowed only when labelled as bar-resolution. Exact-second or tick-resolution fields require a separate data contract and provenance.

2.2 Session rule

Attribution follows the existing signal-bar convention:

  • CME session roll at 17:00 CT;
  • SessionDay, Weekday, and Intraday_Window are derived from the signal bar;
  • the fill or exit bar never moves a trade into a different facet;
  • an overnight position remains occupied after its assigned slot ends.

2.3 Excursion rule

MAE_BarRange_Ticks and MFE_BarRange_Ticks are the initial excursion fields (D-10, BarRange-first): excursion-from-entry measured on intra-bar highs/lows. Because no intra-bar sequencing is claimed, this variant carries no same-bar ordering ambiguity — and it is the variant the engine can populate with a minimal additive diff (the daily-limit check already computes the intra-bar extremes per active bar). Bar-close variants (MAE_Close_Ticks / MFE_Close_Ticks) are added later.

Semantic guard: the existing per-trade tl_max_dd is a peak-to-trough unrealized drawdown, not an excursion from entry — it must never be relabelled as MAE. Each excursion field carries Excursion_Method; an intra-bar sequencing claim (beyond H/L extremes) requires a separate method tag and provenance.

2.4 Accounting rule

Friction is stored as components and the canonical engine net value is preserved. The mixer consumes Net_Ticks; it never subtracts slippage or commission again. A tick-like commission value must never be labelled USD without an explicit unit conversion.

Multi-instrument support requires a versioned contract-spec table. A hard-coded MNQ tick value is not a universal contract.

3. Facet index contract

The facet index is a derived, disposable artifact. The trade ledgers and manifest are the source of truth.

One row per record_id may contain:

  • global net, drawdown, trade count, win rate, and active-session count;
  • weekday net/drawdown/trade/active-day metrics;
  • intraday window metrics;
  • joint weekday×window metrics;
  • stored Opp_v21, Risk_v21, Score_v21, Anchor_Net, and Lifetime_Net provenance;
  • minimum-activity flags and confidence/coverage flags;
  • identity collision classification.

Metrics must preserve active-session denominators. Zero-filled non-trading days must not be counted as losses or as active exposure. The index must record its source generation and schema version.

Search latency is a performance target, not a correctness substitute. The index may be rebuilt from the ledgers at any time.

4. Generation and publication contract

4.1 Build stages

  1. Snapshot the source DB and Master datasets.
  2. Fingerprint sources, code revision, engine version, and schema.
  3. Build ledgers, index, catalog, and manifest in staging.
  4. Run parity, identity, coverage, and storage checks.
  5. Publish the generation atomically through a manifest/pointer switch.
  6. Retain the previous generation for rollback.

4.2 Manifest requirements

The manifest must contain:

  • generation id and creation time;
  • source DB/dataset fingerprints;
  • code revision and engine version;
  • ledger schema version;
  • population census and inclusion/exclusion rules;
  • record and trade counts;
  • per-file checksums and index checksum;
  • collision/conflict summary;
  • parity results — at record granularity: exact integer match on net ticks (mandatory); the daily PnL allocation sums to the record total exactly (same association order or integer daily ticks). A ≤0.5-tick print-only tolerance (the daily-PnL extractor pattern) is not bit-perfect and is not accepted; the manifest records the per-record diff distribution;
  • storage measurements;
  • active consumers, parent-generation pointer (lineage), and rollback target;
  • execution-environment block (python/numba/numpy versions, njit-vs-fallback mode) — platform codegen has proven material (SIGSEGV saga);
  • physical fingerprint per dataset — row count + first/last bar stamp in addition to (path, mtime, size); mtime alone is Dropbox-volatile;
  • pinned champion-taxonomy definition used for the generation (D-03).

4.3 Failure rule

A failed parity, integrity, identity, or storage check rejects the generation. Published artifacts are never updated in place, and a warning-only path is not a valid release mode.

5. Portfolio occupancy and account model

5.1 Interval semantics

Bar indices are the sole authority for interval math (D-13) — monotonic and DST-immune by construction; timestamps are display-only. SessionDay/Weekday/Intraday_Window remain signal-bar CT attribution and never move a trade between facets.

A trade occupies a half-open interval:

[Fill_Bar, Exit_Bar)

with one engine-verified exception: a same-bar fill→exit round-trip (explicit same-bar exit block in the simulation core) occupies a minimum of 1 bar — never an empty interval.

Pending orders are a separate occupancy state, with two sub-cases derived from Fill_Window_Bars:

  • filled orders tile cleanly: pending [Signal_Bar, Fill_Bar) + position [Fill_Bar, Exit_Bar);
  • unfilled orders are never exported by the Python engine, so their occupancy is derived: [Signal_Bar, Signal_Bar + Fill_Window_Bars + 1).

A portfolio must not claim zero collision while an unexpired order can fill over another position. Ambiguous same-timestamp overlap between different strategies on one account is treated conservatively as a collision (D-05). A mandatory roll-edge fixture covers a signal at 16:59 CT with fill at 17:01 CT: bar-index occupancy persists across the CME roll.

5.2 Single-account relay

Proposed default rules:

  • at most one open position;
  • no simultaneous long/short exposure;
  • a lower-priority signal is rejected when admission would exceed the contract limit;
  • no automatic flattening is implied by priority gating;
  • a position that crosses a session, window, or weekday boundary remains occupied.

Every admitted or rejected trade records the rule and the competing assignment.

5.3 Fleet mode

Fleet mode must define:

  • account assignment;
  • netting versus isolated positions;
  • contract quantity per account;
  • partial-fill behaviour;
  • peak concurrent margin;
  • cross-account symbol exposure;
  • capital reservation and release timing.

A display estimate such as N × $800 is not an account model and must not be presented as a certification.

6. Optimizer contract

The optimizer selects portfolios, not merely genomes. It must therefore state:

  • candidate universe and record-id pinning;
  • objective formula and horizon;
  • capital/concurrency constraints;
  • minimum trade and active-session floors;
  • complexity/turnover penalty;
  • train, validation, and untouched forward-anchor partitions;
  • baseline comparison;
  • tie-breaking and deterministic ordering.

The forward anchor must not influence candidate discovery or parameter tuning. Stored score axes are provenance and ranking inputs; they do not replace portfolio-level replay.

Search algorithm (D-14): the recommended default is a deterministic greedy + local-swap search with explicit tie-breaking. A portfolio-level genetic algorithm re-opens the Round-1 overfit surface (F-07) and must be explicitly justified if chosen.

Anchor partition reuse (D-07): the untouched forward anchor is the v3.24.1 universal forward anchor (chronological last 20% of sessions), so Anchor_Net gating in champion_vault.py stays consistent — no parallel partition scheme.

Anchor-leak canary (mandatory): a synthetic record trading only inside the anchor window with outsized returns must never be scored, ranked, or selected; the optimizer code path must physically exclude anchor sessions from objective evaluation, and the canary test proves it end-to-end.

7. Targeted expert evolution contract

The first implementation is PA-first.

A child campaign records:

  • parent record_id and parent manifest generation;
  • assigned weekday/session window;
  • pinned entry and exit genes;
  • scratch results location;
  • validation and anchor verdict;
  • catalog disposition and lineage id.

The parent ledger, parent CLI, and prior gate evidence remain immutable. A child is not eligible for publication until its validation protocol passes. Engine-B support requires a separate design amendment because the current Stage-2 seam is Engine-A only.

8. Known limitations (proposed release)

  • Bar-resolution timestamps do not establish exact second-level execution.
  • Bar-range excursions are intra-bar high/low extremes from entry, not tick-path MAE/MFE; the Close variants and any intra-bar sequencing claim require a separate method tag.
  • The Python engine never exports expired orders — unfilled pending occupancy is a derivation from Fill_Window_Bars, not an observed fact.
  • Anchor provenance is sparse on historical rows (only 163 champion rows carry anchor_net > 0); anchor-based filters degrade to coverage-flagged fallbacks.
  • Correlation across zero-filled or non-overlapping sessions can be misleading; active-session denominators and coverage flags are mandatory.
  • A day-granularity schedule map is not a concurrency proof.
  • The forward-anchor split reduces leakage but does not eliminate regime/era bias.
  • Multi-instrument support is blocked until contract specifications are versioned.

9. Decision state

The following are proposed defaults and are not yet approved:

Decision Proposed value
Timestamp precision BAR
Primary key record_id (replay identity; schema version + generation live on the row identity — D-11)
UID handling alias only; retain all conflicts incl. DIVERGENT_ENGINE_CONTRACT class
Same-timestamp overlap conservative collision; same-bar fill→exit = minimum 1-bar occupancy
Relay conflict action reject new lower-priority entry
Fleet margin explicit versioned model
Optimizer validation train/validation/untouched anchor; anchor = v3.24.1 universal forward anchor
Optimizer algorithm greedy + local-swap, deterministic (D-14, recommended)
Targeted evolution scope PA-first; Engine-B later
Storage budget measure first, then set
Initial population tiered (D-03): Gen-1 ≈ 1,200 records (805 champions + 423 family reps); V300 + pre-v3.20.0 OPT excluded-with-reason; full pool = classified background backfill
Excursions BarRange-first via existing intra-bar extremes (D-10); Phase 0 byte-identity-gated commit

10. Review provenance

Round 1 (2026-09-25, commit 0d82a8fe) — Senior review against the original draft. Implementation surfaces inspected:

  • Backtesting/backtest.py:4079-4170 — trade reconstruction and facet attribution;
  • Backtesting/engine/simulate.py:232-235, :530-541 — friction and mark-to-market excursion;
  • Backtesting/scripts/extract_champion_daily_pnl.py:114-125, :242-245, :403-421 — pool identity, parity, publication;
  • Backtesting/Dashboard/callbacks/portfolio.py:555-603, :1096-1118 — metadata collision approximation;
  • Backtesting/scripts/backtest_exit.py:2-21 — Engine-A Stage-2 scope.

Round 2 (2026-09-28, commits 95a97b1b→6141547f, engine v3.24.1→v3.30.1 across the review window) — combined Senior + jr-dev (Big Pickle) feasibility review; all figures independently re-verified on disk. Additional surfaces inspected:

  • live DB census (read-only immutable=1): 7,442 rows / 5,851 distinct UIDs / 805 champion-class at ~20:30 — 10,620 / 9,966 / 154 by ~20:45 after the parallel Genome Evaluation DB rebuild (census volatility = the F-15 argument for the manifest-as-census rule);
  • master parquet set (Exports/Master/): six production masters rebuilt 2026-09-22 + DI variant datasets;
  • Backtesting/engine/simulate.py:556-563 — existing intra-bar unr_best/unr_worst extremes (D-10 reuse block); same-bar exit block; pending fill/expiry window;
  • Backtesting/backtest.py:250 — BACKTEST_VERSION = "3.30.1" single source (splash banner v3.30.0 lags it — staleness class persists, P0.1);
  • Backtesting/backtest.py:4140-4195 — trade log gated on not silent or include_trades (Phase-1 re-replay scope);
  • extract_champion_daily_pnl.py:170-183, :289-301 — preload print pattern (stdout-interleaving root cause for the "mislabel", F-14) and worker preload architecture (Phase-1 template).

Status: Draft contract. Update only through an approved Gate 0 decision or an explicit post-implementation verification refresh.