Two curves can tell different stories
A profitable backtest beside a losing forward experiment creates an obvious question: what changed? It is tempting to start with market conditions, model drift or execution quality. But there is an earlier question that can completely change the investigation: were the two systems running the same experiment in the first place?
The Copybetter work brought that question into focus. A retained September audit found that the displayed historical replay and the forward paper experiment differed in execution, capital constraints, settlement and score provenance. The paper loss was real. The older profitable curve was also an output of a particular research process. They were not an exact controlled comparison.
My work on the research-to-forward-testing system connects the models, replay contracts, operational evidence and reporting. The useful outcome of this investigation was a more precise account of what could be trusted and what had to be rebuilt. It did not establish that the identified differences caused the loss, or that correcting them would make the strategy profitable.
This article focuses on the engineering and analytical lessons. It uses aggregate diagnostics and public-safe process diagrams rather than private strategies, wallet identities, execution settings or account finances. The objective is to show how a research system earns a meaningful comparison, not to publish a trading recommendation.
First, establish what happened
The retained observation covers paper positions opened from 25 August through the 16 September audit checkpoint. There are 511 accepted positions, 508 settled and three still open. Those positions span twelve experiment instances and five distinct recorded policy configurations. The sample is a history of an evolving system, not one unchanged strategy running continuously.
The audit checked whether recorded source events existed, whether actions mapped to the intended assets, whether accepted positions respected their recorded time boundaries and whether the ledger arithmetic reconciled. It found no duplicate accepted events in the examined history and no failures in those basic action, timing and arithmetic checks.
That evidence supports the mechanics of the paper ledger. It does not prove that simulated liquidity was executable, that every eligible opportunity reached the system or that the model had economic value. Getting the arithmetic right is essential, but it is a different accomplishment from demonstrating a useful predictive signal.
The history also includes missing score coverage and inactive dates. A flat section of a curve can mean that nothing happened, that nothing qualified or that part of the system did not supply evidence. The reporting layer needs to retain those distinctions. Otherwise an apparently calm period can be mistaken for successful risk control when it was actually a gap in admission or data.
The contract matters more than the label
Calling two systems “backtest” and “shadow” tells us little about whether they are comparable. A meaningful comparison needs the same eligible opportunities, model and policy identities, time boundaries, fees, capital rules and settlement assumptions. Differences can be legitimate, but they must be visible before the results are interpreted.
The older replay used a historical execution proxy and different capital and settlement treatment from the forward paper experiment. It also tracked cash in a way that did not constitute a fully marked portfolio value with open principal. Those choices affect what a displayed curve means, even when every point is calculated correctly under its own rules.
| Dimension | Comparison question | Why it matters |
|---|---|---|
| Opportunities | Did both systems see the same eligible events? | A missing opportunity changes the experiment. |
| Information | Were scores available before each action? | Later reconstruction cannot prove original availability. |
| Execution | Do price, timing and liquidity assumptions agree? | A proxy fill is not an observed exchange fill. |
| Capital | Are exposure and reservation rules comparable? | Prior fills change later capacity. |
| Settlement | When was outcome evidence usable? | Settlement timing changes available capital. |
| Valuation | Is the curve cash, settled result or portfolio value? | These measure different things. |
One concrete defect involved freshness logic for opposite-outcome purchases. The forward-contract replay did not apply the same tighter rule as shadow. The repair put that behaviour in a shared Rust method and added regressions across the relevant action combinations. That reduces a real comparability gap; it does not explain all of the observed performance difference.
Another repair aligned historical scoring with the full previous UTC day rather than an earlier partial-day boundary. The successor history contract rejects older evidence where it cannot satisfy the new requirement. This is a good example of a change that improves future comparability without proving that the same mismatch existed throughout every earlier forward period.
Improve the system without improving the past
A corrected binary should produce a new result with a new identity. It should not inherit the label of an old replay and make the historical record appear as if the better implementation had always been used. The same principle applies to models, policies and data revisions.
The scoring audit also found cases where the mutable score table could not certify original availability. Some recorded scores were missing; others had been reimported after the original action. The accepted positions still retained their recorded predictions, but the surviving evidence did not establish the original import and cutoff provenance for every case.
The right conclusion is an evidence gap. It is not proof of future leakage, and it is not permission to fill the gap using whichever reconstructed value looks plausible. Preserved experiment artifacts are valuable precisely because mutable operational tables may no longer describe the exact state an earlier process observed.
Operationally, this means binding replay outputs to the relevant source revision, executable, model and policy. A later improvement can reuse unaffected work, but affected stages need new receipts. That approach makes recovery more efficient without letting convenience erase the identity of the experiment.
It also changes the conversation around a bad result. The team can separate a valid negative outcome from a broken comparison. Both deserve attention, but they lead to different work. One may challenge the model or strategy; the other challenges whether the displayed evidence can answer the question being asked.
Matching policies do not guarantee matching trades
Live and paper systems can apply the same policy and still encounter different opportunities. They observe different books at different times, receive different fills and consequently follow different inventory and capacity paths. A policy-match flag is useful evidence about configuration, but it is not a promise that their trade lists will be identical.
The retained investigation found both paper positions without live fills and live fills without matching paper positions. Those differences need to be decomposed by their actual causes: missing signals, selection, freshness, observed books, fills and sizing. Simply subtracting the final curves loses the operational path that produced them.
Retrospective attribution also needs restraint. If a subset of paper positions contributed much of the loss, that does not prove a strategy would improve by excluding them. The subset may have been selected using information only visible after the event. A proposed filter needs a fresh experiment or a properly separated chronological evaluation.
The same caution applies to tuning. Causal model folds protect one part of the research process, but they do not automatically remove optimism from repeated execution-policy selection. If a final replay reuses the source that selected the policy, it is not an untouched confirmation merely because the model predictions themselves were generated out of sample.
For the product, the implication is to keep research, replay, shadow and live histories identifiable. They can inform one another without being collapsed into a single success narrative. That makes the system more useful for research and more honest for the person reading the dashboard.
Make the next result easier to interpret
The next comparison should freeze an incumbent and successor against explicit opportunity, information, execution and accounting contracts. It needs qualified settlement timing and a clearly identified forward period. Where historical execution remains a proxy, that limitation should remain attached to the result rather than disappear after a successful run.
There is no need to wait for a perfect simulator to make progress. A bounded proxy can answer a bounded question. The problem starts when its result is described as if it had observed a richer world. Making the assumptions visible lets the team decide which uncertainty matters enough to investigate next.
The leadership contribution here is the discipline of comparable evidence. It means resisting premature explanations, giving defects explicit ownership and preserving the negative results alongside the fixes. It also means making the interface tell the same story as the underlying ledger, which is the subject of the trustworthy-dashboard case study.
A backtest is most useful when it prepares a better experiment. In this work, the gap between replay and forward operation exposed places where the contract needed to become more precise. The result is a clearer research path, not a retrospective claim that the strategy had been right all along.
Evidence: retained 16 September 2026 shadow-versus-backtest audit and aggregate audit receipt. Counts are available in the public evidence file. No current performance, investment suitability or future profitability is implied.