Selection, Not Strategy

1Introduction

A parameter sweep that searches thousands of configurations and keeps the best one will always report a good backtest. The question that matters is whether the configuration it kept was kept for a reason that survives contact with unseen data. This paper answers that question twice, for the same production pipeline, and gets the same answer both times: no.

The system under study searches fourteen strategy families over a grid of parameters, at three bar sizes, for each instrument in a 59-instrument universe. Candidates pass through a fixed protocol — a training window, a 35-day embargo, an out-of-sample holdout, a set of viability tiers, and a buy-and-hold comparison — and the survivor is promoted with a quality rating from one to five stars. That protocol is conventional. It is also, we find, close to uninformative about what happens next.

Two studies are reported. Study I asks whether anything observable at selection time predicts forward return: it freezes the entire protocol at a cutoff twelve months in the past, replays every family’s champion over the unseen year, and correlates each selection-time metric against what actually happened. Study II asks the follow-up question that Study I forces. If fitted metrics do not predict, the natural replacement is to rank candidates by their realised out-of-sample return — to promote what has actually been working. That substitution is only sound if such a ranking persists, and Study II measures the persistence directly over 52 consecutive weekly anchors.

Both results are negative and are reported as such. We have not found a selection rule that works; we have measured, with some precision, how badly two candidate rules fail. The practical consequences for the pipeline are set out in §5, and the reasons not to over-read any of this are in §6.

2Data

2.1Instruments and venues

The universe is the pipeline’s own tradeable set: 31 cryptocurrency perpetual-swap symbols priced from Binance spot/perp OHLCV, and 30 US equities and ETFs (9 ETFs, 21 large caps) from Alpaca IEX bars. Equity series are regular-trading-hours only — there is no feed between 16:00 and 09:30 ET — and the NYSE calendar including half-days gates every session. Crypto trades continuously.

Bars are stored at 15-minute, 1-hour and 4-hour resolution. Equities are validated on native 15m and 1h only; their 4h series is derived from 1h and appears in this paper as research material, never as a tradeable interval.

2.2Coverage, and why the panel is ragged

Instrument histories differ by years, and the protocol has a hard minimum: 180 days of training, a 35-day embargo and at least 90 days of holdout must fit before any selection cutoff. Symbols that cannot meet it fail closed and are reported as untestable rather than being quietly dropped or padded. In Study I that removed two crypto symbols (HYPE, ONDO), leaving 59 instruments. In Study II, where the requirement compounds with a 365-day ranking window to 670 days of prior bars per anchor, coverage is reported per anchor. Table 1 gives the breakdown for both studies.

Equity history is itself unequal. Only SPY, QQQ, AAPL and NVDA are backfilled to 2023 and were selected on the full 790-day lookback; the other 26 tickers begin 2024-06-06 and were selected on a 428-day lookback of which roughly 303 days are training. All 30 cleared the protocol’s minimums, but they did not all get the same amount of evidence.

2.3A known hole in the tape

Binance served nothing for five consecutive 15-minute bars on 2023-03-24 (12:45–13:45 UTC). This is a venue outage, not a backfill failure: the exchange still returns nothing for that window and no refetch can close it. Study II does not clamp its windows around the gap — the five bars are simply absent and the surrounding bars are treated as adjacent. A 75-minute hole once in a four-year 15-minute series moves no indicator materially, whereas clamping every training window to start after it would delete the study’s entire first year of anchors. The trade is deliberate and is recorded here so it can be argued with.

2.4Cost model

Every simulated fill in both studies pays 4 basis points taker, 2 basis points maker and 5 basis points slippage; equities are levered 2×. This is the same cost model the production sweep validates against, so the numbers here are directly comparable to the ones the pipeline reports about itself. Equity fills additionally respect session semantics: a gap through a stop fills at the next session’s open, never at the stop price.

3Method

3.1The selection protocol

Both studies re-run the production protocol, disjoint_holdout_v1, unmodified. For a cutoff C it builds three strictly ordered, disjoint windows ending at or before C: a training window of at least 180 days, a 35-day embargo, and a holdout of at least 90 days or 20% of the lookback. A grid search fits parameters on training only; candidates are filtered through six descending viability tiers and a training-only buy-and-hold comparison at 6, 12 and 24 months; the survivor is scored on the holdout and assigned a 1–5 star rating. Nothing that closes after C is visible to any part of this.

3.2Study I — the blind matrix

The cutoff was pinned to 2025-08-09, twelve months before the run. For every instrument, the protocol was executed to completion and the winning configuration of every one of the fourteen families was retained — not only the family the pipeline would have promoted. Each was then frozen and replayed exactly once over the following year, 1,971 tests in total.

Retaining all fourteen champions is what separates two failure modes the pipeline could not previously tell apart: the library has no edge versus the library has edge and the picker misses it. With every family measured on the same frozen year, the best available configuration per instrument is observable after the fact, and the distance between it and the protocol’s actual pick is selection error by construction.

3.3Study II — 52 weekly anchors

Study II replaces the single cutoff with a rolling one. At each Monday anchor A from 2024-07-01 to 2025-06-23, three nested windows are built:

1. Selection, ending at the cutoff C = A − 365 days. Training, embargo and holdout all close at or before C, so every candidate’s parameters are fitted on bars older than C.
2. Ranking, the window [C, A). Each family champion is replayed here and ranked by realised return. This window is blind with respect to selection.
3. Test, the week [A, A+7d). The top 3 ranked configurations are replayed once more. This window is blind with respect to both selection and ranking, and it is the only number that scores the rule.

Rolling A forward one week at a time converts “does the out-of-sample ranking persist?” into 52 out-of-sample observations per instrument. The full selection sweep is re-run at every (instrument, anchor) pair, because the training window moves with C; 8,046 picks were scored in total.

3.4Controls

• No cached features. Indicators are recomputed in memory from bar zero. The persistent feature cache is neither read nor written, so no stale or version-skewed cache row can influence a number.
• No look-ahead. Every series is truncated at the cutoff before selection, and the harness asserts that the last selection-visible bar closes at or before it. Window ordering was verified across all artifacts: training → embargo → holdout are strictly ordered and end at or before the cutoff, and every out-of-sample bar starts at or after it.
• No writes. Both studies are read-only against the production database. The live strategy table was read once, for the comparison in §4.6, and never modified. Nothing measured here was promoted.
• Failures are reported, not filled. Instruments that cannot support the protocol at a given cutoff are recorded as untestable or insufficient-history for that anchor. The ragged panel that results is part of the result.

4Results

4.1Two opposite markets, the same loss

Table 2 is the headline. Over the frozen year the crypto tape fell 63.1% and the equity tape rose 22.1%. The median strategy returned −28.2% and −22.1% respectively — the same loss, in opposite regimes.

The symmetry is the finding. Against the crypto crash, a system that is mostly out of the market reads as protection: 79.3% of picks beat buy-and-hold. Against the equity rally the identical behaviour is simply expensive, and the same comparison collapses to 16.7%. “Beats buy-and-hold” is the metric the sweep optimises toward, and it flatters low exposure. Absolute return is negative in both regimes.

4.2The edge exists; the picker misses it

58 of the 59 testable instruments had at least one strategy×interval configuration that made money out of sample (29/29 crypto, 29/30 equities). Table 3 sets the protocol’s actual pick against three alternatives, including that hindsight ceiling.

Two things stand out. The gap between the protocol’s −17.5% / −21.5% and the ceiling’s +50.9% / +31.1% is selection error, not a missing strategy — the ceiling ranks configurations the protocol itself generated and then placed below its choice. And picking by the embargoed holdout is the worst rule tested in both cohorts, worse than picking by the training return the holdout exists to correct.

4.3Decay, and what the metrics predict

Figure 1 tracks the same configurations through the three stages at which they are measured. Median training return (+11.1% crypto, +2.4% equity) is fitted by construction. The embargoed holdout — the number the star rating is minted from — already gives most of it back. The frozen year takes the rest.

Table 4 puts the three selection-time metrics side by side. In both cohorts the training return ranks the future marginally better than the holdout does. That is the signature of a validation stage that is not adding information: it measures the same fitted noise one window later. The star rating, which is derived from the holdout, inherits the problem, and its ordering is not even monotone (Table 5).

One family illustrates the whole mechanism. PMax leads the crypto training column at +75.3%, prints −8.4% on the holdout, and returns −20.8% over the frozen year. A configuration can look like the best thing in the sweep and be worthless forward.

4.4Bar size is the largest effect

The strongest and cleanest gradient in Study I is not a strategy property at all. Holding family, instrument and year fixed and varying only the bar size moves the crypto median from −48.8% at 15 minutes to −9.9% at four hours, monotonically (Table 6, Figure 2). Crypto 4h champions were profitable 4.6× more often than 15m champions.

Unlike the correlation results this is not a subtle statistical claim: the 15-minute tape is where fee drag and noise consume the edge. Of all the findings in Study I this is the one that translated directly into a pipeline change (§5).

4.5Which families held up

Table 7 ranks the families. Not one has a positive median in either cohort, and family ranking by training return and by out-of-sample return are close to unrelated. VWAPMeanReversion has a negative median crypto training return (−9.6%) and finishes third out of sample; PMax leads training and finishes fifth. Ranking families by backtest return is not merely noisy, it is misleading.

4.6The live rows look good for the wrong reason

Replaying the strategy rows that were actually approved and live in production over the same year gives a median of +10.6% on crypto — apparently far better than the blind picks’ −17.5%. It is not evidence that production is well tuned. Those rows carry validation dates inside the test window, so their parameters were fitted on windows overlapping the very year they are scored on. They are contaminated by construction, and the gap between the two numbers is a direct measure of how much of production’s apparent quality is in-sample fitting. The equity rows, with the same advantage and a rising tape, still beat buy-and-hold only 25.3% of the time and returned a median −3.8%.

4.7Study II: the ranking does not persist

Study I leaves one obvious move: stop ranking by fitted metrics and rank by realised out-of-sample return instead. Table 8 tests it directly.

The correlation between a configuration’s realised return over a 365-day out-of-sample window and its return in the very next week is −0.059. Restricting to the strongest picks does not help (−0.059 for rank-1 only), and neither asset class differs materially. A ranking that does not survive one week cannot be the basis of a weekly promotion rule.

Table 9 shows the same thing from the other side. Mean weekly return is not monotone in rank: rank 2 (−0.265%) is worse than both rank 1 (−0.056%) and rank 3 (−0.129%). The top-ranked configuration is also unstable, changing in 27.3% of consecutive weeks — so a rule that followed it would be re-selecting a different strategy roughly every fourth week, paying costs for a signal that carries no information.

4.8Study II: what the panel returned

An equal-weight portfolio of every scored pick compounds to −7.69% over the 52 weeks, with 30.8% of weeks positive and a mean of −0.152% per week (Figure 3). The drawdown is not dramatic; it is a slow, consistent bleed, which is what a zero-information rule paying transaction costs should look like.

Table 10 makes a distinction that the pipeline’s own reporting historically blurred. 10 of 53 instruments finished the panel profitable, and 10 beat buy-and-hold — but only 4 did both. The two lists barely overlap because they measure different things: beating buy-and-hold on this panel mostly means the instrument fell and the rule was not in it.

Figure 4 plots every instrument’s walk-forward return against its own buy-and-hold over the same weeks. Points below the dashed parity line lost to simply holding the asset; the great majority are, and the vertical spread is narrow while the horizontal spread covers two orders of magnitude. The rule neither tracked the winners nor avoided the losers.

The extreme cases in Table 11 are instructive. AAVE rose +469% over the panel while the rule returned −42.0%. These are not tail events in a working system; they are the same behaviour Study I identified — low exposure — priced by a rising tape.

For completeness: choosing, in hindsight, the single best-performing configuration per instrument across the whole panel yields a median +12.01% compounded, positive for 41 of 53 instruments. As in Study I this is an upper bound on what better selection could have captured, not a strategy — it requires knowing the answer in advance.

5Discussion

The two studies compose into a single, uncomfortable statement. The metrics the pipeline selects on do not rank forward return (Study I), and the most natural non-fitted replacement — realised out-of-sample performance — does not rank it either (Study II). Between them they eliminate both the incumbent signal and its obvious successor.

What survives scrutiny is narrower and worth stating precisely:

• Absolute return is the only defensible yardstick. “Beats buy-and-hold” is regime-dependent and rewards being out of the market. The same behaviour scored 79.3% against a crash and 16.7% against a rally while losing money in both.
• The star rating is not a forecast. Rank correlation with realised return is 0.237 / 0.213 and the ordering is not monotone. Its only defensible use is ordering eligibility; 1★ retains value as an exclusion.
• Bar size dominates strategy choice. A 38.9-point monotone gradient across every family is larger than any difference between families, and unlike the correlation results it is robust.
• The library is not the bottleneck. A profitable configuration existed for 58 of 59 instruments. Generating candidates is solved; choosing among them is not.

Three changes followed from Study I and are already in the production sweep: candidate ranking now prefers 4-hour bars ahead of the in-sample viability tier, which moves the median crypto selection from −17.5% to −5.9% when this study’s own artifacts are re-ranked through it; OpeningRangeBreakout was removed from the search after finishing last of fourteen in both cohorts, independently reproducing an earlier benchmark on different windows [2]; and a daily job now re-runs the whole protocol and records the realised forward ranking as evidence.

Study II is the reason a fourth change did not follow. Rewiring execution onto that forward ranking was the obvious next step, and the persistence measurement says it would have been a mistake: a −0.059 correlation and a 27.3% weekly turnover describe a rule that would churn positions and pay costs for nothing. The daily job continues to accumulate evidence; nothing was promoted on the strength of it.

6Limitations

These are the reasons not to over-read anything above. They are ordered by how much they should change your confidence.

1. Two regimes, one of each. Study I covers twelve months of a crypto crash and twelve of an equity rally; Study II covers a single overlapping year. That is enough to demonstrate the exposure asymmetry and to reject the persistence hypothesis at this horizon. It is not enough to characterise behaviour across a full cycle, and a different pair of years could move every point estimate here.
2. Hindsight ceilings are upper bounds, not strategies. The +50.9% / +31.1% of Study I and the +12.01% median of Study II are computed by choosing, after the fact, the configuration that turned out best. They size the prize available to better selection. No rule can capture them and none is proposed.
3. Sparsity inflates the persistence result. Roughly half of Study II’s picks never trade in their test week (Table 9). A correlation computed over a sample where the modal observation is exactly zero is a weaker instrument than the sample size suggests. The direction of the finding is unaffected — there is no positive correlation hiding in the traded subset — but the precision is overstated by n.
4. The panel is ragged. 50 of 53 instruments contributed all 52 anchors; three contributed 16, 22, 44 and 7 contributed none, all because they listed too recently to satisfy the 670-day history requirement. Aggregate statistics therefore weight long-listed instruments more heavily, and the crypto cohort in particular is 23 names rather than 31.
5. A known gap in the tape. The 2023-03-24 Binance outage (§2.3) sits inside the crypto 15-minute series and is not clamped around. We judge a 75-minute hole immaterial to any indicator used here; that judgement is not free of assumption.
6. Current code, historical data. These are the selections today’s protocol would have made, not the ones that ran at the time. Study I also had to be re-run after a calendar defect was found mid-study: an exchange calendar that began in 2024 made 2023 holidays look like missing bars and silently clamped four deep-history equity tickers’ training windows forward. Those four were re-run on the corrected calendar with zero gap warnings.
7. Unequal equity history. 26 of 30 equity tickers were selected on a 428-day lookback against 790 days for the other four (§2.2). All cleared the protocol minimums; they did not all get the same evidence.
8. Single pipeline, single cost model. Everything here describes one implementation of one protocol at one cost assumption. The negative results are about this construction, not a general claim about technical trading.

7Conclusion

Across 1,971 blind out-of-sample tests and 8,046 walk-forward observations, we could not find a selection signal that predicts forward return. In-sample metrics rank it at ρ ≈ 0.312 at best, with the embargoed holdout ranking it worse than the training return it corrects; realised out-of-sample performance ranks it at −0.059, which is to say not at all. In two opposite market regimes the pipeline lost approximately the same amount, which identifies its apparent defensiveness as low exposure rather than skill.

The one durable, actionable effect we did find is structural rather than strategic: longer bars are monotonically better, by a margin larger than any difference between strategy families. That result changed the production sweep. Nothing else here did, and nothing measured in either study was promoted to trade.

We publish both results because negative results about selection are scarce and expensive to produce, and because the alternative — reporting the backtest and omitting the year that followed — is how this class of system usually gets described. A profitable configuration existed for 58 of 59 instruments. Finding it in advance remains an unsolved problem, and until it is solved these strategies are not a reason to allocate capital.

References

1. Zarattini, C. and Aziz, A. (2023). Can Day Trading Really Be Profitable? Evidence of Sustainable Long-Term Profits from Opening Range Breakout (ORB) Day Trading Strategy vs. Benchmark in the US Stock Market. SSRN Working Paper. — the source for the OpeningRangeBreakout family evaluated in §4.5, which finished last of fourteen in both cohorts of Study I.
2. Meet Zoro Research (2026). TradingView strategy benchmark. Internal report, 7 August 2026. — an independent evaluation on different windows that reached the same conclusion about OpeningRangeBreakout.
3. Meet Zoro Research (2026). Classic out-of-sample test — every strategy, every ticker. Internal report, 9 August 2026. — the full Study I writeup, including per-instrument tables omitted here.