Survivorship bias: what it does to a backtest, with numbers
Between 10% and 23% of the positions in a historical run are companies that no longer exist. If your data does not contain them, your backtest never took their losses.
There is a specific reason strategies that look excellent on a chart lose money the moment they are funded, and it is almost never the strategy. It is the list of companies the test was allowed to see.
The mechanism
Take the current members of an index. Test a rule on their price history going back ten years. Every company on that list survived the ten years — that is precisely how it ended up on the list you downloaded. The bankruptcies, the delistings, the acquisitions at a discount, the reverse splits that preceded a collapse: none of them are in your sample, because none of them are in the index today.
Your test therefore never bought a company that went to zero. Not because your rule avoided them — it would have bought plenty — but because they were removed from the universe before the test began.
This is not a small correction applied at the end. It is a different experiment.
How big is it, actually
On our own data, when the screener rebuilds the universe as it stood on each historical date rather than using today’s list, between 10% and 23% of the positions in a run are in instruments that no longer trade. The range depends on the period and the filter: it is widest in tests that lean toward small caps and high volatility, which is to say, exactly the tests that look most exciting before you correct them.
We hold price history for 2,042 delisted instruments. That number is worth pausing on in both directions. It is large enough that removing them changes results materially. It is also far smaller than the raw count of inactive tickers in the registry — around 17,770 — because the vast majority of those never had a single price bar recorded. Counting all of them would be a more impressive claim and a false one.
Why almost nobody fixes it
Because the fix is not clever code. It is data that has to be kept deliberately, from before you needed it.
When a company delists, the commercial incentive is to drop it. It generates no quotes, nobody searches for it, it costs storage, and no customer complains about its absence — the absence is invisible by construction. A dataset that quietly discards dead companies looks identical to one that keeps them, right up to the moment you run a historical test, and even then it does not look wrong. It looks good. That is the problem.
The second correction, which is bigger than you expect
Survivorship is the famous one. The one that has killed more of our own results is the missing control group.
A result like “this signal produced +1.78% excess return, t = 4.55” reads as conclusive. It is not, until you know what comparable companies did over the same window. We ran exactly that test, believed it, and then added a matched control group — the same kind of names, the same period, without the signal.
The edge disappeared entirely.
Nothing about the original computation was wrong. It measured a real thing: the selected names went up 1.78%. It just was not the thing anyone cared about, because so did everything that looked like them.
What survives
We measure 14 event families across 1.6 million recorded occurrences, each with its own control group. Most do not survive. Volume shocks and range breakouts — two of the most widely traded setups in retail technical analysis — underperform their controls at t = −7.84. That is not a null result. It is a strong result in the wrong direction.
Post-earnings announcement drift did survive: +0.64% excess return, t = 2.83, measured over 40 quarters. Small, real, and about the size you would expect from something that has been public in the academic literature since 1968.
One thing worth knowing before you build anything: the edges live at 21 to 63 trading days. At a one-day horizon only 2 of 25 measured families clear t = 3. A great deal of event-driven trading is done at horizons where the measured effect does not exist.
Three questions to ask any backtest
- Was the universe rebuilt as of the test date? If the answer is “we used the current constituents”, the test is not measuring what it says.
- What did comparable names do? Without a control group, a positive number is a description of the market, not of the signal.
- How were the standard errors computed? Two hundred positions opened on the same morning are one bet on that morning. Clustered by date or not at all — anything else inflates the t-statistic by a factor nobody bothers to quote.
If a tool cannot answer all three, its backtest is a chart, not a measurement.
Our screener rebuilds the point-in-time universe, attaches a control group, subtracts transaction costs and clusters errors by date — and we publish the results that came back against us. You can run it without an account.