Method3 Nov 2026From the stored backtest

How the test is kept honest

A test on past prices is easy to flatter by accident. These are the rules ours follows, and what happened when we ran it twice.

Hypothetical backtest, not live results

The question

A test on past prices is easy to flatter without meaning to. Test only the companies that are still around, buy at prices nobody could have got, forget the costs, let one lucky stock carry everything, and almost any rule looks good.

So what stops ours from doing that? And how do we know the test itself has no mistake in it?

How we tested it

Every strategy, the random picks and the S&P 500 go through the same test (3 Nov 2016 to 3 Nov 2026, $100 every trading day, dividends counted, nothing sold). These are its rules.

Companies that later left the index are included. On each past day the test only picks from the companies that were in the S&P 500 on that day. Firms that later shrank, were bought or went bust are in there. Leaving them out would quietly remove many of the losers.
Buys happen at the next open. Stocks are ranked on prices up to the day's close. The buy is made at the next trading day's opening price, never at a price the ranking had already seen.
Costs are charged. Every trade pays 0.05%, which is 5 cents on a $100 buy.
No stock or sector can take over. Once the portfolio is past $2,000, no more than 10% of it can sit in one stock or 30% in one sector. If the top pick would break that, the money goes to the next best.

Then we checked the test itself by running it twice. The first run was the original research, on one public set of prices. The second is the site's own engine, which is the same rules written again in another programming language, on data from another provider (FMP prices and S&P 500 membership history).

The result

The same test run twice: yearly return over the ten years

Winners on sale

Research run▲ 23.8%

Stored run▲ 21.8%

Steady winners

Research run▲ 23.2%

Stored run▲ 21.2%

Winning streak

Research run▲ 23.5%

Stored run▲ 20.7%

For the S&P 500 the two runs end within $22,669 of each other: $577,208 in the research run and $554,539 in the stored one, from the same $251,200 put in.

For the strategies, the stored run came out lower every time, by between 2.0 and 3.1 percentage points a year. The widest gap is Buy the dip: 21.9% in research, 18.8% stored. Every strategy finished ahead of the index in both runs.

The limits matter too. In the research runs, looser limits of 20% and 40% lifted a plain past-winners rule from 24.1% to 27.5% a year, with a worst drop of −45.2% in place of −40.8%. The better-looking number came from taking more risk.

What it means

Two builds of the same rules, on two sets of data, give answers that are close. That makes it less likely that a slip in one build, or a quirk in one set of prices, produced the results. Where the two differ, the site shows the figures from its own stored run.

What it does not prove

An honest test of the past is still not a forecast. These ten years were unusually kind to large US companies whose prices were already rising.
Both builds share the same rules. Running them twice cannot catch a mistake in the rules themselves.
A few companies are missing. The second price provider has no history for some companies that left the index long ago, so the stored run cannot pick them. That flatters its results slightly.
Real buying has frictions the test leaves out. Taxes, broker fees above the cost charged here, and the price moving before your order goes through.
Nobody invested this money. Live results are likely to be lower.

Hypothetical backtest, not live results. These figures come from testing rules on past prices. Nobody invested this money, and past performance does not guarantee future results. Stocks Not Stuff is research and education, not financial advice: nothing here is a recommendation to buy or sell anything.

The question

So what stops ours from doing that? And how do we know the test itself has no mistake in it?

How we tested it

Every strategy, the random picks and the S&P 500 go through the same test (3 Nov 2016 to 3 Nov 2026, $100 every trading day, dividends counted, nothing sold). These are its rules.

Companies that later left the index are included. On each past day the test only picks from the companies that were in the S&P 500 on that day. Firms that later shrank, were bought or went bust are in there. Leaving them out would quietly remove many of the losers.

Buys happen at the next open. Stocks are ranked on prices up to the day's close. The buy is made at the next trading day's opening price, never at a price the ranking had already seen.

Costs are charged. Every trade pays 0.05%, which is 5 cents on a $100 buy.

No stock or sector can take over. Once the portfolio is past $2,000, no more than 10% of it can sit in one stock or 30% in one sector. If the top pick would break that, the money goes to the next best.

The result

The same test run twice: yearly return over the ten years

Winners on sale

Research run▲ 23.8%

Stored run▲ 21.8%

Steady winners

Research run▲ 23.2%

Stored run▲ 21.2%

Winning streak

Research run▲ 23.5%

Stored run▲ 20.7%

For the S&P 500 the two runs end within $22,669 of each other: $577,208 in the research run and $554,539 in the stored one, from the same $251,200 put in.

What it does not prove

An honest test of the past is still not a forecast. These ten years were unusually kind to large US companies whose prices were already rising.

Both builds share the same rules. Running them twice cannot catch a mistake in the rules themselves.

A few companies are missing. The second price provider has no history for some companies that left the index long ago, so the stored run cannot pick them. That flatters its results slightly.

Real buying has frictions the test leaves out. Taxes, broker fees above the cost charged here, and the price moving before your order goes through.

Nobody invested this money. Live results are likely to be lower.