Hello all,
It happened to me a few times. I write a strategy and check it with Strategy Analyzer. It works perfect for a day or two or three days. But when I am checking it for a month or longer time, results are terrible.
Did this happen to you too? Is it just a bad strategy or something else?
Please share
This can definitely happen and it does not always mean the strategy itself is bad. One thing to look at is the data series type the strategy depends on. Time-based bars and derivative/custom series, like tick-based logic or calculated values from intrabar movement can behave differently depending on how the test is run. Even a slight difference each time a tick chart is processed can create compounding effects if the strategy’s calculations depend on those values. I would not rely only on Strategy Analyzer results. Strategy Analyzer can make a strategy look good over a short period, but over a longer test the assumptions and small calculation differences can start to show up more clearly.
In my experience, Market Replay is the better way to test this kind of strategy, especially if the strategy depends on intrabar behavior. It is important to run replay at a lower speed. Even 1x replay is still replaying one second at a time, not necessarily every individual tick exactly the way live data forms, so it can still be a little off.
Sometimes the strategy really is weak over a larger sample, but sometimes the difference comes from how the backtest/replay engine handles the data. I would compare Strategy Analyzer, Market Replay, and live/sim results before deciding the strategy is bad. Also, even profitable strategies can go through drawdowns. I have seen quants talk about strategies being in drawdown for weeks or even months, while still being profitable over a longer multi-year period. So a bad month does not automatically mean the strategy is broken, but it does mean you need to understand whether the drawdown is normal for that system or outside what the strategy should reasonably experience.
If a strategy works only for a few days and is failing for a 30 days test, it clearly is a bad strategy.
A test should at least last several months to be sure you have all kinds of markets in your test.
Thank you! good idea to check the strategy on Market replay also!
I wouldn’t take the strategy analyzer results to heart whatsoever - it gives an extremely quick and rough idea of potential performance. If you are green or treading water I would run it historical or SIM. Its fun to run and see the results but I wouldn’t use it make go/no go decisions, I see it as test 1 of multiple tests, the absolute bottom rung test.
Hi @WaleeTheRobot,
I am really interested in your comment from last week, where you wrote:
In my experience, Market Replay is the better way to test this kind of strategy, especially if the strategy depends on intrabar behavior. It is important to run replay at a lower speed. Even 1x replay is still replaying one second at a time, not necessarily every individual tick exactly the way live data forms, so it can still be a little off.
Regarding the importance of running Market Replay at lower speeds, would you be willing to provide some more details about inaccurate performance results you’ve experienced when running it on too fast of a speed?
Have you noticed specific kinds of inaccuracies that are introduced to the performance results when running Market Replay at too fast of a speed, and that become more and more of a problem as the Market Replay speed is increased?
Finally, and my apologies for all the questions, but what is the fastest Market Replay speed that you would personally feel comfortable using if your goal was performance results that were accurate enough for a typical day trader who isn’t a scalper and who holds positions for minutes or hours, or what do you feel comfortable using for your own trading?
Anything you can add is greatly appreciated, and I realize you may not have answers for some of these questions. Even if you could answer one of them that would be fantastic and appreciated by many community members, including future ones that find this forum thread in the future.
Best regards,
Brook
It really depends on the strategy and chart type. With NinjaTrader Market Replay, the main issue I’ve seen is that higher replay speeds can change the effective evaluation behavior of the strategy. The same replay period can produce different entries at 1x, 10x, 100x, etc. The practical result is that entries, stops, and targets can be missed. Entries can occur at different prices or times, stops and targets can be filled differently, and order modifications may not happen the same way.
This usually becomes more noticeable as replay speed increases, especially when the strategy depends on intrabar movement. It can also create a compounding effect. For example, if an entry is missed or triggered at a different point, then everything after that trade is now different. It can also affect built-in indicator values like EMA, ATR, etc., especially if the candles or price sequence being evaluated are no longer matching the slower replay.
This is especially important for strategies that rely on tick-by-tick behavior, tight stops, fast entries or exits, or non-time-based charts like Renko and tick charts. Range and volume charts might be fine, but it’s been a while since I’ve looked into it. In general, the more a setup depends on the exact sequence of price movement and the exact timing of strategy evaluation, the more cautious I would be with higher replay speeds.
For a typical day-trading strategy that is not scalping and holds trades for minutes or hours, I would probably be comfortable somewhere around 10x to 15x on a 1-minute chart, assuming the results are still mostly consistent when compared with slower replay speeds. The reason I use that range is that, at 15x, a 1-minute bar is compressed into about 4 seconds of replay time. So within that 1-minute candle, the strategy still has several opportunities to detect a potential entry, rather than only checking once for the entire bar. To me, that feels like a reasonable balance between moving through the replay quickly and still giving the strategy enough chances to catch setups inside the candle. If I were using a faster replay speed, like 50x or 100x, then a full 1-minute bar could pass in roughly one second or less. At that point, I would be much less comfortable trusting the results, especially if the strategy depends on intrabar conditions.
One thing I should add is that I do not personally rely on NinjaTrader’s Strategy Analyzer for my backtesting. My workflow is different. I use a NinjaTrader strategy to extract the bars and any custom indicator values I need into a database. For example, if I want to use ATR, I do not rely on NinjaTrader’s built-in ATR for the backtest. I calculate my own ATR so that I have a consistent and reliable anchor every time. The same idea applies to other indicators or custom values I want to test.
Once the data is extracted, I use custom Python scripts for the actual backtesting. In my experience, that approach is much faster and gives me more control over the logic, assumptions, and indicator calculations than relying only on Strategy Analyzer. It also makes it easier to keep the data and calculations consistent from one test to another.
After I’m happy with the Python backtest, I convert the strategy logic into C# for NinjaTrader and then test it using Market Replay. I use Market Replay more as a validation step to see how the converted NinjaTrader strategy behaves in the platform, including order handling, fills, and real-time-style execution behavior.
Wow, thank you @WaleeTheRobot for taking the time to share all of this. Your answer is pure gold. I got a ton out of it and I am sure many other community members will as well.
Thank you, I’ll try Market replay
I’ve been doing a controlled test on whether NinjaTrader Market Replay speed materially changes strategy results.
This matters quite a lot to me because I routinely replay years of historical data, and running at low Playback speeds would make that workflow practically impossible.
My setup is quite specific:
Strategy runs headless — it is not applied to a chart
Calculate.OnBarClose
Same strategy build and parameters
Same Market Replay dataset
Same test window
Same execution configuration
Only the Playback speed changes
For this test I compared:
10Ă— vs Max
The comparison window covered 10 trading days, producing:
25 completed trades
50 strategy events
Hundreds of strategy-state values captured at every entry
Rather than simply comparing final P&L, I compared the two runs at the individual event and trade level.
The comparison included:
Trade count
Event sequence
Entry timestamps
Entry prices
Exit timestamps
Exit prices
Commissions
Realised P&L
MFE
MAE
Strategy-state snapshots captured at each entry
This was intended to detect not only obvious differences in final performance, but also subtler differences in timing, execution or calculated strategy state.
For the 25 trades tested, 10Ă— and Max produced exactly the same trading behaviour.
25 / 25 trades matched
50 / 50 strategy events matched
Entry timestamps matched
Entry prices matched
Exit timestamps matched
Exit prices matched
P&L matched
Commissions matched
MFE matched
MAE matched
The aggregate statistics were consequently identical as well, including:
Gross Profit
Gross Loss
Net P&L
Profit Factor
Win Rate
Average MFE
Average MAE
The large strategy-state snapshot captured at each entry also showed no divergence in the strategy-relevant calculations.
I’m not suggesting that Playback speed can never affect NinjaTrader strategies generally.
In particular, I think there is an important distinction between strategies that depend heavily on:
intrabar evaluation
price-crossing behaviour
OnEachTick
OnPriceChange
other timing-sensitive logic
and a strategy such as mine that runs headlessly and evaluates completed bars using Calculate.OnBarClose.
But for this specific configuration and test window, I found:
No material difference whatsoever in the resulting trades or execution between 10Ă— and Max Playback.
I thought this was worth contributing because the result suggests that Playback-speed sensitivity may depend considerably on strategy architecture and calculation mode, rather than the simpler assumption that:
higher Playback speed = less accurate strategy results
Of course everything matches. You wipe out all differences/errors by using Calculate.OnBarClose.
You ignore all intrabar data.
I will give an extreme example: You go long on a daily chart if the price crosses an 20 days EMA. So no matter when the cross happens during the day, your price will always be the close of the bar. Even if, in real time, the crossing happened 3 hours ago at a price 25 ticks more profitable.
So if somebody uses the exact moment the conditions are filled, the price will never be the one at the close of the bar. So your profits are not correct.
My vision is that all depends on how you handle the timing when the conditions for a signal are there. For the way I trade, Playback gives wrong returns as he uses the wrong prices. My signal is at the moment that the conditions are fulfilled, no matter what time it is. I never wait the close of the bar.
I think you’re conflating two different things here.
My strategy is genuinely designed and run as Calculate.OnBarClose. I did not switch an intrabar strategy to OnBarClose for this test in order to “wipe out” differences.
OnBarClose controls when the strategy evaluates. It does not mean the underlying Playback data can be corrupted or skipped without consequence. If higher Playback speed were materially altering the event stream used to build bars and indicator state, that can still propagate into different completed bars, timestamps, indicator values, entry states and ultimately different trades.
That is exactly why I compared more than final P&L. I compared the full event sequence, entry/exit timestamps and prices, MFE/MAE, and hundreds of captured strategy-state values at entry. They matched.
What my test does not establish is that Max-speed Playback is accurate for a strategy whose logic depends on the exact intrabar moment a condition becomes true. I already said that in the original post.
Your daily-EMA example is therefore a different question. It shows the difference between an intrabar-entry strategy and an OnBarClose strategy; it does not, by itself, demonstrate a Playback-speed error.
And saying “your profits are not correct” simply does not follow. That would only be true if my intended rules were to enter intrabar and I were incorrectly modelling them at bar close. They aren’t.
That is what I mean. People should be careful and pay attention to the difference in strategy between you and me. Because that will define if the results will be correct or not. Your strategy gives correct results for you.
For my strategy playback gives wrong execution prices and timing. I trade on intrabar data and intrabar generated signals. At high speed my signals are not correct because at that speed not every tick is checked to generate a signal. All fills are later and at a different price.
You are right. I was thinking from my strategy’s point of view instead of yours.