Hello all,
It happened to me a few times. I write a strategy and check it with Strategy Analyzer. It works perfect for a day or two or three days. But when I am checking it for a month or longer time, results are terrible.
Did this happen to you too? Is it just a bad strategy or something else?
Please share
This can definitely happen and it does not always mean the strategy itself is bad. One thing to look at is the data series type the strategy depends on. Time-based bars and derivative/custom series, like tick-based logic or calculated values from intrabar movement can behave differently depending on how the test is run. Even a slight difference each time a tick chart is processed can create compounding effects if the strategy’s calculations depend on those values. I would not rely only on Strategy Analyzer results. Strategy Analyzer can make a strategy look good over a short period, but over a longer test the assumptions and small calculation differences can start to show up more clearly.
In my experience, Market Replay is the better way to test this kind of strategy, especially if the strategy depends on intrabar behavior. It is important to run replay at a lower speed. Even 1x replay is still replaying one second at a time, not necessarily every individual tick exactly the way live data forms, so it can still be a little off.
Sometimes the strategy really is weak over a larger sample, but sometimes the difference comes from how the backtest/replay engine handles the data. I would compare Strategy Analyzer, Market Replay, and live/sim results before deciding the strategy is bad. Also, even profitable strategies can go through drawdowns. I have seen quants talk about strategies being in drawdown for weeks or even months, while still being profitable over a longer multi-year period. So a bad month does not automatically mean the strategy is broken, but it does mean you need to understand whether the drawdown is normal for that system or outside what the strategy should reasonably experience.
If a strategy works only for a few days and is failing for a 30 days test, it clearly is a bad strategy.
A test should at least last several months to be sure you have all kinds of markets in your test.
Thank you! good idea to check the strategy on Market replay also!
I wouldn’t take the strategy analyzer results to heart whatsoever - it gives an extremely quick and rough idea of potential performance. If you are green or treading water I would run it historical or SIM. Its fun to run and see the results but I wouldn’t use it make go/no go decisions, I see it as test 1 of multiple tests, the absolute bottom rung test.
Hi @WaleeTheRobot,
I am really interested in your comment from last week, where you wrote:
In my experience, Market Replay is the better way to test this kind of strategy, especially if the strategy depends on intrabar behavior. It is important to run replay at a lower speed. Even 1x replay is still replaying one second at a time, not necessarily every individual tick exactly the way live data forms, so it can still be a little off.
Regarding the importance of running Market Replay at lower speeds, would you be willing to provide some more details about inaccurate performance results you’ve experienced when running it on too fast of a speed?
Have you noticed specific kinds of inaccuracies that are introduced to the performance results when running Market Replay at too fast of a speed, and that become more and more of a problem as the Market Replay speed is increased?
Finally, and my apologies for all the questions, but what is the fastest Market Replay speed that you would personally feel comfortable using if your goal was performance results that were accurate enough for a typical day trader who isn’t a scalper and who holds positions for minutes or hours, or what do you feel comfortable using for your own trading?
Anything you can add is greatly appreciated, and I realize you may not have answers for some of these questions. Even if you could answer one of them that would be fantastic and appreciated by many community members, including future ones that find this forum thread in the future.
Best regards,
Brook
It really depends on the strategy and chart type. With NinjaTrader Market Replay, the main issue I’ve seen is that higher replay speeds can change the effective evaluation behavior of the strategy. The same replay period can produce different entries at 1x, 10x, 100x, etc. The practical result is that entries, stops, and targets can be missed. Entries can occur at different prices or times, stops and targets can be filled differently, and order modifications may not happen the same way.
This usually becomes more noticeable as replay speed increases, especially when the strategy depends on intrabar movement. It can also create a compounding effect. For example, if an entry is missed or triggered at a different point, then everything after that trade is now different. It can also affect built-in indicator values like EMA, ATR, etc., especially if the candles or price sequence being evaluated are no longer matching the slower replay.
This is especially important for strategies that rely on tick-by-tick behavior, tight stops, fast entries or exits, or non-time-based charts like Renko and tick charts. Range and volume charts might be fine, but it’s been a while since I’ve looked into it. In general, the more a setup depends on the exact sequence of price movement and the exact timing of strategy evaluation, the more cautious I would be with higher replay speeds.
For a typical day-trading strategy that is not scalping and holds trades for minutes or hours, I would probably be comfortable somewhere around 10x to 15x on a 1-minute chart, assuming the results are still mostly consistent when compared with slower replay speeds. The reason I use that range is that, at 15x, a 1-minute bar is compressed into about 4 seconds of replay time. So within that 1-minute candle, the strategy still has several opportunities to detect a potential entry, rather than only checking once for the entire bar. To me, that feels like a reasonable balance between moving through the replay quickly and still giving the strategy enough chances to catch setups inside the candle. If I were using a faster replay speed, like 50x or 100x, then a full 1-minute bar could pass in roughly one second or less. At that point, I would be much less comfortable trusting the results, especially if the strategy depends on intrabar conditions.
One thing I should add is that I do not personally rely on NinjaTrader’s Strategy Analyzer for my backtesting. My workflow is different. I use a NinjaTrader strategy to extract the bars and any custom indicator values I need into a database. For example, if I want to use ATR, I do not rely on NinjaTrader’s built-in ATR for the backtest. I calculate my own ATR so that I have a consistent and reliable anchor every time. The same idea applies to other indicators or custom values I want to test.
Once the data is extracted, I use custom Python scripts for the actual backtesting. In my experience, that approach is much faster and gives me more control over the logic, assumptions, and indicator calculations than relying only on Strategy Analyzer. It also makes it easier to keep the data and calculations consistent from one test to another.
After I’m happy with the Python backtest, I convert the strategy logic into C# for NinjaTrader and then test it using Market Replay. I use Market Replay more as a validation step to see how the converted NinjaTrader strategy behaves in the platform, including order handling, fills, and real-time-style execution behavior.
Wow, thank you @WaleeTheRobot for taking the time to share all of this. Your answer is pure gold. I got a ton out of it and I am sure many other community members will as well.
Thank you, I’ll try Market replay