After 9 years of using Ninjatrader I still don’t understand the difference between market replay data and historical data.
I first compared the data of both. I exported the data delivered by the datafeed for live trading and compared that data with market replay data that I downloaded. I checked dozens of 1 minute data (open, high, low, close), and there was not a single quote that did not match. His lead for me to the conclusion that market replay data is for 100% identical to historical data.
So I don’t understand why you should use market replay data.
Market replay data has to be downloaded day by day, only one day at the time. That’s hugely time consuming. Or you should select “market replay” in the settings, which can have an negative impact of the function of NT8.
Historical data is much easier as you can download a whole month in one time. And if you “reload all historical data” from your trading charts, you even don’t have to do the download separately thru TOOLS.
I check each trading day after the close in market replay, and I see no real difference between the quotes and the real execution. So I always used (for at least 9 years) historical data.
You probably don’t notice a difference if you are just looking for OHLC. Looking at the file size, you’ll see the market replay data possibly be 100MB+ per day. While the files for historical data might reach a couple 1000KB together if that. I would imagine that the market replay data has more data to be processed to make it realistic, while the historical is meant to be faster and given more accuracy through tick replay without the real-time simulation. They are both different. One is designed to support back-testing, while the other is designed to be forward-testing with past data.
Are you able to get all the data in historical data…when I tried running an indicator where I checked for the presence of tick data in the loaded and reloaded historical data I found the tick data missing for many days…like one entire thursday friday had no tick data, although there was tick data prior to those days and after those days, very random behaviour, My indicator had added tick data series as a secondary series…AI told me there is nothing i can do and I have to rely on market replay if I want the tick data for those 2 specific days.
I’ve never noticed, but I have the habit of going back and actually downloading the historical tick data even if I have it for that day from having the platform opened.
I think that the important thing is: how is the distribution of the data?
If the data is evenly distributed over the total session, then the additional ticks that are missing in historical, will have almost no impact on the results.
If however the missing data would be concentrated in a way that the distribution is not evenly distributed anymore there would be a problem.
The historical data should have enough data to give a reliable result. If you have one tick every two minutes the distribution would not be valid for a reliable analysis.
My opinion on point 1:
I took all tickdata for january 6 for the RTH session. I had 440,016 ticks MES.
Average per second was 18,67 ticks.
Average 1 tick every 54 ms.
Only the first 15 minutes after the opening the volatility was much higher.
Within 1 second periods, the max difference in price was:
• 31.8% 0 difference
• 78.1% 1 tick, or less
• 95.5% 2 ticks, or less
• 99.9% 3 ticks, or less
My opinion on point 2:
440.016 ticks are enough for a statistical valid observation that is reliable. And 18.67 ticks average per second means that there are no gaps, but rather a smooth evolution of data. Average there is 1 tick every 54 ms.
As there are no big ups or down in price in the historical data, the missing data will fall within the evolution that we see in the historical data.
If you would have all the historical ticks, would adding 100 K data make any difference? No. More data is not always improving the result.
The missing ticks I found were 2 entire days, not a single tick on Thursday and Friday and it was in the first week of december, so over a month old date, i was looking back at 200 days of historical data.
Sorry mate I cant remember, could be 6A could be NQ or something like that, but it definetely happened because I took a different approach after that, I knew the 1 minute candles data is there in historical data because I can see that on the chart so I based my logic off the candle and price instead of the tick and price and then everything worked fine.
BTW I was looking back in data from State.DataLoaded, not running OnBarUpdate, but I assume the results would be the same.
I use several timeframes, including smaller than 1 minute. I would see every gap immediately in my charts because timeframes <1 minute are based on ticks.
I trade ES, MES, NQ and MNQ. Never saw any gap in the last years.
there could be so many factors, since you have the code to test already, you should try running it for a instrument you have never used…in my case i never open a chart, the time i saw this issue was the first time i opened a chart to write an indicator to plot lines between my entry and exit points of strategy placed trades
so open a chart you have never opened and load last 200 days of data, reload multiple times just to be sure and then run your code and see if you still getting very high quality data, if you are then thats very great…
In my case 1 minute was the chart data series and i was adding the tick data series in my indicator State.Configure
Market Replay data and Historical data are two different approaches for working with “historical” data on the NinjaTrader platform. Note the quotes around “historical,”
as both Market Replay data and Historical data, serve specific use cases for reviewing past events on the platform.
Market Replay allows you to replay the market/price exactly as it occurred during real live trading sessions,
replicating actual market conditions using Market Replay data that has been downloaded
from the Historical Data window. The content within the Market Replay data contains
Level 1, Level 2, Level 3, and tick data, which is very useful, ideally for backtesting and offers the highest
accuracy available, if traders need reliable backtesting results from past data.
also there are plenty of providers are available where traders can source high-quality Market Replay data from. Additionally, @WaleeTheRobot is correct, file size matters when comparing Market Replay data vs. historical data storage.
on the other hand historical data allows you to replay the market using just tick data, that is cached in your system or downloaded from the Historical data window.
The content within the historical data files, contains just Level 1 data only.
Historical data is typically used as a cache to fill your charts with price data.
If you open your charts without making a connection to a broker,
you would have noticed that Ninjatrader has automatically filled your charts
with pricing data.
The data you see on the chart is produced from the cached historical data, in addition if you are using the Ninjatrader tick replay function, which is used often by order flow traders,
without having any cached historical data in your system, your charts will fail to show the right pricing data. it is also well documented within the Ninjatrader documentation, that using historical data for backtesting will produce inaccurate results and it is not the recommended data to use for backtesting and simulations