Data Completeness and Quality Assurance

Kibot maintains roughly 28 years of minute-level historical market data, and delivering clean coverage at that scale requires continuous monitoring, automated detection, and careful repair. This page explains why gaps exist in historical market data, how Kibot finds and fixes them before delivery, what level of completeness to expect per instrument type, and how a user should handle any remaining gaps in their own analysis.

Why Gaps Exist

No historical data vendor can guarantee 100% coverage for every instrument on every trading day. Gaps arise from a variety of legitimate causes, most of which reflect real market structure rather than data loss.

Market holidays. US equity markets are closed on approximately 9 holidays per year. No data is generated on these dates. See Market holidays for the full calendar and rules.
Trading halts. Exchanges halt trading in individual securities for regulatory reasons, pending news, or when circuit breakers are triggered. During a halt, no trades execute and no bars are produced. Recent examples include:
June 3, 2024: Approximately 40 NYSE-listed symbols, including Berkshire Hathaway Class A (BRK.A), were halted for roughly two hours due to a technical glitch in a software release that caused pricing data errors and triggered LULD (Limit Up-Limit Down) pauses.
October 2025: The SEC suspended trading in multiple Nasdaq-listed companies (including Etoiles Capital Group and Pitanium Limited) for up to 10 trading days due to concerns about social-media-driven price manipulation, with Nasdaq extending the halts pending further information.
2024-2025: Several Nasdaq-listed companies were suspended for failing to meet listing compliance standards (minimum price, filing requirements), leading to extended halts and eventual delisting.
Illiquid instruments. Thinly traded stocks may have no trades for extended periods, minutes, hours, or even entire sessions. This is not a data gap but a reflection of actual market conditions.
Specialist and market maker delayed opens. Some securities open later than the 9:30 AM bell, particularly after overnight news or during periods of order imbalance. The first bar of the day may not appear until the security actually begins trading. Per kibot's support Faq, this sometimes happens even for very liquid securities.
Corporate events. Mergers, acquisitions, earnings surprises, and other material news can trigger regulatory halts that last from minutes to hours.
Exchange outages or technical issues. While rare, exchange-side technical problems can interrupt the flow of market data at the source.
Pre-market and after-hours sessions. Coverage during extended hours (8:00-9:30 AM and 4:00-6:30 PM ET) is naturally lower than during the regular session, since fewer participants trade and many instruments see little to no activity.

How Kibot Detects and Fixes Gaps

Kibot's data infrastructure is designed to catch and repair gaps before they reach downloads. Four techniques work together.

Multi-server verification. Data is stored across multiple independent servers, and cross-comparison between servers allows detection of discrepancies that would be invisible on any single system.

Binary search algorithms. For multi-gigabyte data files, Kibot uses Binary search algorithm techniques to rapidly locate specific date ranges and identify missing intervals without scanning entire files sequentially.

Automated expected-vs-actual checks. Automated tools compare the trading dates and bar counts expected (based on exchange calendars and session schedules) against what actually exists in Kibot's files. Deviations are flagged for review.

Surgical repair from source files. When a gap is found in a destination file, the missing data is recovered from source files and inserted atomically using binary I/O operations. For example, a 7-minute gap on 02/23/2026 affecting stock intraday data (minutes 13:19-13:24) was detected and repaired by surgically inserting the missing lines at the correct byte offset, without rewriting the entire file. This approach preserves file integrity and minimizes the risk of introducing new errors during repair.

Expected Completeness by Instrument Type

Completeness expectations vary by asset class and liquidity.

Stocks and ETFs show very high completeness for liquid names (SPY, AAPL, MSFT, QQQ). Less liquid names, particularly small-cap or micro-cap stocks, may have natural gaps where no trades occurred.

Futures completeness varies by contract liquidity. Front-month (nearest expiration) contracts have the best coverage. Back-month contracts and less popular commodities may have sparse data, especially during overnight sessions.

Forex has continuous coverage during market hours (Sunday evening through Friday afternoon ET). Forex data consists of bid/ask quotes only, there is no centralized exchange volume for spot FX, so volume fields are not meaningful.

Indexes are calculated values published by the exchange during trading hours. There is no volume data, and coverage follows exchange hours only.

Handling Gaps in Your Analysis

Four practical guidelines apply when gaps matter to downstream analysis.

Reconstruct missing bars if needed. Some backtesting frameworks require a bar for every expected interval. If your framework cannot handle missing bars, you can fill gaps using the previous bar's close price with zero volume. Be aware that this introduces synthetic data points. Per kibot's support FAQ, the standard pattern is to repeat the previous valid bar's close as Open, High, Low, and Close, with Volume set to 0, until the next bar with real trades appears.

Account for shortened sessions. The US equity markets close early on certain days (the day before Independence Day, the day after Thanksgiving, Christmas Eve). These shortened sessions produce approximately 210 minutes of bars instead of the usual 390. You can check Kibot's API for current shortened session dates: http://api.kibot.com/?action=shortenedsessions. See Market holidays for the full list.

Do not filter out flash crashes. Extreme price moves that appear to be errors are often real events. These spikes show up more often in tick data than in minute bars. The August 24, 2015 flash crash, for example, produced genuine prints at prices far from the prior close. Filtering these out will give you an inaccurate picture of historical market behavior and understated risk metrics.

Pre-market and after-hours coverage. If your analysis uses extended-hours data (8:00-9:30 AM and 4:00-6:30 PM ET), expect lower bar coverage than during the regular session. Many instruments have no pre-market or after-hours trades at all.

Report an Issue

If you find what you believe is a data gap or quality issue, Kibot welcomes reports. Contact through the contact form with the symbol, date range, and data type affected.

Key Takeaways

Historical market data gaps are usually real, holidays, halts, illiquidity, delayed opens, corporate events, not vendor errors.
Kibot uses multi-server cross-checks, binary-search range lookups, automated expected-vs-actual bar-count checks, and surgical byte-level repair to catch and fix gaps before delivery.
Liquid stocks and ETFs (SPY, AAPL, MSFT, QQQ) have very high completeness; micro-caps, back-month futures, and overnight sessions naturally have more gaps.
To get a fixed bar count (e.g., 390 per regular session), fill missing bars by repeating the previous close with zero volume, but treat these as synthetic.
Shortened sessions produce ~210 bars instead of 390; the current list is queryable at http://api.kibot.com/?action=shortenedsessions.
Do not filter extreme-move bars as errors, flash crashes and similar events are historically real and removing them understates risk.

Data sources, the exchanges, brokers, and vendors Kibot cross-references to define "expected" bar counts
Market holidays, the calendar that drives expected-vs-actual gap detection
Extended hours coverage, why pre-market and after-hours bars are often missing for less liquid symbols
Bid ask and NBBO quotes, how the quote stream interacts with the spike-filtering and completeness pipeline
Binary search algorithm, the lookup technique Kibot uses to locate date ranges in multi-gigabyte files
Data format reference, schemas for the files whose completeness is being monitored
Trading hours, regular vs. extended session definitions that determine expected bar counts