File Sizes and Disk Planning
Customers placing first orders almost always ask the same three questions: how big is the data, how long will it take to download, and how much disk do I need to keep it. This page gives the concrete numbers for each major package, explains the compression choices Kibot has made (RAR for FTP archives, plain text for the API and direct downloads), and covers the practical disk and network considerations for both one-off backtests and ongoing production use.
Rough Sizes by Package
| Package | Uncompressed | Compressed (RAR) |
|---|---|---|
| S&P 500 daily (current + historical members) | ~150 MB | ~30 MB |
| S&P 500 1-minute (current + historical members) | ~25 GB | ~3 GB |
| S&P 500 tick + bid/ask | ~600 GB | ~80 GB |
| All stocks and ETFs, 1-minute + daily | ~160 GB | ~20 GB |
| All stocks and ETFs, tick + bid/ask | ~10 to 12 TB | ~1.5 TB |
| Top 40 futures continuous, 1-minute | ~3 GB | ~400 MB |
| Top 40 futures continuous, tick + bid/ask | ~250 GB | ~35 GB |
| Major forex pairs, 1-minute | ~8 GB | ~1 GB |
| Major forex pairs, tick | ~400 GB | ~55 GB |
Numbers are approximate. They grow each year as the archive extends. Tick-with-bid-ask is by far the largest format; a single liquid name (SPY, AAPL, ES) can produce 10 to 20 GB of tick data per decade. Customers who only need 1-minute or daily resolution can fit the entire universe on a single consumer SSD.
Why RAR, Not Gzip, on the FTP Channel
FTP archives are distributed as RAR. Two reasons. First, RAR's solid-mode compression on tabular text achieves roughly 10:1, against gzip's typical 4:1 or 5:1 on the same data; for the 160 GB intraday-plus-daily set that is the difference between a 16 GB and a 32 GB download. Second, RAR supports per-file random access inside a solid archive better than gzip's stream-only format, which matters when only a few symbols need to be extracted from a multi-gigabyte archive.
The API and direct-download channels serve plain text. The API supports gzip via the standard HTTP Content-Encoding negotiation; the client opts in by sending Accept-Encoding: gzip and the server gzips the response on the fly. See Compression for the negotiation details and a known middlebox gotcha where corporate proxies decompress in transit.
Parallel Downloads Do Not Improve Throughput
A frequent question: can I run 10 or 20 parallel downloads to finish faster. The answer is no. The bottleneck is the destination disk's random-write throughput, not the network. Writing many files concurrently to the same drive (HDD, SSD, or NVMe) creates contention that slows aggregate throughput below what a single-stream download would achieve. Five concurrent connections is the practical maximum for most consumer hardware and is also the recommended API connection ceiling. Above five, the network may still saturate but the disk cannot keep up.
For the very largest tick archives (multi-TB), a single download manager with a sequential queue is usually faster than even five parallel API workers, because the manager's TCP window and the disk's sequential-write path are both optimised for one large file at a time.
Estimating Download Time
Approximate wall-clock times at common consumer link speeds:
| Package size | 20 Mbps | 100 Mbps | 1 Gbps |
|---|---|---|---|
| 3 GB (S&P 500 1-minute compressed) | ~20 min | ~4 min | under 1 min |
| 20 GB (all stocks+ETFs intraday+daily compressed) | ~2 hours | ~30 min | ~3 min |
| 80 GB (S&P 500 tick+bid/ask compressed) | ~9 hours | ~2 hours | ~12 min |
| 1.5 TB (all stocks+ETFs tick+bid/ask compressed) | ~7 days | ~1.5 days | ~3.5 hours |
Real-world throughput is typically 60 to 80% of the link's nominal speed because of TCP overhead, server-side per-account rate-limiting, and shared infrastructure during peak hours. The afternoon US market hours are the worst time for large downloads from Kibot's data servers because the same servers are also handling live update requests. Early morning ET or weekend overnight runs see the highest sustained throughput.
NTFS Compression and Storage Tricks
For Windows users planning to keep the archive online for backtesting, NTFS file-level compression on the storage folder reduces the on-disk footprint of plain-text CSV by roughly 4:1 with no application-level changes. The cost is a small CPU overhead at read time, which is dominated by disk I/O on most workloads anyway. NTFS compression is the recommended way to keep both the uncompressed text files (for ad-hoc inspection and scripting) and a reasonable disk footprint.
Do not apply NTFS compression on the destination folder during the active download. The compression-on-write overhead can throttle download speed enough to time out individual requests. Download first, then enable compression on the folder, which compresses existing files in the background.
For very large tick archives, a separate option is to convert from CSV to a columnar binary format (Parquet, Arrow, custom binary) after download. The size reduction is similar to NTFS compression but the read speed in Python or Rust is 10 to 100 times faster on aggregate scans. The tradeoff is the upfront conversion time and loss of human-readable inspectability.
Incremental Storage Planning
For ongoing subscribers, the daily delta is modest:
| Frequency | Daily delta (all stocks+ETFs) |
|---|---|
| Daily bars only | ~10 MB |
| 1-minute bars | ~150 to 250 MB |
| Tick + bid/ask | ~8 to 15 GB |
A subscriber on the 1-minute tier should budget roughly 50 to 70 GB per year of incremental storage. A subscriber on the tick tier should budget 3 to 5 TB per year. Both budgets include uncompressed text; halve those numbers if the working directory is on NTFS compression. Pruning old data is also an option; the on-disk archive can be kept rolling at any window the customer needs.
Key Takeaways
- The full stocks-and-ETFs intraday-plus-daily archive is roughly 20 GB compressed, 160 GB uncompressed. Tick-with-bid-ask for the same universe is roughly 1.5 TB compressed, 10 to 12 TB uncompressed.
- FTP uses RAR for its ~10:1 compression ratio and per-file random access. The API and direct downloads serve plain text with optional gzip negotiation.
- More than 5 concurrent downloads does not improve throughput. The bottleneck is the destination disk, not the network or the API.
- Afternoon US market hours are the slowest time to download from Kibot. Early morning ET and overnight weekends are the fastest.
- NTFS compression on the storage folder roughly halves the on-disk footprint of plain-text CSV with negligible read overhead. Apply it after the download finishes, not before.
- For sustained subscribers, plan 50 to 70 GB per year of 1-minute incremental storage, or 3 to 5 TB per year for full tick.
Related
- Data format reference, the CSV schemas that drive the sizes quoted above.
- API compression, gzip negotiation and the middlebox decompression gotcha.
- Data updates, the four delivery channels (API, FTP, Agent, direct) and their update cadences.
- Kibot Agent, the free Windows downloader that respects the 5-connection guideline by default.
- API rate limits and throttling, the connection limits that interact with download parallelism.
- Binary search algorithm, O(log n) random-date access into the multi-gigabyte sorted CSV files once they are on disk.