kibot

Response compression

The Kibot API supports gzip and deflate response compression using the standard HTTP Accept-Encoding / Content-Encoding negotiation. Compression is opt-in: the server only compresses when the client advertises support, and it picks gzip when both are on offer. For tick and intraday responses the wire savings are large (typically 5 to 10 times smaller), which on a typical broadband link is the difference between a 5-minute download and a 30-second one. This article covers the request/response contract, the per-language client setup, and the failure mode that quietly turns compression off in practice.

The contract

The client sends the standard Accept-Encoding request header:

Accept-Encoding: gzip, deflate

The server reads the header, picks the strongest compression method both sides support (gzip by default), compresses the response body, and returns:

Content-Encoding: gzip
Vary: Accept-Encoding
Transfer-Encoding: chunked

The Content-Encoding header tells the client which algorithm was used. The Vary header tells caches that the same URL can return either a compressed or uncompressed body depending on the request header. Most HTTP libraries decompress transparently when they see Content-Encoding: gzip; a few require an explicit opt-in or read raw bytes off the socket and leave decompression to the caller.

A request that omits Accept-Encoding entirely gets the raw uncompressed body back, which is correct behaviour by HTTP spec but easy to leave on the table when porting code or writing a quick test.

Wire savings

The data the API returns is text (CSV for history, TAB-separated for adjustments) and compresses well. Rough rules of thumb:

  • Tick and tick-bid-ask: 5 to 10 times smaller on the wire. A 100 MB raw tick file lands as 10 to 20 MB.
  • 1-minute intraday: similar 5 to 8 times ratio. Material on multi-year history pulls.
  • Daily and weekly: 2 to 4 times, but the absolute payload is small enough that the savings are pocket change per call.

The constant cost of advertising gzip support is zero, so it's always worth setting even when the per-call saving is small.

Enabling compression by language

curl

Pass --compressed to advertise the standard Accept-Encoding: gzip, deflate header and decompress transparently:

curl --compressed 'http://api.kibot.com/?action=history&symbol=MSFT&interval=tick&period=1&user=...&password=...'

Python (requests)

requests sends Accept-Encoding: gzip, deflate automatically and decompresses transparently. No setup is required:

import requests
r = requests.get("http://api.kibot.com/?action=history&symbol=MSFT&interval=daily&period=10")
print(r.text) # already decompressed

.NET (HttpWebRequest / HttpClient)

HttpWebRequest requires an explicit opt-in:

request.AutomaticDecompression = DecompressionMethods.Deflate | DecompressionMethods.GZip;

HttpClient goes through a HttpClientHandler with the same property:

var handler = new HttpClientHandler {
 AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate
};
var client = new HttpClient(handler);

This is the path used by Kibot's official .NET sample application.

Go (net/http)

Go's net/http auto-decompresses when the request omits Accept-Encoding and the server still responds with gzip. If the request explicitly sends Accept-Encoding: gzip, the runtime assumes the caller wants the raw body and leaves decompression to user code. Simplest pattern is to not set the header and let the transport handle it.

Rust (reqwest)

Enable the gzip feature in Cargo.toml and turn it on in the client builder:

let client = reqwest::Client::builder().gzip(true).build()?;

Node.js (axios / undici)

Modern Node HTTP clients decompress transparently by default. axios sets Accept-Encoding and decompresses; undici / native fetch do the same as of Node 18+.

When compression silently stops working

The most common cause of "I thought I was getting compressed responses but now I'm not" is antivirus or proxy software on the client side intercepting the HTTPS connection. The interceptor sees the request, decompresses the server's response to scan it for threats, and then rewrites the response headers on the fly before passing the body to the actual HTTP library, often dropping or corrupting Content-Encoding in the process.

From the application's perspective the body arrives uncompressed and several times bigger than expected, downloads slow down dramatically, and the issue is intermittent because the interceptor only kicks in for some categories of traffic.

Diagnostic checklist when this happens:

  1. Capture the raw response headers (curl with -v, or a packet trace). If Content-Encoding: gzip is missing, the server didn't compress. If it's present but the body is plain text, something between server and application stripped the compression.
  2. Temporarily disable the antivirus's HTTPS / web-shield component and retry. If the response is suddenly compressed again, the antivirus was the culprit. Common offenders are Avast, ESET, Kaspersky, and corporate TLS-intercepting proxies.
  3. If a proxy is in the path, check its rules. Some content-filtering proxies decompress, scan, and re-emit uncompressed to simplify their own logging.

The same pattern explains a related class of "downloads are corrupted" reports: a proxy that mis-handles chunked transfer encoding can truncate or pad the body and produce CSVs that start with garbage bytes or end mid-row. Kibot has investigated several such reports and they have consistently resolved by disabling or reconfiguring the intercepting middleware on the client side.

Server-side pre-compression (a question that comes up)

For very large pulls (multi-year tick history for a single instrument can hit hundreds of megabytes), some customers ask whether the server can pre-compress the requested file into a 7-zip or similar archive and stream the archive instead. The answer is no, by design: pre-compressing a multi-gigabyte file is CPU-intensive and would block the request until the compression finished, often longer than the original transfer. The on-the-fly gzip path is a better tradeoff: the first byte ships immediately and every subsequent incremental update is small enough that further compression buys little. The recommended pattern for the largest historical pulls is to download once over gzip-compressed HTTP, then keep it locally updated with daily delta requests.

Key Takeaways

  • Send Accept-Encoding: gzip, deflate on every request. Most HTTP libraries do this by default; verify yours does.
  • The server responds with Content-Encoding: gzip when compression was applied. Most libraries decompress transparently; check the .NET, Rust, and Go notes above if your language is in the manual-opt-in category.
  • Tick and intraday responses compress 5 to 10 times; daily and weekly are smaller wins but still worth the zero-cost header.
  • "Compression stopped working" almost always means an antivirus or TLS-intercepting proxy is stripping Content-Encoding. Disable the HTTPS-inspection component temporarily to confirm.
  • Server-side pre-compression into 7-zip archives is not offered. The gzip-on-the-fly path is faster end to end because the first byte ships immediately.
  • Server responses, HTTP status codes and the compression header summary.
  • Rate limits and throttling, How compression and connection limits interact in bulk-download workflows.
  • File sizes and disk planning, RAR vs gzip on the FTP channel and uncompressed-text sizes on direct download.
  • API client examples, Working code in curl, Python, Rust, Go, TypeScript, C++, R, and the official .NET sample, every example with compression enabled.
  • History request, Where compression matters most because tick payloads are the largest.
  • Adjustments request, The incremental-update pattern that keeps daily delta downloads small.