kibot

Installing wget for Kibot downloads

wget is the practical default for pulling Kibot data outside a browser when the customer is on Linux or macOS, when they need a scheduled or unattended run on Windows without installing Kibot Agent, or when they want a one-command download of the full plain-text URL list that the account page exports. This page covers installation on each of the three platforms and the few Kibot-specific patterns that come up in support correspondence.

Why wget for Kibot data

Three properties matter for the way Kibot ships files. wget resumes interrupted downloads with -c against the same URL, which matters because individual tick files can exceed 30 GB and home connections drop them mid-transfer. It accepts a plain text file of URLs through -i, which lines up exactly with the "Plain Text Format" link every Kibot data package exposes on its account download page. And it preserves the server-side filename via --content-disposition, so a URL like http://api.kibot.com/?get=kIPnkBH5 writes out as APA.txt rather than a meaningless query-string name.

curl works too. One customer (per support correspondence) reported switching from wget to curl on a slow connection and seeing throughput improve, which is occasionally true on links where wget's default retry/backoff is more conservative than curl's. The defaults below are tuned for wget, but the same URLs work with curl -OJL if a platform happens to ship curl rather than wget. On macOS specifically, curl is preinstalled and wget is not; see the macOS section below.

Installing on Windows

Windows does not ship wget. The simplest path is the pre-compiled binary from Eternally Bored.

  1. Download the binary. Open https://eternallybored.org/misc/wget/, find the latest version in the table, and click the EXE link in the x64 column to download wget.exe.
  2. Create a folder. Put the binary somewhere stable, for example C:\wget\, rather than leaving it in Downloads.
  3. Add the folder to PATH. Press the Windows key, type Edit the system environment variables, and press Enter. In System Properties click Environment Variables. Under System variables (for all users) or User variables (for the current user only), select Path, click Edit, click New, and add C:\wget. Click OK on every dialog.
  4. Verify. Open a fresh Command Prompt or PowerShell window (an existing window will not pick up the new PATH) and run:
wget --version

A one-liner alternative for step 3, run from an elevated PowerShell, sets PATH machine-wide without clicking through the dialog:

[Environment]::SetEnvironmentVariable("PATH", $env:PATH + ";C:\wget", [EnvironmentVariableTarget]::Machine)

Both approaches require closing every open command-line window before the new PATH is visible. Substitute the actual install folder if it is not C:\wget.

Customers on Windows who want a full GUI for downloading and scheduling are usually better served by Kibot Agent; wget is the right tool when an existing pipeline already runs from a shell on Windows (WSL, Git Bash, scheduled PowerShell) and pulling wget in is cheaper than adding Agent to the mix.

Installing on Linux

wget is preinstalled on most distributions. Check first:

wget --version

If the command is missing, install via the system package manager. The three common cases:

# Debian, Ubuntu, Linux Mint
sudo apt update && sudo apt install wget

# RHEL, CentOS, older Fedora
sudo yum install wget

# Fedora 22 and newer
sudo dnf install wget

The package manager places wget on the system PATH automatically; no further configuration is needed. Linux is the most common Kibot environment for unattended bulk pulls precisely because cron plus wget plus the plain-text URL list is a three-line setup with no GUI involvement.

Installing on macOS

macOS ships curl but not wget. The supported path is Homebrew.

  1. Install Homebrew (skip if already installed). Open Terminal (Applications/Utilities, or Spotlight + "Terminal") and run:
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
  1. Install wget.
brew update
brew install wget
  1. Verify.
wget --version

Homebrew puts wget on PATH automatically. macOS customers who do not want to install Homebrew can use the preinstalled curl instead: curl -OJL "<kibot-url>" mirrors wget --content-disposition for a single file, and a while read loop over the plain-text URL file replicates wget -i well enough for unattended pulls. Either way, Kibot Agent is not available for macOS, so wget or curl is the only option short of running a Windows VM. One customer summarised the trade-off plainly: "your software is windows only it seems. I am using MacOS/Linux. Can you recommend any other download manager? Clicking thousands of links is really out of the question" (per support correspondence). The answer for that workflow is the plain-text URL list piped through wget.

Downloading Kibot data with wget

Every Kibot data package's account download page exposes a "Plain Text Format" link that returns the full list of file URLs, one per line. The recommended pattern is to save that list and feed it to wget:

  1. Login to your account and open the download page for the data package.
  2. Right-click the "Plain Text Format" link and choose "Save link as..." to save the URL list as, for example, downloadtext.txt.
  3. Run wget against the file:
wget --content-disposition --trust-server-names -i downloadtext.txt

--content-disposition tells wget to use the filename the server provides in the HTTP Content-Disposition header (the symbol name and .txt suffix), and --trust-server-names tells it to honour redirects when picking the output name. Without these two flags the resulting files are named after the raw query string and need a rename pass before they are useful.

For a single file rather than a list, the same flags apply to a direct URL:

wget --content-disposition --trust-server-names "http://api.kibot.com/?get=<file-token>"

Quote URLs that contain query-string ampersands. A customer hitting 500 Internal Server Error from a long unquoted URL had the fix confirmed in support: "the original link works but wget probably modifies or shortens the URL. Try enclosing it with quotes" (per support correspondence). Another customer working through the same issue resolved it independently: "I needed to include the url in quotation marks otherwise it wouldn't work" (per support). Always wrap a Kibot API URL in double quotes on the command line.

Credentials and rate limits

The plain-text URL list embeds a per-file token that authorises the download without a separate login, so a basic wget call needs no --user/--password. For the FTP archive side described in Data updates, FTP server, the same website email and password are the FTP credentials; support's standard guidance is "you can use your website credentials to login to ftp.kibot.com" (per support correspondence). For programmatic API calls authenticated through the historical data API, the login flow returns a session cookie that wget can preserve across calls with --save-cookies and --load-cookies.

Two rate limits apply regardless of which command-line tool is used. All requests must originate from a single IP address, and the server enforces a five-concurrent-connections-per-IP ceiling above the API's own per-server limits (see Data updates, API rate limits). Splitting a wget run across many parallel processes is the most common way to trip both limits at once; symptoms are 402 Unauthorized responses on most files or download links that temporarily stop working. The fix is to drop back to a single wget process and let -i walk the URL list sequentially, or to add --limit-rate to throttle voluntarily.

Resume and retries

Long tick-data pulls almost always benefit from making wget resumable across interrupted runs:

wget --content-disposition --trust-server-names -c --tries=20 --retry-connrefused -i downloadtext.txt

-c continues a partial file rather than overwriting it. --tries=20 retries each URL up to twenty times on transient errors. --retry-connrefused extends the retry behaviour to outright connection refusals, which matter when the server briefly load-sheds during a busy window. For very long-running pulls add --read-timeout=120 so an idle connection does not stall the entire job past a couple of minutes of silence.

For the size of file Kibot's tick archive produces (the ES tick file alone has historically run around 40 GB per support correspondence, and the broader tick set is much larger), the practical pattern is a nightly scheduled wget run with -c rather than a single 24-hour invocation: each run picks up where the last one stopped, and the resume support means a failed run does not waste the gigabytes already on disk.

Scheduling unattended pulls

Linux and macOS schedule wget through cron. A nightly catch-up against the Kibot update window (full archives are ready before 8 AM Eastern, tick history typically around 05:00 Eastern; see Data updates), targeting a local archive in the user's home directory:

# crontab -e
30 3 * * * cd /home/me/kibot && wget --content-disposition --trust-server-names -c -nc -i downloadtext.txt >> wget.log 2>&1

-nc skips any file that is already fully downloaded, so the cron job is idempotent against a partially populated directory. Combine with the package's "plain text format" link being regenerated on each refresh window and the wget run pulls only what has actually changed.

On Windows the same idea lives in Task Scheduler, but customers who want scheduled Windows downloads are usually better off with Kibot Agent's built-in scheduler; it handles credential storage, incremental refreshes against existing files, and the resume logic in one tool rather than three.

Key Takeaways

  • wget is the practical default for unattended bulk Kibot downloads on Linux and macOS, and a clean alternative on Windows when an existing shell pipeline already runs there.
  • Windows install is manual: download the Eternally Bored binary, drop it somewhere stable like C:\wget, and add that folder to PATH (GUI or one-line elevated PowerShell). Linux installs through the system package manager (apt, yum, or dnf) and macOS through Homebrew (brew install wget).
  • The canonical Kibot pattern is the plain-text URL list from the account download page fed to wget --content-disposition --trust-server-names -i downloadtext.txt; the two flags preserve the server-side filename, which would otherwise default to a query-string name.
  • Quote any Kibot API URL with query-string parameters when passing it on the command line. Unquoted long URLs trip 500 Internal Server Error responses because the shell mangles them before wget sees them.
  • Resume long pulls with -c --tries=20 --retry-connrefused --read-timeout=120. Tick files run tens of gigabytes each, so scheduled nightly runs that pick up partial files are more reliable than a single 24-hour invocation.
  • The same email and password that log into the website log into ftp.kibot.com. The per-file plain-text URLs already embed an authorisation token, so HTTP-based wget runs do not need --user/--password.
  • Stay under five concurrent connections from a single IP. Spawning many parallel wget processes is the fastest way to trigger 402 Unauthorized responses and temporary download blocks.
  • curl works as a drop-in alternative (curl -OJL mirrors the wget flags). One customer reported curl was measurably faster than wget on a slow link (per support); the choice between them is usually convenience rather than capability.
  • Data updates, the four delivery channels in context, including the plain-text URL list and where wget fits among Agent, FTP, and the API.
  • Kibot Agent, the Windows GUI that bundles download, resume, and scheduling for customers who would rather not script wget themselves.
  • Kibot Updater, the FTP-side merge tool that pairs with wget when the FTP archive is the chosen channel.
  • Kibot API, the programmatic interface that produces the per-file URLs wget consumes.
  • Server responses, context for 402 Unauthorized and the other HTTP-level errors that signal a rate-limit trip rather than a credential problem.