v0.10.0

The last two releases added things. This one makes the things that were already here survive a long run, and then says out loud what each of them is.

Nothing in ccrawl fails on the happy path. It fails twelve minutes into a query over six crawls, when one index page comes back short and takes the other eleven minutes of work with it, or at hour nine of a publish when the machine runs out of memory holding a map of every URL it has seen. Those are the failures this release is about.

A search no longer loses everything to one page

A wide query is thousands of index pages, and the index truncates or refuses one often enough that it is not an edge case. Until now the first page that could not be read ended the command, and everything already streamed was thrown away.

Two things were being conflated. A page whose body stops early looks like a success: the status line and the headers arrived long ago, and the connection drops halfway through the records, so the retry loop never saw it. Pages are now buffered whole before a single record is emitted, which makes a truncated body detectable and makes retrying it safe, since no half page has been emitted downstream. A short page costs one more request, up to --retries.

A page that fails every attempt is named on stderr with its crawl and page number, and the run carries on:

search: CC-MAIN-2026-30: CDX page 252: HTTP 503, skipping the page
search: the result is incomplete, 1 index page could not be read; run it again or pass --strict to fail instead

The summary prints once at the end, so a partial result never passes for a whole one. --strict on search and export restores the old behaviour for a caller who would rather have nothing than a result with a hole in it. The library default is still strict: CDXQuery.OnPageError is nil unless you set it, so code that has not thought about partial results gets the error.

One more rule protects exit 3. A run that emitted nothing and lost part of the query exits 1, not 3, because it has not found that Common Crawl holds no captures; it failed to ask. That distinction is the whole reason exit 3 exists, and an outage wearing its clothes is worse than no exit code at all.

This one was found the honest way, by a real run dying at page 252 of a *.gov query with nothing to show for it.

Memory has a ceiling

--at, --latest-only and --dedup each held a map keyed on every URL the query touched, and a wildcard over a large domain across every crawl touches hundreds of millions of them.

A CDX response is sorted by urlkey and every capture of a URL sits in one urlkey group, so the winner can be decided while the crawl is being read. --at reduces each group as it goes and merges the per crawl runs at the end. --latest-only writes the URLs it emitted to a log in the same order and checks the next crawl against it with a cursor that only moves forward. Both stay exact at any result size.

--dedup is the exception, because payload digests arrive in no order. Past its ceiling it forgets the digests it has not seen for longest and says so, which lets a duplicate through rather than dropping a unique record.

--max-buffer is the shared budget, 5,000,000 records by default, counted across every crawl at once rather than per crawl. Past it the run spills to TMPDIR and cleans up however it exits.

One rate limit for the whole machine

--rate spaced the requests one process made, which was never the number Common Crawl saw. Running the URL publish, the domain publish and a Markdown export at once was three processes each pacing themselves politely and three times the traffic arriving at a nonprofit that serves this for free.

--global-rate is a shared budget over a lock file in the data directory. Taking a slot is a read and a write of sixteen bytes under the lock, so processes queue on timestamps rather than on the lock and the aggregate rate is exact however the requests are spread. The default is 200ms, the same as the old per process delay, so one process behaves exactly as it did and three now split the budget one of them used to take alone.

Only Common Crawl hosts draw on it. A recrawl fetching arbitrary sites is not spending Common Crawl's bandwidth, and columnar scans are exempt because they are thousands of small footer reads that a five per second budget would turn into an hour. A host where the lock file cannot be created keeps running on the per process delay and warns once, because a rate limit that silently stopped being global is the exact failure it exists to prevent.

Verify what you published

Resume checked that a path exists and recount recomputed totals, but nothing ever looked at the bytes. A truncated upload that landed as a valid looking object was invisible until a reader hit it.

publish verify reads each shard's Parquet footer over ranged requests and asks whether the file parses, whether it is the schema the dataset promises leaf column by leaf column, whether the row groups add up to the row count in the footer, and whether every column chunk sits inside the bytes the hub is holding. Then it reconciles shard count, rows and bytes against the ledger the dataset card is built from, and reports a disagreement even when every shard passes, because that means the numbers the dataset advertises are not the numbers it holds. --repair re-uploads what failed, --sample checks a subset.

A library that knows what it holds

A directory listing tells you file names and nothing else: not whether the bytes are still the ones that were downloaded, not which crawl a file came from once it moved, not what any of it costs. A WARC that lost a block to a bad disk still parses, still converts, and quietly produces wrong rows.

library.json now records every artifact with its path, crawl, kind, format, size, sha256, creation time, and the ccrawl version that wrote it. download and convert maintain it as they go, under an flock, written to a temporary and renamed so the manifest on disk is always whole. Checksums are computed as the bytes stream past rather than by reading the file back.

ccrawl library list                  what is in the library
ccrawl library du                    what it costs, per crawl
ccrawl library verify                rehash it all, report what moved
ccrawl library gc --older-than 90d   free the crawls you are done with
ccrawl library scan                  record a tree that predates this

Verified against 1.1 GB of real archives: every sha256 matched shasum -a 256, library du matched du -sh to the byte, a single flipped byte in a 65 MB WET file was caught as corrupt, and the gc dry run and the real run removed exactly the same set.

A config file, and profiles

~/.config/ccrawl/config.toml, with a [default] table for every run and named tables that --profile selects between.

[default]
workers = 8
global_rate = "500ms"

[bulk]
workers = 64
global_rate = "50ms"
crawl = "CC-MAIN-2025-33"

Precedence is flag, then environment, then profile, then [default], then the built-in default. The file is read before a single flag is registered and folded into what the flags default to, so the rule of applying a flag only when it was given produces that precedence for free. config show gains a source column, because a run that behaves oddly is nearly always a setting arriving from somewhere nobody looked.

Scope, written down

Two subsystems were sitting between a demo and a product, and the docs described the more flattering one.

The BM25 index is a reference implementation, and now says so in its own help. --input reads JSONL now rather than being declared and ignored, --workers registers, index search honours -n, and a build with no input is a usage error instead of a success over an empty index. The ceiling is measured rather than asserted: about 26 KB per document, so roughly 600,000 documents on a 16 GB machine, which is 45 of the 100,000 WET files in a crawl.

The api server is a local exploration tool. It binds 127.0.0.1 by default and warns on any address other machines can reach, naming what is missing: no authentication, no rate limiting, no request log. A partial rank table load is now a startup failure rather than a warning followed by a server that answers with half a table. /v2/health reports which stores actually loaded instead of ok from a server whose endpoints both return 503.

Both statements, and the state of every other part of the tool, are now in one table at how far each part goes.

What each command needs, and what it returns

The binary returns two exit codes that were documented nowhere: 4 when a command that pushes to HuggingFace has no token, and 8 when download cannot read the bytes. Both are documented now, in exit codes, in the CLI reference, and in ccrawl --help. The taxonomy behind them reserves 5, 6 and 7, which no ccrawl command returns, and the page says so rather than leaving a script author to write handlers for codes that never arrive.

Proving that turned up three token checks that returned a plain error and so exited 1 while the other five exited 4. All eight are classified now, and all eight were reworded.

What each command needs is a new page organised by dependency rather than by command: what runs with nothing but the binary, which of the three Common Crawl hosts each command talks to, what every DuckDB command does when there is no DuckDB, every command that wants a token and the opt out for each, what --source s3 needs and why, the disk floors, and the ports the servers bind. scripts/docs-drift.sh checks the flags on that page too, so it fails CI rather than rotting.

The tests behind all of it

internal/fakecc stands a whole Common Crawl up on localhost: the crawl list, the CDX server, the path manifests, and the WARC file those manifests point at. The pieces agree with each other, so the byte ranges the index reports really are where those records sit in the WARC, and search --locations | fetch - is a genuine round trip rather than two mocks that line up. One page 403s the first time it is asked for, and specific pages can be made to truncate or 503 on demand, which is how the page loss handling above is tested.

Coverage is floored in CI and the floors only go up. ccrawl is at 65.8 percent, cli at 50.7, and the archive packages at 74, 85 and 100. cli was at 5.8 percent two releases ago.

Not in this release

search still sends its URL substring filters after the fact rather than pushing them to the index server. The work is written and open in #112, and its whole claim is bytes over the wire, which cannot be measured while index.commoncrawl.org is refusing connections. It ships in v0.10.1 with the two runs that prove it.

Install

brew install tamnd/tap/ccrawl
scoop install ccrawl

The release attaches the prebuilt archives, the deb, rpm, and apk packages, and the container image at ghcr.io/tamnd/ccrawl, and refreshes the apt and dnf repositories.