v0.6.0

v0.6.0 adds no new user facing feature on purpose. It makes the claims ccrawl already made true: the binary is pure Go with no runtime dependencies, the reference describes the program you installed, and the exit codes are written down.

HuggingFace uploads are pure Go

urls publish, domains publish, markdown export and markdown refetch used to shell out to uv run hf_commit.py, a Python script embedded in the binary and extracted to ~/.cache/ccrawl/ at runtime. Every dataset ccrawl published went through it, so the box needed Python, uv and huggingface_hub installed before a publish run could start.

The commit is now spoken directly by the binary. It calls preupload to find out which files the hub actually wants, uploads LFS objects in parts against the returned URLs, and posts the NDJSON commit body itself. Nothing is extracted to the cache directory and no interpreter is looked for.

The upload also got faster, which was not the goal but is welcome. Measured on real CC-MAIN-2026-30 shards against real HuggingFace repos, in two rounds with disjoint shard sets so neither path benefited from the other's uploads:

Round	Go	Python
Go first	195.6 MB in 21s, 9.3 MB/s	187.4 MB in 30s, 6.2 MB/s
Python first	201.5 MB in 21s, 9.6 MB/s	202.6 MB in 37s, 5.4 MB/s

Ten shards were committed through both paths and every LFS object id the hub reported back matched the local sha256, so the repo contents are identical either way.

Failures now come back as typed errors instead of a subprocess exit code. ErrHFRateLimited, ErrHFAuth, ErrHFQuota and ErrHFConflict let the retry loop and the stall clock tell a throttle apart from a bad token, and a quota wall now stops the run immediately rather than retrying into a wall for 45 minutes.

If something in the new path bites you, CCRAWL_HF_COMMIT=python still selects the old one for this release. It goes away in v0.7.0, so please open an issue rather than pinning to it.

The docs describe the binary that ships

Six places in the documentation described a program that was not installed. The worst was the recrawl engine guide, which documented a crawl fetch reading seeds on stdin with --delay, --workers and .seen resume files, none of which exists. The second worst was that markdown export and markdown refetch, the two pipelines most people come here for, appeared in no reference page and no guide at all.

New in the reference:

• Markdown pipelines, covering both schemas, the HuggingFace path layout, resume semantics, and how to tune the parallelism knobs.
• A guide for building a markdown corpus end to end.
• Every command group in the README table, including markdown, crawl, sched, index, api, host, content and publish.
• What --source s3 actually does today, which is choose the URLs ccrawl writes down rather than the ones it fetches.

To stop the drift coming back, scripts/docs-drift.sh builds the binary, walks --help for every command, and fails if a command is missing from the reference or a documented flag does not exist. It runs in CI on every pull request. It found two examples in the markdown help text advertising a --no-push flag that only urls publish and domains publish have, which are fixed here.

Exit codes are documented

There is now an exit codes page covering all five, with the two that are worth branching on explained properly.

Exit 3 means the command worked and matched nothing, which is a normal answer when you search for a URL that was never crawled. Exit 75 is EX_TEMPFAIL and means run it again: a publish run either stalled with no commit landing inside --max-stall, or made progress without finishing. The page includes a systemd unit with RestartForceExitStatus=75, which is what the code was designed for. A run that made no progress at all does not exit 75, so a dead source cannot spin a supervisor forever.

sched diff no longer puts every noisy host on a 24h recrawl

DiffCDX passed a harmonic rank position of 0 into CrawlTier, and 0 read as the best rank there is. Any host with a high change rate came back as tier 1 no matter how unimportant it was, which inverts the whole point of a scheduler that is supposed to crawl important things often rather than noisy things often.

A position of 0 or less now means unknown, and the tier is capped at what the change rate alone justifies. A volatile host with no rank comes back as tier 3 at best. The contract is written into the doc comment on CrawlTier and pinned by tests so the next caller does not hit it.

domains diff

ccrawl domains diff compares two web-graph domain releases already published to the dataset and reports how many domains are new in the later one, how many dropped out of the earlier one, and how many the two share. With no flags it diffs the two most recent complete releases; --from and --to pick releases explicitly. It reads only the domain column of each shard straight from the hub, so it never pulls the rank fields, and --added-out writes the new domains to a file, one per line.

Install

brew install tamnd/tap/ccrawl
scoop install ccrawl

The release attaches the prebuilt archives, the deb, rpm, and apk packages, and the container image at ghcr.io/tamnd/ccrawl, and refreshes the apt and dnf repositories.