v0.7.0

v0.7.0 is the release where ccrawl stops needing anything else installed. Python went in v0.6.0, duckdb goes here, and what is left is one static binary that answers the bulk questions on its own. The other half of the release is about long runs: a job that takes days now leaves a record of what happened instead of whatever scrolled past on stderr.

The columnar index without duckdb

ccrawl columnar needed a duckdb binary on PATH for every one of its subcommands. When duckdb was missing it printed the SQL, which is a polite gesture and not an answer.

urls, locations, count, langs, mimes and schema are now answered by a native engine that reads the Parquet index itself over ranged HTTP, with any combination of --domain, --host, --tld, --mime, --lang, --status, --path-prefix and --subset. Nothing needs installing for it.

ccrawl columnar locations --lang vie -n 5 -o jsonl | ccrawl fetch -

It works the way a query planner does. Open a part, read the footer, pull the page index only for the columns the filters actually mention, and drop any page whose minimum and maximum rule the filter out. Because the files are sorted by url_surtkey, a --domain or --host filter also gets a prefix predicate on that column, which is the one that prunes hardest. What survives is read a column at a time, cheapest column first, so a row group the first filter empties never has the expensive columns touched at all. Deciding a part holds nothing costs about 200 KiB out of an 8.7 MiB file.

Counting the Vietnamese TLD across the 300 parts of one crawl's robotstxt subset, both engines agreeing on 240377:

Engine	Wall clock
native, -j 16	15.3s
native, default -j 8	22.1s
duckdb	38.8s

Being faster than duckdb is not the point and will not hold for every query, since duckdb is vectorized and will win where a query really does have to read a lot of rows. The point is that the no-dependency path is not a consolation prize.

--engine picks between auto, native, duckdb and print. The default is auto, and it takes native for anything native can answer whether or not duckdb is installed, so the same command does the same thing on every box. columnar query and columnar sql --run take arbitrary SQL, so those still need duckdb and always will. An engine that accepts arbitrary SQL is a database, and this is a command line tool.

Before shipping it, every subcommand was run under both engines against three live crawls and the sorted output compared byte for byte, 18 comparisons, all identical. That run earned its keep twice. It caught toInt64 zeroing every warc offset the native engine produced, because it knew the float64 duckdb's JSON gives and not the int64 a Parquet column gives, which meant 370 locations with the right filename and URL and an offset of 0. It also caught langs and mimes writing a different JSON key per engine. Neither was visible from the unit tests.

The details, including what each engine can answer, are in the new columnar engines reference page.

--source s3 reads over S3

--source s3 changed the URLs ccrawl printed and none of the ones it fetched. download.go read the path for the chosen source and then threw it away, and export.go, markdown.go and warc_cache.go each hardcoded the HTTPS mirror. Pinning a job to us-east-1 for free in-region reads therefore bought nothing at all.

Every bulk read now renders its URL through the run's source in one place: get, fetch, download, export, news, markdown export, markdown refetch, and the columnar projection behind urls publish. An s3:// URI is translated to the bucket's REST endpoint inside the existing retry loop, so the throttle, the backoff, the retry budget and the user agent are shared with the HTTPS path rather than reimplemented beside it.

The bucket is not anonymous any more. s3://commoncrawl answers AccessDenied without a signature, so this signs with SigV4 rather than leaving a hook for it. Credentials come from AWS_ACCESS_KEY_ID and friends, then the AWS_PROFILE profile in ~/.aws/credentials. Instance roles are deliberately not consulted, because a run that takes hours wants a key it was handed rather than one that expires halfway through. With no credentials the command fails on the first attempt and says what to set, instead of spending its whole retry budget on a 403 that will never turn into a 200.

There is also a one line warning on the first S3 read when the caller is outside us-east-1, since that combination pays egress for nothing. It stays quiet when it cannot tell where it is running.

TestSignS3MatchesBotocore pins the whole Authorization header against one botocore produced for the identical request under a frozen clock, so the signature is checked against an implementation nobody in this repo wrote.

Long runs write a journal

A publish run takes days, and when one finished short the only record was stderr scrollback. Every long command now writes one JSON object per event to a journal file, so the two questions worth asking afterwards, which shards failed and when the rate dropped, are a jq filter away.

Three new global flags:

Flag	Meaning	Default
--progress	text, json, or none	text on a terminal, json otherwise
--journal	Where the JSON Lines go	run.jsonl beside the ledger
--metrics-addr	Address to serve /metrics on	off

The --progress default is the useful one: run it in a terminal and you get human lines, redirect it into a log file or run it under a supervisor and you get JSON, without anybody having to remember a flag. The metrics address is bound at startup, so a busy port fails the command there rather than an hour in.

One event schema covers the shard pipelines and the streaming commands, so a single query works across all of them. markdown export and markdown refetch emit start, per-shard, tick and end events with an inflight gauge. download, export, host enrich and crawl seed report on the same schedule and in the same shape, and those four reported nothing at all before.

The ticker recomputes rate and ETA from its own clock rather than from the run stats, which only advance on a commit and so say nothing at exactly the moment you need them, when a run has stalled. fetch_failed is kept separate from failed, because a refetch shard that hit a thousand dead hosts still succeeded and folding the two together would make a healthy run look broken.

The Prometheus registry is hand rolled. client_golang would do more of it, but it is a large dependency for twelve metrics in a binary whose selling point is that it has none.

RSS reporting moved to build-tagged files, /proc/self/status on Linux and ps on macOS. It was Linux only, which meant it always read 0 on the machine most of this gets developed on.

The full event schema, the metric table and the queries worth keeping are in the run journal reference page.

The Python escape hatch is gone

v0.6.0 shipped the Go HuggingFace commit path as the default and kept CCRAWL_HF_COMMIT=python reachable for one release. That release is out, so the hatch goes, along with the embedded hf_commit.py, the code that extracted it to ~/.cache/ccrawl/ on first use, and the interpreter hunt. grep -ri python ccrawl/ cli/ finds nothing that runs an interpreter.

Also fixed

ccrawl export 'example.com/*' -n 3 exited 1 on a completely successful run. CDXStream wraps the callback's error with the page it came from, so the stop sentinel was compared against a wrapped error and never matched. It uses errors.Is now.

Install

brew install tamnd/tap/ccrawl
scoop install ccrawl

The release attaches the prebuilt archives, the deb, rpm, and apk packages, and the container image at ghcr.io/tamnd/ccrawl, and refreshes the apt and dnf repositories.