Release notes
What changed in each ccrawl release, newest first.
What shipped in each release, newest first. Every tagged version builds the same set of artifacts: archives for Linux, macOS, Windows, and FreeBSD, Linux packages (deb, rpm, apk), a multi-arch container image on GHCR, and entries for the package managers. Binaries are pure Go, so there is nothing to install alongside them.
v0.5.3
Publish the newest domain web-graph release that actually has a ranks table, not just the newest listed.
v0.5.2
Mirror a whole crawl's URL index and the domain ranks to HuggingFace, one output shard per source shard.
v0.5.0
Pull a whole crawl's URLs into a sharded seed, then offload it to a HuggingFace dataset.
v0.4.2
Smarter retry backoff, faster columnar domain queries, and a rewritten README with a demo.
v0.4.1
Install ccrawl from Homebrew, Scoop, and the apt and dnf repositories.
v0.4.0
Reproducible WARC export, parquet output, multi-crawl selection, search parity flags, and a columnar rename.
v0.3.0
Markdown pipeline: CC WARCs to Markdown Parquet on HuggingFace, with parallel export and live refetch.
v0.2.5
Fix web-graph release lookup when commoncrawl.org uses absolute hrefs.
v0.2.4
Native HuggingFace publish with hive-partition layout and per-shard commits.
v0.2.3
Per-URL dataset shards: 20 raw CDX fields per row, no aggregation, WARC byte-range access.
v0.2.2
host dataset command: build all 262M CC hosts as partitioned Parquet shards.
v0.2.1
New guides for all v2 command groups.
v0.2.0
v2 platform: host enumeration, recrawl engine, BM25 search index, REST API, and content pipeline.
v0.1.0
The first public release of ccrawl: the full command surface, the dataset library, and the WARC/WAT/WET parser packages.