v0.5.2

v0.5.2 replaces the old seed-and-upload path with two straight mirrors that copy Common Crawl to HuggingFace shard for shard. Nothing is aggregated, deduplicated, or filtered: the rows and their order match the source, and one output Parquet maps onto one original source part. The layout is plain data/<crawl>/part-NNNNN.parquet, so a dataset browses like a folder tree rather than a Hive path.

urls publish

ccrawl urls publish -c <crawl> reads the crawl's columnar URL index and republishes it to open-index/ccrawl-urls, one shard per source part. It projects down to the URL-level columns and keeps the source row order, so part-00042 on the hub is part-00042 at the source. Pick crawls with the global -c flag: a single id, a year, the newest N, all, or a comma-separated list.

ccrawl urls publish -c CC-MAIN-2026-25
ccrawl urls publish -c CC-MAIN-2026-25 --no-push   # scan and report, upload nothing

domains publish

ccrawl domains publish streams the web-graph domain ranks top to bottom and cuts rank-ordered Zstandard Parquet shards into open-index/ccrawl-domains. The one edit to the data is convenience: the source keys each row by a reversed host string (com.example), and we un-reverse it into a plain domain (example.com). Rows stay in rank order, so part-000 holds the highest-centrality domains and rank falls as the part number rises.

Resumable from remote truth

Both pipelines are idempotent against what is already on the hub. Before a run does any work it asks the hub which shards exist, skips those, and commits only what is missing, so a killed run resumes cleanly and re-uploads nothing. Each local shard is deleted the moment its commit lands and new work pauses when free disk drops under a floor, so a whole crawl fits on a small box. A stalled run exits 75 for a supervisor to restart, and urls recount and domains recount repair drifted totals in stats.csv straight from the hub.

This release also hardens the resume probe: a rate-limited existence check used to look like "nothing is published" and re-upload shards that were already there, and it now retries the probe instead of trusting the empty answer.

Dataset cards that report the build

Each dataset card now carries a "How this dataset is built" section with live numbers for the newest release: input size, output size and compression ratio, elapsed publish wall-clock, throughput, and, for the URL mirror where the shard count is known ahead of time, an estimated completion time from the current rate. The domain mirror learns its length only when the stream ends, so it reports throughput and elapsed but is honest that there is no mid-run finish estimate.

Retired commands

The old seed, seed cc, seed publish, host dataset, and dataset commands are gone, along with the open-index/commoncrawl-urls and open-index/cc-host-dataset datasets they wrote to. ccrawl publish delete-obsolete removes those two retired repos once the new datasets are populated.

Install

brew install tamnd/tap/ccrawl
scoop install ccrawl

The release attaches the prebuilt archives, the deb, rpm, and apk packages, and the container image at ghcr.io/tamnd/ccrawl, and refreshes the apt and dnf repositories.