Release notes

v0.10.2 Two wrong answers that looked right. A misspelled subcommand that wrote help into your data file and exited 0, and a URL filter that ran in the wrong place and moved 1,500 times more of the index than it needed to. v0.10.1 What the docs promised. A guide that described a command nobody wrote, a dataset card naming the wrong dataset, errors and help printing flags that do not exist, JSON keys that matched nothing else, a network outage exiting 1, a flag that moved half the tree, and a handful of… v0.10.0 Durability and scope. A search survives a bad index page, memory has a ceiling, the rate limit is shared across processes, published shards get verified, and every part of the tool says how far it goes. v0.9.0 The crawler. A frontier on disk, an RFC 9309 robots parser, ISO 28500 WARC output, and ccrawl crawl run to drive all three. v0.8.0 Language filters, a choice of extractor, deduplication, coalesced ranged fetch, and a markdown export that runs off an index query instead of whole shards. v0.7.0 The columnar index no longer needs duckdb, --source s3 finally reads over S3, and long runs write a journal you can query. v0.6.0 Publishing to HuggingFace no longer needs Python, and the docs now describe the binary that ships. v0.5.3 Publish the newest domain web-graph release that actually has a ranks table, not just the newest listed. v0.5.2 Mirror a whole crawl's URL index and the domain ranks to HuggingFace, one output shard per source shard. v0.5.0 Pull a whole crawl's URLs into a sharded seed, then offload it to a HuggingFace dataset. v0.4.2 Smarter retry backoff, faster columnar domain queries, and a rewritten README with a demo. v0.4.1 Install ccrawl from Homebrew, Scoop, and the apt and dnf repositories. v0.4.0 Reproducible WARC export, parquet output, multi-crawl selection, search parity flags, and a columnar rename. v0.3.0 Markdown pipeline: CC WARCs to Markdown Parquet on HuggingFace, with parallel export and live refetch. v0.2.5 Fix web-graph release lookup when commoncrawl.org uses absolute hrefs. v0.2.4 Native HuggingFace publish with hive-partition layout and per-shard commits. v0.2.3 Per-URL dataset shards: 20 raw CDX fields per row, no aggregation, WARC byte-range access. v0.2.2 host dataset command: build all 262M CC hosts as partitioned Parquet shards. v0.2.1 New guides for all v2 command groups. v0.2.0 v2 platform: host enumeration, recrawl engine, BM25 search index, REST API, and content pipeline. v0.1.0 The first public release of ccrawl: the full command surface, the dataset library, and the WARC/WAT/WET parser packages.