v0.8.0

v0.8.0 is about the corpus rather than the tool. Every release up to here made ccrawl better at moving Common Crawl around. This one is about what comes out the other end: what language it is in, which extractor produced it, how much of it is the same page twice, and how to build a dataset out of a query result instead of out of ninety thousand shards.

The driving requirement was a recovery pass over Vietnamese: find the pages the general corpora dropped, fetch only those pages, and turn them into a Markdown dataset. That pass now runs end to end with nothing but ccrawl commands.

markdown export off a location set

markdown export took a shard range and nothing else. That is the right shape for building a corpus from a whole crawl, and the wrong shape for every question of the form "convert these particular pages". A columnar query hands back a few thousand locations scattered across a few thousand WARC files, and reading those files whole to reach them means moving something like a hundred thousand times the bytes the pages are worth.

--locations takes the location JSONL that columnar locations and index already emit, and converts exactly the records it points at with coalesced ranged GETs.

ccrawl columnar locations --crawl CC-MAIN-2026-30 --lang vie --status 200 -o jsonl \
  | ccrawl markdown export --locations - --lang vie --dedup-digest --push=false --out ./md

Everything after the fetch is the pipeline a shard export already uses: the same extractor, the same language filter, the same digest dedup, the same schema, the same ledger and resume. A recovery pass that produced a slightly different corpus from an export of the same pages would not be a recovery of anything.

3000 Vietnamese locations from CC-MAIN-2026-30, which landed in 2923 distinct WARC files:

markdown: 3000 locations in 3 parts, extracting with [email protected]
markdown: 3 committed, 0 skipped, 0 failed of 3 | 276 rows | html=26.1 MB md=1.5 MB parquet=431.6 KB | 13m4s elapsed
dedup: --dedup-digest dropped 1 duplicate payloads before conversion, 276 rows written
language: --lang vie kept 276 of 2987 documents, dropped 2711 (90.8%)
language: detected eng=1997 vie=518 rus=320 ind=50 bul=21 fra=21 deu=13 ita=6

33.8 MB came off the wire for those 3000 records. The shards holding them are 959 MB each, so the shard route would have moved about 2.8 TB for the same 276 rows.

A part is to a location run what a shard is to a full export: the unit that gets one Parquet file, one ledger entry, and one digest dedup set. --part-size cuts the stream in order, so an interrupted run resumes where the ledger says it stopped. A location that will not fetch is skipped rather than failing the part, because a recovery pass runs against an index that can disagree with the archive.

Language detection and filtering

Both Markdown pipelines now label every row with a detected language and a confidence, and --lang keeps only one of them.

ccrawl markdown export --shards 0-9 --lang vie --push=false --out ./md

Detection runs on the extracted Markdown rather than on whatever the page declared, so it describes the text in the row. Without --lang nothing is dropped and every row still carries language and language_confidence, which is what makes an unfiltered shard filterable later without extracting it again. A document with too little text to identify is dropped by --lang rather than kept, since a filtered export asks for documents known to be in one language and "we could not tell" is not that.

The run prints the drop rate and the detected mix, so a filter that threw away more than expected says so instead of quietly returning a small dataset.

This is a coarse pre-filter on a trigram identifier, not a substitute for a language specific classifier. It tells Vietnamese from Malay well enough to cut a corpus down to something worth looking at.

A choice of extractor, and WET as a source

Which engine turns a captured page into text is a corpus quality decision, so it is a flag now instead of a build time constant.

Engine	Reads	Does
h2m (default)	WARC	go-trafilatura tuned for recall, rendered as GitHub-flavored Markdown
readability	WARC	go-readability plus mdconv, the engine open-markdown-v2 shipped
raw	WARC	the whole document as Markdown, no boilerplate removal at all
wet	WET	the plain text Common Crawl already extracted, passed through unchanged

The same shard through two engines is two different corpora, and the only way to find out which suits a downstream task is to build both and compare. Every row records the engine that produced it in the extractor column as name@version, because extraction changes between releases and a name alone cannot explain why two shards built months apart disagree about the same page.

raw sounds useless and is not. Every extractor is a lossy judgement about what a page was for, and on the pages where that judgement goes wrong the output alone does not say so. Raw is the control you measure the others against.

wet reads a different manifest entirely, which --source-kind selects and which defaults to whatever the extractor needs. Asking for a pairing that cannot work is a usage error rather than a silently reinterpreted run.

Deduplication

Common Crawl captures the same payload more than once, and a corpus that does not say so trains on it more than once.

--dedup-digest skips records whose payload digest was already seen in the shard. The check runs before extraction, so a duplicate costs a hash lookup rather than an HTML parse.

Every row also carries a 64 bit simhash fingerprint of the Markdown, whether or not the flag is set, and the new ccrawl dedup command reports what a built dataset actually contains.

276 rows in 3 files
  exact duplicates            9 in 4 clusters, 5.0 kB
  near duplicates            17 in 7 clusters, 138.4 kB  (distance <= 3)
  redundant                  26  (9.4% of rows)

Near duplicates are reported rather than dropped during the run. Deciding which of two near identical copies is the good one is not a decision a converter should make on its own, and a fingerprint stored in the row lets that decision be made later, by whoever is training on it, without extracting the corpus again.

On one shard of CC-MAIN-2026-30 the digest pass dropped 83 duplicate payloads, leaving 20819 rows against the 20861 the same shard produces without the flag. An independent scan of the same WARC found 21278 HTML response records, 21195 distinct payloads, and 83 duplicates in 28 groups, an exact match with no false drops.

The near pass ignores documents under 512 bytes. A 64 bit fingerprint is decided by one vote per distinct three word shingle, so a document with fewer features than the hash has bits has a fingerprint decided by noise, and two short pages that share a phrase land a bit or two apart while having nothing to do with each other. Short documents still get a fingerprint and still cluster as exact duplicates, which is the only claim worth making about them.

Bulk ranged fetch

ccrawl fetch --batch sorts locations by file and offset, coalesces the ones that sit close together, and reads each run of them in a single ranged GET. Records that share a request are sliced back apart by their own offset and length, so the output is byte for byte what the one at a time path produces.

ccrawl columnar locations --tld vn -o jsonl | ccrawl fetch - --batch --ledger fetched.txt --dir

On 486 real records packed into 20 files the default gap turned 486 requests into 20 and finished in 6.6 seconds against 97.4 seconds for the one at a time path, a 14.7x speedup while reading 34x the bytes. That trade is the whole flag: a round trip to data.commoncrawl.org costs far more than a megabyte of transfer does. --dry-run reports both halves of it without fetching anything, so a gap can be chosen for free.

--ledger appends every finished location and skips what is already there on a rerun, so a killed run picks up where it stopped.

Negated and set valued columnar filters

The recovery pass needs ".vn hosts that Common Crawl did not label Vietnamese", which is a filter the index query could not express.

--not-lang, --not-mime, --not-status and --not-tld negate their positive counterparts and treat a missing value as a match, since a page with no language recorded is exactly the kind of page a language ID miss produces. --hosts-file and --domains-file take a set, one per line, - for stdin.

ccrawl columnar locations --tld vn --not-lang vie --status 200 -o jsonl
ccrawl columnar locations --not-tld vn --hosts-file hosts.txt --status 200 -o jsonl

Both engines answer them, and --print shows the SQL either way.

Also fixed

markdown export --push=false wrote every Parquet part and then deleted it. The committer reclaims disk once a batch is safe on the hub and never checked that the batch had gone to the hub, so a local only run reported its rows and left an empty directory. The local copy is only redundant once it is somewhere else, so the delete is conditional on the push now. refetch had the identical line and got the identical fix.

A data race in the fetch engine's HTTP trace callbacks, surfaced by the first ccrawl test to drive a real socket under -race. ConnectDone runs on whichever goroutine the transport dialled from, and those dials outlive the request that started them, so it could write the peer address into a Result the caller already owned. Fixed upstream in ami v0.3.1.

Install

brew install tamnd/tap/ccrawl
scoop install ccrawl

The release attaches the prebuilt archives, the deb, rpm, and apk packages, and the container image at ghcr.io/tamnd/ccrawl, and refreshes the apt and dnf repositories.