Skip to content
ccrawl

CLI

Every command and subcommand, with the flags that matter.

ccrawl <command> [subcommand] [flags]

Run ccrawl <command> --help for the full flag list on any command.

Commands

Command What it does
crawls List, resolve, and inspect the monthly crawls
search Query the URL index (CDX) for captures of a URL or pattern
get Fetch what Common Crawl captured for a URL
fetch Retrieve WARC records by explicit location, or from stdin
export Write matching captures into WARC files with provenance
download Download whole archive files for a crawl
paths List the archive file paths for a crawl
parse Decode a local WARC/WAT/WET file into records
extract Pull text, links, title, or Markdown from a captured page
content Live-fetch content signals: text, outlinks, quality
news Work with the continuous CC-NEWS dataset
columnar Query the columnar Parquet index
markdown Build Markdown Parquet datasets from CC WARCs and publish them
rank Look up host and domain ranks from the web graph
host Enumerate and enrich hosts from the CC web graph
urls Mirror the Common Crawl URL index to a HuggingFace dataset
domains Mirror the Common Crawl domain ranks to a HuggingFace dataset
publish Maintenance for the published Common Crawl datasets
crawl Recrawl engine: seed, fetch, and write WARC output
recrawl Recrawl a published dataset, streaming the work list rather than queueing it
sched Recrawl scheduling: tier assignment and differential CDX analysis
index Build and query a local BM25 full-text search index
api Start the v2 REST API server
db Build and query a local DuckDB database
convert Convert WARC/WAT/WET archives to Parquet or JSONL
library Inspect, verify, and collect the dataset library
dedup Report exact and near duplicate documents in a Markdown Parquet dataset
stats Show the shape of a crawl: file counts per archive kind
serve Serve the operations over HTTP as NDJSON
mcp Run as an MCP server over stdio
config Show resolved configuration and data paths
cache Inspect and clear the on-disk cache
version Print the version and exit

crawls

Subcommand Does
crawls list List the monthly crawls, newest first
crawls latest Print the newest crawl ID
crawls resolve <ref> Resolve a year or latest to a crawl ID
crawls info [id] File counts per archive kind for a crawl

crawls info and stats are the same command under two names, so they count the same kinds and answer with the same rows: crawl, kind, files. Both take --kinds to narrow the list, and both honour -o, so crawls info -o csv writes CSV the way every other read command does. The one difference is how you name the crawl: crawls info takes it as a positional argument, stats reads -c, and either one falls back to the configured crawl when you leave it out. A kind whose manifest could not be fetched comes back with files of -1 rather than being dropped, so a row is never silently missing.

ccrawl search <url|pattern> [flags]

A trailing /* matches everything under a path. Filters: --mime, --status, --from, --to, --filter. URL filters: --url-contains, --url-not-contains. Pick the capture closest to a date with --at (for example --at 2023-06). Order with --sort newest|oldest. Estimate the size of a result instead of listing it with --estimate. Shaping: --fields, --template, -o, -n. Alias: cdx.

Where the filtering happens

--status, --mime, --lang, --filter, --url-contains and --url-not-contains are all sent to the index server, which drops the rows before they leave it. The two URL substrings become regex filters on the url field, and they are checked again on the way past here, which costs nothing and means a server that filtered differently cannot widen the result.

Everything that has to compare one record against another stays on this machine: --at, --latest-only and --dedup need the whole result set, and the index has no way to express them.

--explain prints the split, along with the request the index server actually answers and the bytes the run read from it:

$ ccrawl search '*.senate.gov' --url-contains /budget --explain -o url
search: 1 crawl: CC-MAIN-2026-30
search: the index server answers https://index.commoncrawl.org/CC-MAIN-2026-30-index?filter=~url%3A.%2A%2Fbudget.%2A&matchType=domain&output=json&url=senate.gov
search: pushed to the server: --url-contains /budget
search: applied here: --url-contains /budget (again, on what the server sent)
search: read 49.4 KB from the index (50576 bytes)

Paste that URL into curl and the rows that come back are the rows the command reads. Run the same query twice, once with --no-push-filters, and the two byte counts say what the push saved. That query is 10 index pages over CC-MAIN-2026-30 and 77 of its rows hold /budget. Pushed, it reads 50,576 bytes. With --no-push-filters it reads about 75 MB to find the same rows, which is roughly 1,500 times as much for the same answer.

--no-push-filters keeps the URL substrings off the wire and filters here instead. It is an escape hatch, not a tuning knob: the only reason to reach for it is an index server whose filtering disagrees with ours, and the cost of it is every unwanted row downloaded before it is dropped. The export command takes the same flag, for the two URL substring filters of its own.

The cost is worse than the bytes suggest, and it is worth knowing before reaching for the flag. The index truncates a large page under load and closes the connection as if it had finished, so a run that moves 75 MB to answer a query loses a few rows to that and cannot tell you which. The *.senate.gov query above was run twice with --no-push-filters on 2026-08-18: it read 70.3 MB and returned 75 rows, then read 77.3 MB and returned a different 75, each time missing two the pushed run found. Pushed, the same query returned all 77 every time. Filtering on the server is the difference between a complete answer and a nearly complete one, not only a faster one.

Memory over a wide query

--at, --latest-only and --dedup each have to remember something about every URL the query touches, and a wildcard over a large domain across every crawl touches hundreds of millions of them. The three of them share one budget, --max-buffer, which is how many records they hold in memory before the run starts writing to a temporary file in TMPDIR instead. The default is 5,000,000 records, which is a few hundred megabytes, and the temporary files are removed when the command exits however it exits.

Flag Description
--max-buffer Records --at, --latest-only and --dedup hold in memory before spilling to disk (default 5000000)

A CDX response is sorted by urlkey and every capture of a URL sits in one urlkey group, so --at and --latest-only stay exact past the budget: they reduce each group as it goes by and merge the per crawl runs afterwards. --dedup is the exception, because payload digests arrive in no order at all. Past the budget it forgets the digests it has not seen for the longest and says so on stderr, which costs you a duplicate that gets through rather than a unique record that gets dropped.

One thing does change past the budget. --at normally sorts its result newest first; when the result itself will not fit in the buffer it comes out in index order instead, and the command says so on stderr. That sort gets the same budget again, so --at can hold twice --max-buffer records at the moment it hands the result over.

A page the index will not serve

A wide query is thousands of index pages, and the index truncates or refuses one often enough that a long run used to end on it and throw away everything that had already arrived.

A page whose body stops early is read again, up to --retries times, so a truncated response costs a second request rather than the records it was carrying. A page that fails every attempt is named on stderr with its crawl and page number, and the run carries on with the next page:

search: CC-MAIN-2026-30: CDX page 252: HTTP 503, skipping the page
search: the result is incomplete, 1 index page could not be read; run it again or pass --strict to fail instead

The summary line is printed once at the end of the run, so a partial result never passes for a whole one. Pass --strict to get the old behaviour, where the first page that cannot be read ends the command.

Flag Description
--strict Fail the run if an index page cannot be read, rather than skipping it

export takes --strict and reports the same way, for the same reason.

get

ccrawl get <url> [flags]

Content flags (pick one): --text, --markdown, --links, --headers. With none, prints the raw HTTP response body.

fetch

ccrawl fetch [-] [flags]

Locate a record with --file, --offset, --length, or stream JSONL locations on stdin with -. Content flags: --body (default), --text, --markdown, --links, --headers, --meta. Write one file per record with --dir and --out-dir.

Batch mode

One record per HTTP request is fine for a few thousand locations and hopeless for a few million. --batch sorts the locations by file and offset, coalesces the ones that sit close together, and reads each run of them in a single ranged GET. Records that share a request are sliced back apart by their own offset and length and parsed individually, so the output is byte for byte what the one at a time path produces.

Flag Description
--batch Coalesce nearby records in the same WARC file into shared ranged GETs
--gap Coalesce records at most this many bytes apart (default 1 MiB)
--max-span Never read more than this in one GET (default 16 MiB)
--order input or file: emit in the order given or the order on disk (default file)
--ledger File of finished locations, to skip on a resume
--lookahead Ranged GETs allowed to run ahead of the writer (default 64) 
ccrawl columnar locations --tld vn -o jsonl | ccrawl fetch - --batch --ledger fetched.txt --dir

Choosing a gap

--gap is the price you are willing to pay in wasted bytes to save one request. Every merge drops one round trip and reads the hole between the two records, so the flag is that trade written down as a number. Add --dry-run and nothing is fetched: the grouping is pure arithmetic on the locations, so it reports both halves of the trade and you can try a few values for free.

$ ccrawl fetch - --batch --dry-run --gap 65536 < locations.jsonl
1000000 locations in 612078 requests, 1.6x fewer than one at a time; 10.1 GB read for 1.6 GB of records, 6.3x amplification

$ ccrawl fetch - --batch --dry-run < locations.jsonl
1000000 locations in 101238 requests, 9.9x fewer than one at a time; 116.0 GB read for 1.6 GB of records, 71.5x amplification

Those are real numbers from a million robots.txt locations, which is close to the worst case: tiny records scattered ten to a WARC file. The 71x amplification looks alarming and is still the right call, because a round trip to data.commoncrawl.org costs far more than a megabyte of transfer does. On 486 real records packed into 20 files the default gap turned 486 requests into 20 and finished in 6.6 seconds against 97.4 seconds for the one at a time path, a 14.7x speedup while reading 34x the bytes. Lower --gap when bandwidth is what you are paying for, raise it when latency is. --max-span is the separate ceiling that stops one dense file from becoming a single enormous read.

--order file streams: groups are written out in the order they sit on disk and no more than --lookahead of them are ever in flight. --order input has to put back an ordering the grouping destroyed, so it holds finished records in memory until their turn comes, and a single slow group early in the input will hold everything behind it. Use it when something downstream is lining the output up against the input, and leave it alone otherwise.

Pass --ledger and every finished location is appended to that file, flushed after each group. Rerunning the same command with the same ledger skips what is already in it, so a killed run picks up where it stopped rather than starting over. The ledger is one filename@offset per line, so it is greppable and safe to trim by hand.

export

ccrawl export <url-or-pattern|-> [flags]

Run a query, pull each matching capture, and write them into one or more .warc.gz files. Each file opens with a warcinfo record carrying provenance (the tool and version, the prefix, and the exact command line), so the output is self-describing. Pass a URL or wildcard pattern to run a query, or - to read location records (filename, offset, length) as JSONL on stdin, exactly what search --locations and columnar locations produce.

Naming: --prefix, --subprefix. Rotation: --size (bytes, default 1 GB). Destination: --out-dir. Provenance: --creator, --operator. Query filters mirror search: --match, --from, --to, --status, --mime, --lang, --filter. URL filters: --url-fgrep, --url-fgrepv, which go to the index server as regex filters the same way search sends its own, and --no-push-filters to keep them here instead.

ccrawl export example.com/* --prefix example
ccrawl search example.com --locations | ccrawl export - --prefix example

download

ccrawl download <kind|-> [flags]

Kinds: warc, wat, wet, robotstxt, non200responses, cc-index, cc-index-table. Use - to read paths on stdin. --out sets the directory, --flat drops the source tree, -j/--workers sets concurrency.

paths

ccrawl paths <kind> [flags]

Kinds: warc, wat, wet, robotstxt, non200responses, cc-index, cc-index-table, segment. --kinds lists them. -o url prints full URLs.

parse

ccrawl parse <file|-> [flags]

Force the format with --format (warc|wat|wet). Filters: --type, --status, --mime, --lang, --url. Content flags: --links, --text, --markdown, --meta.

extract

Subcommand Does
extract title <url> The page title
extract text <url> Readable plain text
extract markdown <url> HTML converted to Markdown
extract links <url> Outbound links

content

Live-fetch a URL and compute content signals. Unlike extract, these commands use the v2 crawler config (10 MB body limit, brotli support, redirect following).

Subcommand Does
content extract <url|-> Clean text, title, description, canonical URL, language, word count
content outlinks <url|-> Outbound links as (source, url, host) rows, without anchor text
content quality <url|-> Quality signals: word count, title length, main-content flag, spam score, parked detection
content lang <url|-> The language the markdown pipelines would detect, with the confidence and the text it judged 
ccrawl content extract https://golang.org/
ccrawl content quality https://example.com/ -o json
ccrawl content outlinks https://news.ycombinator.com/ -n 20
ccrawl content lang https://vnexpress.net/ -o json

All four take - in place of the URL and read a list from stdin, one URL per line or JSONL with a url field, which is what search, columnar and crawl fetch produce:

ccrawl content quality - -o jsonl < seeds.txt
ccrawl search 'example.com/*' -n 100 -o jsonl | ccrawl content quality - -o jsonl

Over a stream a URL that cannot be fetched is named on stderr and the rest of the list carries on. A run that scored nothing because every URL failed exits 1, and empty stdin exits 3. Content signals has the field tables and the pipelines.

content lang

content lang runs the same identifier markdown export --lang applies, on one URL at a time, so a document that was kept or dropped can be asked about directly.

$ ccrawl content lang https://vnexpress.net/ -o json
{
  "url": "https://vnexpress.net/",
  "language": "vie",
  "confidence": 1,
  "cc_language": "vi",
  "chars": 24196,
  "sample": "Báo tiếng Việt nhiều người xem nhất Thứ hai, 3/8/2026 ..."
}

language is detected in the extracted Markdown, not in the raw HTML and not read off the page's own lang attribute, because the Markdown is what the pipelines keep and filter on. cc_language is what the page declares, printed next to it so a disagreement is visible instead of silent. sample is the text the identifier actually saw, truncated; when an answer looks wrong it is almost always the input that is wrong.

news

CC-NEWS is the one Common Crawl dataset with no index of any kind: no CDX, no columnar table, nothing but the WARC files and a list of their names. So finding one publisher's articles in a month means reading the month, which is around 350 files of roughly a gigabyte each. news publish builds the missing index and puts it on HuggingFace, and news search reads that index when it exists.

Subcommand Does
news list List CC-NEWS files for --year/--month
news download Download CC-NEWS files
news search <host> Find a publisher's articles in a month, from the index or by scanning
news publish Build the missing CC-NEWS index and mirror it to a HuggingFace dataset 
ccrawl news search bbc.co.uk --year 2026 --month 7
ccrawl news search bbc.co.uk --year 2026 --month 7 -o jsonl | ccrawl fetch - --text
ccrawl news search bbc.co.uk --year 2026 --month 7 --no-index   # force the full scan

It answers from the published index when that month is indexed, and falls back to streaming the archives when it is not, saying on stderr which one it did. A month that is indexed but still building is searched from the part that is published, and the shortfall is reported on stderr rather than passed off as the whole month.

The rows are the CDX columns, so the output reads like index output and -o jsonl pipes straight into fetch -.

Flag Meaning
--year CC-NEWS year
--month CC-NEWS month
--repo HuggingFace dataset repo to read the index from (default: open-index/ccrawl-news, or CCRAWL_NEWS_REPO)
--no-index Skip the index and scan the archives, which is the old behaviour
--limit Stop after this many matches
--workers Concurrent readers (0 picks a default from CPU count)

news publish

ccrawl news publish --months 2026/07
ccrawl news publish --months 2026/07,2026/06 --commit-every 16
ccrawl news publish --months 2026/07 --files 4 --no-push   # index a slice, upload nothing

It streams every WARC file of the month, records the byte span of each stored response, and writes one Parquet shard per source file. The archives are never written to disk: they are decompressed, indexed, and dropped as they stream, so a run holds one output shard per worker and nothing else. A stream that dies partway resumes at the last complete record rather than restarting the file.

Reading a month is the cost of this dataset, a few hundred gigabytes of transfer, once. HF_TOKEN (or HUGGINGFACE_TOKEN) must be set to push.

Flag Meaning
--repo HuggingFace dataset repo (default: open-index/ccrawl-news, or CCRAWL_NEWS_REPO)
--months Comma list of months to index, as YYYY/MM
--files Index only the first N WARC files of each month (0 indexes the month)
--commit-every Shards per HuggingFace commit
--workers Stream-and-index workers (0 picks a default from CPU count)
--private Create the dataset repo private
--keep Keep local shards after commit instead of deleting them
--min-free-gb Pause new work when free disk is under this many GB
--max-stall Restart the run (exit 75) after this long with no progress
--no-push Index and stage but skip the upload

The news index reference has the schema and the queries.

columnar

Aliases: table, athena.

Subcommand Does
columnar urls Matching URLs
columnar locations Record locations, ready for fetch
columnar count Count of matching captures
columnar langs Breakdown by content language
columnar mimes Breakdown by MIME type
columnar sql Build the SQL from the filter flags and print it
columnar query <sql> Run raw SQL (ccindex is the source)
columnar schema The columns of the index

Filters: --domain, --host, --tld, --mime, --status, --lang, --path-prefix, --subset. Negated filters: --not-tld, --not-mime, --not-lang, --not-status. A row where the column is missing counts as a match, so --not-lang vie returns the captures Common Crawl never labelled as well as the ones it labelled something else. Set filters: --hosts-file and --domains-file read one value per line, skipping blank lines and # comments, and turn the whole list into a single query. The list prunes the same way a single --host or --domain does, so it costs one pass over the index rather than one pass per value. Engine: --engine (auto|duckdb|native|print). See the columnar engines for what each one can answer and how fast.

ccrawl columnar count --tld vn --not-lang vie
ccrawl columnar urls --hosts-file hosts.txt --not-tld vn -o url

markdown

Build Markdown Parquet datasets from Common Crawl WARC files and commit them to a HuggingFace dataset repo.

Subcommand Does
markdown export Convert the HTML Common Crawl captured to Markdown Parquet
markdown refetch Re-fetch every URL in a shard live, then convert to Markdown Parquet 
ccrawl markdown export --shards 0 --push=false --out ./md
ccrawl markdown export --shards all --parallel 4 --commit-batch 10 --repo org/name
ccrawl markdown refetch --shards 0-9 --fetch-workers 400 --repo org/name

Both take --shards (N, N-M, N,M, or all), resume from a ledger at <out>/.committed, and need HF_TOKEN unless --push=false. The markdown reference has both schemas, the HuggingFace path layout, and the tuning flags.

Choosing an extractor

--extractor picks the engine that turns a captured page into the text in the row.

Engine Reads Does
h2m (default) WARC go-trafilatura tuned for recall, rendered as GitHub-flavored Markdown
readability WARC go-readability extraction plus mdconv, the engine open-markdown-v2 shipped
raw WARC the whole document as Markdown, no boilerplate removal at all
wet WET the plain text Common Crawl already extracted, passed through unchanged 
ccrawl markdown export --shards 0 --extractor readability --push=false --out ./md
ccrawl markdown export --shards 0 --extractor raw --push=false --out ./md
ccrawl markdown export --shards 0 --source-kind wet --extractor wet --push=false --out ./md

Which engine you use is a corpus quality decision, so it is a flag and not a build time constant. The same shard through two engines is two different corpora, and the only way to find out which one suits a downstream task is to build both and compare. Every row records the engine that produced it in the extractor column, as name@version, because extraction changes between releases and a name alone cannot explain why two shards built months apart disagree about the same page.

raw keeps the nav bars and the footer, which sounds useless and is not. Every extractor is a lossy judgement about what a page was for, and on the pages where that judgement goes wrong the output alone does not say so. Raw is the control you measure the others against.

--source-kind picks the manifest a run reads: warc takes warc.paths.gz and extracts the HTML itself, wet takes wet.paths.gz and uses the text Common Crawl already extracted. It defaults to whatever the extractor needs, so it rarely has to be given. The two are not interchangeable, since a WET file holds no HTML and a WARC holds no pre-extracted text, and asking for a pairing that cannot work is a usage error rather than a silently reinterpreted run:

$ ccrawl markdown export --shards 0 --source-kind wet --extractor h2m
ERROR  Extractor h2m reads warc shards, so it cannot be used with --source-kind wet.

markdown refetch takes --extractor too, but only the WARC engines: a live fetch returns HTML, and there is no Common Crawl text to pass through.

WET is much cheaper than any of the extractors, since the text is already there and the files are a fraction of the size of the WARCs. It is also somebody else's extraction decision, boilerplate and all, and it is plain text rather than Markdown, so headings and links are gone.

Language filtering

Both subcommands label every row with a detected language and can keep only one of them.

Flag Default Does
--lang (off) Keep only documents detected as this ISO 639-3 language, for example vie
--min-lang-confidence 0.8 Confidence a document has to clear for --lang 
ccrawl markdown export --shards 0 --lang vie --push=false --out ./md
ccrawl markdown export --shards 0 --lang vie --min-lang-confidence 0.9 --push=false --out ./md

The language is detected in the extracted Markdown, so it describes the text in the row rather than whatever the page declared. Without --lang nothing is dropped and every row still carries language and language_confidence, which is what makes an unfiltered shard filterable later without extracting it again.

A document with too little text to identify is dropped by --lang rather than kept: a filtered export asks for documents known to be in one language, and "we could not tell" is not that. The run prints both the drop rate and the detected mix, so a filter that threw away more than expected says so:

language: --lang vie kept 21482 of 43110 documents, dropped 21628 (50.2%)
language: detected vie=21482 eng=17903 unknown=2011 zho=884 ...

--lang cannot be combined with markdown refetch --fetch-only, since fetch-only never produces the Markdown the identifier reads.

This is a coarse pre-filter. A trigram identifier tells Vietnamese from Malay well enough to cut a corpus down to something worth looking at, and it is not a substitute for a language specific classifier.

Deduplication

Flag Default Does
--dedup-digest false Skip records whose payload digest was already seen in this shard 
ccrawl markdown export --shards 0 --dedup-digest --push=false --out ./md

The check runs before extraction, so a duplicate costs a hash lookup rather than an HTML parse, and the scope is one shard rather than the whole run. On one shard of CC-MAIN-2026-30 it dropped 83 duplicate payloads, which an independent scan of the same WARC confirmed exactly.

Every row also carries a simhash fingerprint whether or not the flag is set. Near duplicates are reported by ccrawl dedup rather than dropped during the run, because deciding which of two near identical copies is the good one is not a decision a converter should make on its own. See Deduplication.

Converting a location set

markdown export --locations reads exactly the records a stream of index locations points at, instead of downloading whole shards.

Flag Default Does
--locations (off) Convert the records in this JSONL location stream, - for stdin
--part-size 50000 Locations per Parquet part
--gap 1 MiB Coalesce records closer together than this many bytes into one ranged read
--max-span 16 MiB Cap on the size of one coalesced ranged read 
ccrawl columnar locations --crawl CC-MAIN-2026-30 --lang vie -o jsonl \
  | ccrawl markdown export --locations - --lang vie --dedup-digest --push=false --out ./md
ccrawl markdown export --locations missed.jsonl --part-size 10000 --push=false --out ./md

This is the recovery pass: a columnar query picks out the pages you are missing, and the export turns those pages and nothing else into Parquet, with the same extractor, language filter, dedup, and schema a full export uses. Reading whole shards to reach a few thousand scattered pages would move something like a thousand times the bytes those pages are worth, so the records are fetched with ranged GETs and the neighbours coalesced.

A part is to a location run what a shard is to a full export: the unit that gets one Parquet file, one ledger entry, and one digest dedup set. The stream is cut in order, so an interrupted run resumes where the ledger says it stopped. Locations that will not fetch are skipped rather than failing the part, because a recovery pass runs against an index that can disagree with the archive.

--locations bypasses --shards, --source-kind, and the manifest fetch entirely, and it cannot be combined with the wet extractor, since WET files have no record offsets to point at. See Converting a location set instead of whole shards.

rank

Subcommand Does
rank domain <domain> Rank of a registered domain
rank host <host> Rank of a host
rank top Top-ranked hosts
rank all Stream every host in a rank table, most central first

All four read the newest web-graph release without being told where it is. --graph <release-id> pins a release, the way the host commands take it, and --table <url> points at a table directly and skips the lookup. rank domain reads the domain ranks and the other three read the host ranks, which are separate tables with separate positions in them: wikipedia.org is domain 14 and host 864 in the same release. The newest release for a domain lookup is the newest one whose domain table is published, which is not always the newest release, because a release is listed as soon as its host tables land.

rank top and rank all take --tld to filter by TLD. The rank table is sorted by harmonic centrality, so rank all -n 1000 is the top 1000 hosts without any sorting on your side.

ccrawl rank host wikipedia.org
ccrawl rank all --tld com -n 1000
ccrawl rank all --graph cc-main-2026-mar-apr-may -o jsonl > hosts.jsonl

host

Enumerate and enrich hosts from the CC web graph. All subcommands accept --graph <release-id> to pin a specific web-graph release (default: latest).

Subcommand Does
host top Top hosts by harmonic centrality, streamed from the rank table
host get <hostname> Enriched profile for one host
host vertices Stream the vertex ID to hostname mapping
host degrees Compute in-degree and out-degree from edge files (~7.7 GB)
host cdx Aggregate CDX statistics per host via DuckDB
host enrich Full enrichment pipeline: rank + degrees + CDX

A host record only carries what was measured

Every host subcommand emits the same record, and each one fills a different part of it. host top and host get read the rank table, so they answer with the rank signals. The degree counts come from host degrees or host enrich, and the CDX counts from host cdx, host enrich, or host get --cdx.

A count nothing measured is left out rather than written as zero, because zero is an answer and this is the absence of one:

$ ccrawl host top -n 1 -o jsonl
{"host":"www.facebook.com","host_rev":"com.facebook.www","tld":"com","registered_domain":"facebook.com","harmonic_pos":1,"harmonic_val":34375268,"pagerank_pos":3,"pagerank_val":0.0055143537769673755}

In -o csv and -o table the column is still there and the cell is blank, so the shape of the output does not change with what a run happened to measure. A count that was taken and came out zero prints as 0, which is the whole point of the distinction.

host top

ccrawl host top -n 20 -o table
ccrawl host top --graph cc-main-2026-mar-apr-may -n 1000 -o jsonl > top1k.jsonl

host get

ccrawl host get golang.org -o json

host vertices

ccrawl host vertices --graph cc-main-2026-mar-apr-may -n 5

host degrees

Streams all edge files to compute per-host in/out-degree. Requires ~7.7 GB of edge data.

ccrawl host degrees --graph cc-main-2026-mar-apr-may -n 100 -o jsonl

host cdx

Runs a DuckDB GROUP BY url_host_name over the columnar Parquet index. Without --filter this scans ~184 GB of Parquet.

ccrawl host cdx --filter example.com -o json
ccrawl host cdx -n 100 -o jsonl
Flag Meaning
--filter Restrict to one host (url_host_name)

host enrich

Runs all enrichment phases in sequence. Phases 3 and 4 are opt-in because they require large data transfers.

ccrawl host enrich -n 20
ccrawl host enrich --graph cc-main-2026-mar-apr-may -n 100
ccrawl host enrich --degrees --cdx -o jsonl > enriched.jsonl
Flag Meaning
--graph Web-graph release ID (default: latest)
--degrees Phase 3: compute in/out-degree from edge files (~7.7 GB)
--cdx Phase 4: aggregate CDX statistics via DuckDB (~184 GB)

urls

Mirror the Common Crawl columnar URL index to a HuggingFace dataset, one output Parquet shard per original source part. Nothing is aggregated, deduplicated, or filtered: the rows and their order match the source, projected down to the URL-level columns. The run is idempotent from remote truth, so shards already on the hub are skipped and each local shard is deleted right after it commits.

Subcommand Does
urls publish Mirror the URL index to a HuggingFace dataset, shard for shard
urls recount Repair drifted URL and byte totals in stats.csv from the hub

urls publish

ccrawl urls publish -c CC-MAIN-2026-25
ccrawl urls publish -c 2 --commit-every 32
ccrawl urls publish -c CC-MAIN-2026-25 --no-push   # scan and report, upload nothing

HF_TOKEN (or HUGGINGFACE_TOKEN) must be set to push.

Flag Meaning
--repo HuggingFace dataset repo (default: open-index/ccrawl-urls, or CCRAWL_URLS_REPO)
--commit-every Shards per HuggingFace commit (default 16)
--workers Download-and-convert workers (0 picks a default from CPU count)
--whole Download each part whole before reading (fallback for range-hostile mirrors)
--private Create the dataset repo private
--keep Keep local shards after commit instead of deleting them
--min-free-gb Pause new downloads when free disk is under this many GB
--max-stall Restart the run (exit 75) after this long with no progress
--no-push Scan and stage but skip the upload

domains

Stream the web-graph domain ranks top to bottom and republish them as rank-ordered Parquet shards on a HuggingFace dataset. The one edit to the data is un-reversing the source host key (com.example becomes example.com); rows stay in rank order, so part-000 holds the highest-centrality domains.

Subcommand Does
domains publish Mirror the domain ranks to a HuggingFace dataset, in rank order
domains recount Repair drifted release totals in stats.csv from the hub
domains diff Count domains added, removed, and shared between two published releases

domains publish

ccrawl domains publish
ccrawl domains publish --no-push   # scan and report, upload nothing
Flag Meaning
--repo HuggingFace dataset repo (default: open-index/ccrawl-domains, or CCRAWL_DOMAINS_REPO)
--commit-every Shards per HuggingFace commit
--private Create the dataset repo private
--keep Keep local shards after commit instead of deleting them
--min-free-gb Pause new work when free disk is under this many GB
--max-stall Restart the run (exit 75) after this long with no progress
--no-push Scan and stage but skip the upload

domains diff

Compare two web-graph domain releases already published to the dataset and report how many domains are new in the later release, how many dropped out of the earlier one, and how many the two share. It reads only the domain column of each shard straight from the hub, so it never downloads the rank fields. With no ids it diffs the two most recent complete releases in the dataset, older against newer.

ccrawl domains diff
ccrawl domains diff --from cc-main-2026-mar-apr-may --to cc-main-2026-apr-may-jun
ccrawl domains diff --added-out new-domains.txt
Flag Meaning
--repo HuggingFace dataset repo (default: open-index/ccrawl-domains, or CCRAWL_DOMAINS_REPO)
--from Older web-graph release id (default: second-newest published)
--to Newer web-graph release id (default: newest published)
--added-out Write the domains new in the later release to this file, one per line
--workers Concurrent shard readers (0 picks a default from CPU count)

publish

Maintenance for the published Common Crawl datasets.

Subcommand Does
publish verify Check that the published shards are readable, complete, and the schema they claim
publish delete-obsolete Delete the superseded first-generation dataset repos

publish verify

Publishing only ever asks whether a shard's path exists, so an upload that was cut off part way through leaves an object the resume path skips forever, and nothing notices until somebody reads the dataset and gets an error out of a Parquet library. publish verify reads each shard's footer over ranged requests and asks what publishing never asks: does the file parse, is it the schema the dataset promises, do the row groups add up to the row count the footer claims, and does every column chunk sit inside the bytes the hub is holding. The totals are then reconciled against the stats.csv ledger the dataset card is built from, and a disagreement is reported even when every shard passes, because it means the numbers the dataset advertises are not the numbers it holds.

ccrawl publish verify -c CC-MAIN-2026-25
ccrawl publish verify -c CC-MAIN-2026-25 --sample 64
ccrawl publish verify -c CC-MAIN-2026-25 --repair
ccrawl publish verify --graph cc-main-2026-mar-apr-may --json

Verifying the 300 shard CC-MAIN-2026-25 crawl reads 269 MB against a dataset holding 150.5 GB, which is 0.175 percent of it, so a full check costs about as much as a listing.

Flag Meaning
--repo HuggingFace dataset repo (default: the dataset the unit belongs to)
--graph Verify a web-graph release of the domains dataset instead of URL crawls
--sample Rows to decode from each shard's last row group (0 reads the footer alone)
--workers Shards checked at once (0 picks a default from CPU count)
--repair Rebuild and re-upload the shards that fail
--no-push With --repair, rebuild locally but skip the upload
--json Print the report as JSON

Each shard comes back ok, or missing, unreadable, truncated, schema, empty, corrupt, or no-access. The last one is not a verdict on the data: the shards are read with plain ranged GETs because the published datasets are public, so a repo that will not serve them reports no-access rather than being called corrupt. The exit status is non-zero when a shard fails and --repair was not passed.

--sample decodes rows out of each shard as well as reading its footer, which is the only way to catch a page whose bytes are wrong rather than missing. It reads a page of every column instead of a footer, so it costs a great deal more than the default check and is worth running on a crawl you have a specific reason to distrust.

--repair works on the URL dataset, where a shard is the projection of exactly one source part and can be rebuilt on its own from the part that made it. A domain shard is a cut of one sequential rank stream, so rebuilding one means reading the source up to it: publish verify --graph reports the bad shards and leaves the rebuild to ccrawl domains publish.

publish delete-obsolete

Delete the obsolete dataset repos that the ccrawl-urls and ccrawl-domains datasets replaced. It removes open-index/cc-host-dataset and open-index/commoncrawl-urls, and asks for confirmation unless --yes is passed.

ccrawl publish delete-obsolete          # prompt before deleting
ccrawl publish delete-obsolete --yes    # delete without prompting

crawl

Recrawl engine commands for seeding and fetching live URLs.

Subcommand Does
crawl seed Generate seed URLs from the web-graph rank table
crawl fetch <url> Crawl a single URL with robots.txt checking and content digest
crawl run Run a crawl from a seed file, with a resumable frontier and WARC output
crawl status Show daily crawl budget allocation across the five recrawl tiers

crawl seed

Streams the rank table and emits one seed URL per host. Use --max-tier to restrict to high-priority hosts: tier 2 is roughly the top million by harmonic rank, tier 5 is everything. Tier 1 wants a change rate above 0.8 and a seed carries no measured change rate, so --max-tier 1 is refused rather than answered with nothing.

ccrawl crawl seed -n 100 -o table
ccrawl crawl seed --max-tier 2 -n 1000000 -o jsonl > seeds.jsonl
ccrawl crawl seed --graph cc-main-2026-mar-apr-may --max-tier 3 -n 5000000
Flag Meaning
--graph Web-graph release ID (default: latest)
--max-seeds Maximum hosts to emit (default 10 000 000)
--max-tier Skip hosts with tier higher than this (2-5, default 5 = all; 1 is unreachable from a seed)

crawl fetch

Fetches one URL with the v2 crawler config: polite user-agent, brotli support, redirect following (up to 5 hops), 10 MB body limit, SHA-1 digest.

ccrawl crawl fetch https://golang.org/ -o json
ccrawl crawl fetch https://example.com/ --robots -o json
ccrawl crawl fetch https://example.com/ --warc-dir warc/
Flag Meaning
--robots Check robots.txt before fetching
--warc-dir Write the fetch to a WARC file in this directory

With --warc-dir the fetch is archived as an ISO 28500 WARC/1.0 file: a warcinfo record, then a request and response pair linked with WARC-Concurrent-To, with WARC-Block-Digest and WARC-Payload-Digest as sha1: base32, WARC-IP-Address, and WARC-Truncated: length when the body cap trips. The stored headers describe the stored body, so a decoded or dechunked response gets a rewritten Content-Length and loses the encoding headers that no longer apply. The path written is reported in the record as warc_file.

crawl run

Drives a whole crawl from a seed file: the frontier hands out URLs in priority order, one host at a time, robots.txt is fetched once per host and enforced, every fetch is written to WARC, and outlinks go back in the queue up to --max-depth.

ccrawl crawl seed -n 100000 -o jsonl > seeds.jsonl
ccrawl crawl run --seeds seeds.jsonl --out warc/ --state crawl.db --max-pages 100000 -j 64
ccrawl crawl run --seeds - --out warc/ --state crawl.db --max-depth 2 --same-host
Flag Meaning
--seeds Seed file: crawl seed JSONL or one URL per line, - for stdin
--out Directory to write WARC files into (empty fetches without archiving)
--state Frontier state file, so a run resumes after a restart
--delay Minimum spacing between two requests to the same host, default 1s, 0 for none
--max-depth How far from a seed to follow links (default 0, the seeds only)
--max-pages Stop after this many fetches (0 = no limit)
--same-host Stay on the hosts the seeds named
--no-robots Do not check robots.txt
--robots Check robots.txt, which is already the default
--warc-size Rotate to a new WARC file past this many bytes
--prefix File name prefix for the WARC output
--shard Which partition of the seed list this process takes, 0-based
--shards How many machines are splitting the seed list, default 1

The frontier in --state is the whole resume story: a run that is killed leaves its queue, its politeness clocks and its seen set on disk, and the next run over the same file picks up where the last one stopped rather than refetching what is done. Politeness is per host and it is the longer of --delay and the host's own Crawl-delay, so raising --workers adds hosts in flight and never adds requests to one host. --delay 0 means no spacing, which is for a benchmark against a server you own and not for the open web.

That per host delay is also what sets the rate, and it is worth knowing which number to reach for. Throughput is roughly the number of distinct hosts in flight divided by the delay, so a crawl over 200 hosts at the default second runs at about 200 pages per second no matter how many workers it has, and adding workers to a crawl of one host does nothing at all. Measured against a local server: 200 hosts at the default settings gave 140 pages per second, and the same binary with the delay off sustained 8234 pages per second at 32 workers. Past a few dozen workers the frontier's lock starts costing more than it returns, 8234 per second at 32 workers falling to 4333 at 128, so more workers than hosts is not a way to go faster.

crawl run uses a client of its own rather than the one --rate and --global-rate configure. Those two exist to be polite to data.commoncrawl.org, which is one host serving bulk files to everybody, and a crawl of unrelated sites has no business drawing on that budget. It also could not: the Common Crawl delay is per process rather than per host, so robots.txt would be fetched five hosts a second however many workers were running.

--shard and --shards split one seed list across several machines that never talk to each other. Give all three servers the same file and pass --shard 0, --shard 1 and --shard 2 with --shards 3, and each one keeps a third of the list and skips the rest. The filter runs as the seeds are read, so a URL a machine does not own never reaches its frontier and three servers keep a third of the state each rather than three full copies of it.

The partition key is the registered domain, taken from the public suffix list, and that is the part worth understanding before you use it. A crawler keeps one politeness clock per host, and that clock only means anything if the host belongs to one process. Hash the URL instead and a busy site's pages scatter across all three machines, each of which waits its own second while the site sees three requests a second and every machine believes it is behaving. Keying on the registered domain keeps a site on one machine, and it keeps a.example.co.uk with b.example.co.uk, which usually are one server behind one budget.

The split is stable across machines and across runs, so a restart on server 2 picks up the same third it had before, and it is even enough to be worth nothing further. Measured over all 121 091 933 domains in the open-index/ccrawl-domains release, three shards land within 0.005 percent of even, seven within 0.030 percent and thirty two within 0.159 percent. Three shards is 40 365 223, 40 361 867 and 40 364 843 domains, a spread of 3356 across forty million.

ccrawl crawl run --seeds domains.txt --state crawl.db --shard 0 --shards 3   # server1
ccrawl crawl run --seeds domains.txt --state crawl.db --shard 1 --shards 3   # server2
ccrawl crawl run --seeds domains.txt --state crawl.db --shard 2 --shards 3   # server3

Shards are numbered 0 to --shards minus 1, and a run that names a partition outside that range stops with a usage error rather than quietly crawling nothing.

crawl status

Prints the daily page budget across the five recrawl tiers assuming 10 000 pages/s sustained throughput.

ccrawl crawl status -o table

recrawl

Recrawl a work list that is already published, streamed out of Parquet rather than loaded into a frontier.

Subcommand Does
recrawl run Walk a published dataset and fetch every URL in it, resumable from a two number checkpoint
recrawl publish Commit closed capture shards to a HuggingFace dataset as they land, with the ledger and card

recrawl run

Reads a published dataset a part at a time, fetches every URL in it, and keeps its place in a checkpoint of a part number and a row offset. robots.txt is fetched once per host and enforced, each host gets one request per --delay, and every fetch is written out as a capture row or a WARC record.

ccrawl recrawl run --from domains --out captures/ --state recrawl.json
ccrawl recrawl run --from urls --dir data/CC-MAIN-2026-25 --out captures/ --state recrawl.json --shard 0 --shards 3
ccrawl recrawl run --from open-index/ccrawl-domains --column domain --format warc --max-pages 1000
Flag Meaning
--from Published dataset to recrawl: domains, urls, or a dataset repo ID
--dir Directory of parts inside the dataset, default the newest release published
--column String column holding the work, domain or url
--out Directory to write output files into (empty fetches without archiving)
--format Output format: parquet rows with the body inline, or warc, default parquet
--state Checkpoint file, so a killed run resumes where it stopped
--delay Minimum spacing between two requests to the same host, default 1s, 0 for none
--max-pages Stop after this many fetches (0 = no limit)
--robots Check robots.txt before fetching, off by default on a recrawl
--no-extract Store the body without rendering it to text and Markdown
--extractor Engine that renders a page to Markdown: h2m, readability, or raw, default h2m
--extractors How wide the extract pool is, separately from --workers, default the core count
--shard-size Rotate to a new output shard past this much payload
--prefix File name prefix for the output files
--writers Output files open at once, each with its own encoder, default 1
--batch Work items fetched between checkpoints, default 2000
--queue-mb Megabytes of finished work that may wait for the sink, default 128
--shard Which partition of the work list this process takes, 0-based
--shards How many machines are splitting the work list, default 1
--dns-lookups How many DNS lookups may be in flight at once, default an eighth of --workers, floored at 16 and capped at 128
--robots-timeout Budget for one host's robots.txt fetch, retries included, default 10s
--profile-dir Write a CPU and a heap profile into this directory, off by default
--profile-name Name the profile files, so two runs can be told apart and diffed

robots.txt is not checked on a recrawl unless --robots is given, and this is the one default in the tool that is worth reading twice. On a link crawl the frontier keeps coming back to hosts it already knows, so the check is paid once per host and amortised over every page that host gives up. On a recrawl of a domain corpus every row is a different host, so it is a second request for every single page, and measured on the live domain list at 256 workers it was 45 percent of the worker time. Worse than the cost is what it was buying: about a third of those hosts never answered at all, so the run sat out the whole ten second budget and then threw the row away, and RFC 9309 reads a host that cannot be asked as a host that said no. Turning it on with --robots gets the full RFC 9309 behaviour back, unchanged, and a run with it off says so on its own summary line rather than leaving a reader to guess from a command line somebody typed a week ago.

--from domains and --from urls are shorthands for open-index/ccrawl-domains and open-index/ccrawl-urls with the right column already chosen. A full repo ID works too, and then --column has to name the column, because there is no guessing what an arbitrary dataset calls its URLs. Leave --dir off and the newest release in the repo is used, which is worked out by asking the dataset what it holds rather than by keeping a list in the binary. A domain is turned into its homepage, since a domain is not a URL and the homepage is what a domain level recrawl means.

--format parquet is the default because the output of a recrawl is meant to be published, and a dataset is rows. Each fetch becomes one row with the body, the request head and the response head inline, in ami's captures schema column for column, so a query written against an ami capture file reads one of these and the other way round. The columns are compressed with zstd columnar rather than one page at a time, and on a real sample of 398 homepages, 139 MB of bodies and heads, that is 22.8 MB against the 25.5 MB the same bytes take gzipped a record at a time the way a WARC stores them. Eleven percent is not the reason to do it and it is worth saying so plainly: the reason is that a reader can count status codes or pull one column without touching the bodies at all, and the compression is a small bonus on top. A 304 is stored as a row with unchanged set and no body, which over a corpus where most pages do not move between crawls is the difference between a dataset and a second copy of one.

Every HTML page is also rendered to Markdown and to plain text as it is fetched, into markdown, text, title, word_count, language, language_confidence, simhash and extractor. Those columns are appended after ami's, so a reader written against the older shape reads a newer file unchanged, and the names match open-markdown-v3 where they overlap so a query written for that dataset runs against a capture shard. Rendering happens in the worker that fetched the page, while the body is still in memory and before the write lock is taken. It is free in the only sense that matters here, which is that the run is waiting on the network and the machine is not: measured against the live domain list at 256 workers, extraction held 0 percent of the pool while fetching held 18 and robots held 39. The alternative is a second pass over the published corpus, and at 363 million domain homepages and 6.3 billion indexed URLs that is a few hundred terabytes read back to look at bodies that were in memory a moment earlier. A page that is not HTML is left alone rather than fed to an HTML extractor, and extractor being empty is how a reader tells a page nobody tried to render from a page that rendered to nothing. --no-extract turns it off, and a WARC run never pays for it at all, since a WARC record has nowhere to put a Markdown column. --format warc writes ISO 28500 the way it always did, and it is still the right answer when the output is going into an archive somebody will read with archive tools.

--shard-size is how much uncompressed payload goes into one output file before the writer starts the next one, and it defaults to half a gigabyte. Every shard is finished with its own footer, so it is readable on its own and can be uploaded and deleted while the run is still going, which is what lets a hundred day run publish from day one instead of at the end. Shards are opened by the first row that goes into them, so a run that stops cleanly leaves no empty file behind for the publish step to find and upload. A run that is killed leaves the shard that was open unsealed, and an unsealed shard has no footer and does not open, which is exactly why the checkpoint did not advance past it and why the next run refetches it. A shard is sealed between batches rather than in the middle of one, so it overshoots the size by up to a batch of pages, and that is deliberate: it is what makes the end of a shard and the end of a batch the same place, which is the only place a checkpoint can sit. Setting it very small is a false economy, since columnar compression is paid for by having many similar pages in the same file and a shard holding a handful of rows gets almost none of it.

--writers is how many shards are open at once, each with its own encoder, its own buffer and its own goroutine. The default of one is right for any run where the network is the slowest thing, which is most of them, and the timing line at the end of a run is what says otherwise: it prints how much of the wall clock the writer was busy for, and a figure in the nineties means the pool is waiting on the sink and more workers will not help. That is what a wide pool on a fast link ends up looking like once the fetches are cheap, and it is the case this flag is for. Rows go round the writers as they arrive, so the parts fill at the same rate, and when one of them reaches --shard-size all of them seal together. Sealing together is not tidiness, it is what keeps the checkpoint moving: a checkpoint may only advance when everything behind it is readable, so parts that rotated on their own would have to be caught empty at the same instant for that to ever be true, and at fleet speed it never would be. The cost is more files of slightly uneven size and more memory held in encoder buffers, about two gigabytes of resident memory going from one writer to four on a run at 256 workers, so this is a knob to raise one step at a time against the timing line rather than a number to set high and forget. Measured on the live domain list on a loaded box, where the writer was busy 88 percent of the run, going from one writer to four took it from 10.6 pages a second to 29.0 and dropped the share of the pool queueing to write from 36 percent to 13. Measured on the same box two hours later, where the writer was busy 65 percent, the same change was worth six percent, which is noise. That is the flag working correctly in both cases: it buys back time the writer was spending waiting, and a writer that was not waiting has none to give back.

--extractors sets the width of the extract pool, which is a separate pool from the fetch workers and answers to a different limit. Fetching waits on the far end, so it wants to be hundreds wide and costs almost nothing while it waits, and rendering waits on nothing at all. Rendering used to happen in the fetch worker, which was right while the pool was narrow and stopped being right as it widened: measured on server2 at 96 workers, a run reported fetching at 44 percent of the pool and extracting at 27, which is 27 percent of ninety six workers doing a job six cores can do. The default is four times the core count, clamped to --workers. The core count is the obvious answer and it was measured and it is wrong: on server2 the same run went 32.5 pages a second at 6 extractors, 35.2 at 12 and 40.3 at 24, because these are shared machines and our runnable goroutines are what win a share of the cores against the other tenants. The clamp to --workers is there because a pool that can render faster than the run can fetch is spare capacity by definition. The end of a run prints what the pool did with itself, as a share of its own width times the wall clock rather than of the fetch pool's, and the three shares in it are the three answers: rendering near the whole pool wants more of the machine, waiting for the sink wants --writers, and idle means the fetch line is the one to read. A run writing WARC or given --no-extract has no extract pool at all, and prints no line about one.

--queue-mb bounds how much finished work may be waiting for the sink, counted as the bodies and the rendered columns it is holding. It is in megabytes because that is the unit the machine's free memory is read in, and it is what the run is promising about its own memory between the pool and the disk. The bound used to be a count of items and a count cannot do the job, since a body on this corpus runs from a kilobyte to the 10 MB cap and the same 256 items is 76 MB on an average stretch and 2.5 GB on a bad one. A run at 288 workers and 6 writers reached 3.6 GB resident on a box with 2 GB free and was killed for it, having been told nothing about a limit it was nowhere near in items. The end of a run prints the bound next to the most that was ever held against it, and the pair is what makes either number readable: a peak well under the bound means the queue never made a worker wait, and a peak sitting on the bound means the sink is the ceiling, which --writers is for and a larger queue is not. A run that renders has two queues rather than one, since there are two stages between the fetch and the disk, and the budget is split between them: what the flag promises is what the run holds in total, and giving each stage the whole figure would cost twice what it said.

--profile-dir writes a CPU profile and a heap profile for the run, and --profile-name decides what they are called. The timing line answers which phase of the pool held the time, which is the right first question and stops being enough as soon as the answer is a phase rather than a knob. A pool spending 55 percent of itself extracting says extraction is the cost and does not say which of the passes inside it is, and a run holding 4.5 GB resident says nothing at all about where, because the timing line does not measure memory. The name matters because go tool pprof -base diffs two profiles and a diff is what a change is worth judging on, so two runs being compared have to be told apart on disk rather than overwriting each other. Profiling costs a few percent and writes tens of megabytes, so it is off unless asked for and no fleet unit sets it. The heap profile is taken as soon as the run stops and before anything is closed, since the encoders hold their buffers until a shard is sealed and a profile written after that has lost the largest thing in the run.

The reason this is a separate command rather than crawl run with a different seed file is the frontier. A frontier earns its keep on a discovery crawl: the queue is not known ahead of time, outlinks arrive as the crawl goes, and a crash has to leave something resumable behind. A recrawl has none of those properties, because the list is already published, deduplicated and sorted, and writing it into SQLite in order to read it back is copying a list in order to walk it. That copy costs about 135 bytes a URL, so one server's third of a threefold recrawl of a monthly crawl is roughly 283 GB of database before a single page is fetched, on machines with 2.8 GB and 6.6 GB free. It does not fit, it does not nearly fit, and the frontier's admit rate already falls fourfold between two million and five million rows on the way to not fitting.

So recrawl run holds one part open and a fixed read buffer, and its memory is the same on the first row and the billionth. The parts are listed once from the dataset and read in order, so nobody has to keep a part count in step with the next release and no part name has to be guessed at. That listing is not a nicety. The published corpora do not agree on how wide a part number is written, part-000.parquet in the domain ranks and part-00000.parquet in the URL index, and a run that built the name from a counter asked for a file that was not there and read the 404 as the end of the dataset. A part that is listed and then cannot be fetched is an error now, and so is a release with no parts, because both used to look exactly like a finished work list. Only the one column the work list needs is read, which is what makes streaming a part cheap: parquet fetches that column's chunks and leaves the rest of the file alone.

The checkpoint in --state is a part number, a row offset and the identity of what they point into, and it is a few hundred bytes whether the work list has a thousand rows or six billion. It is written only after the bytes it accounts for are readable again, and it is written to a temporary file and renamed over the old one, so a kill at any moment leaves either the old checkpoint or the new one and never half of either. What readable again means depends on the format, and the difference is worth stating. A WARC file is durable wherever it is fsynced, so a WARC run checkpoints at every batch. A kill costs the batch that was being worked through plus whatever the pool had in the air behind it: the buffer between the reader and the workers, the item each worker is holding, and the rows already fetched and queued for the writer. At --batch 2000 --workers 256 that is around 2800 rows, and replaying them costs duplicate rows rather than missing ones, which is the direction the checkpoint is built to fail in. A Parquet file is not readable at all until its footer is written, so the only durable position in one is the end of a shard, and a Parquet run checkpoints at shard boundaries and a kill costs at most the shard that was open. That is the price of the publishing format and --shard-size is the flag that bounds it: at 250 pages a second and half a gigabyte of payload a shard closes about every forty seconds. Either way it replays rather than skips, which is the property that matters: a batch cut short is not checkpointed at all, because a checkpoint past rows that were never fetched would skip them silently and nobody would ever find out. A run that stops between batches is the other case, and there the shard is sealed and the checkpoint written before the run exits, whether it stopped at the end of the work list, at --max-pages, or on a signal that landed on the boundary. A stop is the last chance to make the open shard readable, so holding it back for a fuller shard means holding it back forever. A checkpoint written by a different dataset or a different shard is refused rather than resumed, since its row offset points at nothing here.

--shard and --shards split the work list across machines that never talk to each other, on the same registered domain key crawl run uses and for the same reason. Rows a machine does not own are counted and dropped as they stream by, so the row offset means the same thing on every machine in the fleet and the checkpoints are comparable across them.

Like crawl run, the pages and robots.txt go through a client of their own rather than the one --rate and --global-rate configure. Those two are a budget for data.commoncrawl.org, one host serving bulk files to everybody, and applying a per process delay meant for one host to a million unrelated sites is not politeness, it is a queue five hosts a second long. The work list itself is read from HuggingFace, which is a bulk host, and that side does use the ordinary client and its budget.

recrawl publish

Watches the directory recrawl run writes into and commits each shard to a dataset repo as it closes, refreshing the ledger and the dataset card in the same commit, then deleting the local file. It is meant to run alongside the crawl, not after it.

ccrawl recrawl publish --dir captures/ --kind domains --server server1 --shard 0 --shards 3 --watch 30s
ccrawl recrawl publish --dir captures/ --kind urls --server server2 --shard 1 --shards 3 --state recrawl.json --watch 30s
ccrawl recrawl publish --dir captures/ --no-push
Flag Meaning
--dir Capture directory to publish from, the same one recrawl run writes into
--repo Dataset repo to publish to, default the open-index repo for --kind
--kind Which recrawl this is, domains or urls, default domains
--server Name of this machine, default the hostname
--shard Which partition of the work list this machine took, 0-based
--shards How many machines are splitting the work list, default 1
--state The crawl's checkpoint file, read so the card can report progress
--commit-every Shards per HuggingFace commit, default 4
--watch Keep watching the directory and publish shards as they close, 0 makes one pass
--keep Keep local shards after commit instead of deleting them
--private Create the dataset repo private
--no-push Stage and report but skip the upload

--kind domains and --kind urls publish to open-index/ccrawl-recrawl-domains and open-index/ccrawl-recrawl-urls. They are two repos rather than one because a domain recrawl and a URL recrawl finish on completely different schedules, and a card that averaged the two would describe neither.

The reason this is a second process rather than a stage inside recrawl run is that a run measured in months has to survive the publisher being restarted, reconfigured or pointed somewhere else without stopping the fetch. The two share a directory and one rule: a .parquet in it is a whole shard and nothing else is. A shard being written is named .parquet.tmp and is renamed only once its footer is on the platter, so the publisher can never pick up a file whose footer is missing, and a crash leaves a .parquet.tmp that the checkpoint never advanced past.

Each machine writes exactly one ledger file, ledger/<server>-shard<i>of<n>.csv, and never touches another machine's. That is not a tidiness preference. The hub has no compare-and-set on a path, so if the fleet shared one stats.csv, read it, added its rows and wrote it back, whoever committed second would write back a copy without the first one's numbers in it, nothing would error, and the counts would quietly go backwards. One writer per file makes two machines committing at the same instant touch different files, so there is nothing to lose. The card is generated from the union of whatever ledger files are on the hub at the time, which makes it derived rather than authoritative: a card written from a snapshot that missed a row someone else had just committed is corrected by the next commit from any machine, and until then it undercounts rather than overcounts.

Shards are named data/<server>-shard<i>of<n>-<hash>.parquet, where the hash is the first twelve hex digits of the file's sha256. Naming by content rather than by a counter is what makes republishing idempotent. The crawl's own file numbering restarts at zero on every run and the files are deleted once they are committed, so a counter would either collide with a shard already on the hub or publish the same rows twice under two names. With a content hash the existence check catches a duplicate before any of its bytes are uploaded, which is what a publisher killed between the commit landing and the local delete needs.

Resume asks the hub what is published rather than trusting local state, the same way the url and news publishers do. On start the publisher downloads its own ledger row and carries on from it, so a wiped staging directory or a move to a new disk does not restart the file count or the card's totals at zero halfway through a crawl.

With --watch, the publisher stops on its own once the crawl's checkpoint says the work list is walked out and the directory it was draining is empty. Without that the fleet would need a second signal to shut the publisher down, and a forgotten publisher polling an empty directory for a week is exactly the kind of thing nobody notices.

HF_TOKEN (or HUGGINGFACE_TOKEN) has to be set to push. --no-push runs everything except the upload, including the ledger and card generation, so the part most likely to be wrong is the part being rehearsed.

sched

Recrawl scheduling commands. sched diff requires DuckDB on PATH.

Subcommand Does
sched assign Assign crawl tiers to hosts by harmonic rank and change rate
sched diff Compare two crawls and compute per-host content change rates

sched assign

ccrawl sched assign -n 20 -o table
ccrawl sched assign --graph cc-main-2026-mar-apr-may --change-rate 0.5 -o jsonl
Flag Meaning
--graph Web-graph release ID (default: latest)
--change-rate Assumed change rate for all hosts (0-1, default 0.5)

Tier assignment:

Tier Recrawl interval Criteria
1 24 h harmonic rank <= 100 K and change rate > 0.8
2 3 days rank <= 1 M and change rate >= 0.5
3 7 days rank <= 5 M and change rate >= 0.2
4 30 days rank <= 10 M
5 on-demand everything else

sched diff

Joins two CDX Parquet indexes on URL and compares content_digest to compute per-host change rates. Requires DuckDB on PATH. Scans ~368 GB of Parquet (184 GB per crawl).

ccrawl sched diff --crawl-a CC-MAIN-2026-17 --crawl-b CC-MAIN-2026-21 -n 20
ccrawl sched diff --crawl-a CC-MAIN-2026-12 --crawl-b CC-MAIN-2026-17 -o jsonl > changes.jsonl
Flag Meaning
--crawl-a Older crawl ID
--crawl-b Newer crawl ID

index

Build and query a local BM25 full-text search index over a JSONL page corpus. This is a reference implementation with a corpus ceiling of a few hundred thousand documents, not a production search engine; the search index guide has the measured numbers.

Subcommand Does
index build Build a BM25 inverted index from JSONL documents or a list of URLs
index search <query> Query the index; results ranked by BM25 score

index build

Reads JSONL documents from --input, or fetches and extracts the pages named by --urls, tokenizes them, and writes a BM25 inverted index with per-document length normalization. Each --input line is a JSON object with a url and the text to index, and optionally a title and a language, which is the shape ccrawl parse writes for a WET file. The language key can be language, which is what a file written by hand tends to say, or content_language, which is what ccrawl parse wet -o jsonl writes; a file written by a ccrawl older than v0.10.1 says ContentLanguage and is still read. The index directory contains terms.dat, postings.dat, forward.jsonl, and stats.dat, and a rebuild replaces all four. One of --input or --urls is required; without either the command exits 2 rather than writing an empty index.

ccrawl index build --dir /data/idx --input docs.jsonl
ccrawl parse file.warc.wet.gz --lang eng -o jsonl | ccrawl index build --dir /data/idx --input -
ccrawl index build --dir /data/idx --urls https://golang.org/,https://pkg.go.dev/ -o json
Flag Meaning
--dir Directory to write the index into (default: ~/data/ccrawl/index)
--input JSONL file of documents to index, or - for stdin
--urls Comma-separated URLs to fetch and index
-j, --workers Fetch concurrency for --urls (default 8)

Queries the local index using BM25 scoring with per-document length normalization. Query terms are ORed: a document matches if it holds any of them, and the ones holding more score higher. A query that matches nothing exits 3.

ccrawl index search "golang web server"
ccrawl index search "machine learning" --dir /data/idx -n 20 -o json
Flag Meaning
--dir Index directory to search (default: ~/data/ccrawl/index)
-n, --limit Documents to return (default 100)

api

Start the v2 HTTP REST API server. This is a local exploration tool with no authentication, no rate limiting, no request log and no pagination, so it binds loopback by default and warns if pointed anywhere else; see the API server guide. The host store is loaded from the web-graph rank table on startup (top 1 M hosts) and a load that fails is fatal. Full-text search is available when --index-dir points to a built index, and answers 503 without it.

GET /v2/host/{host}       host profile from the rank table
GET /v2/hosts?tld=&n=     top N hosts, optional TLD filter
GET /v2/search?q=&k=      BM25 full-text search (requires --index-dir)
GET /v2/health            liveness, and which stores are loaded

ccrawl api
ccrawl api --addr 127.0.0.1:9090 --index-dir /data/idx
Flag Meaning
--addr Listen address (default 127.0.0.1:8080)
--index-dir Path to a built inverted index directory

db

Subcommand Does
db load Load matching index records into local DuckDB
db sql <query> Run SQL against the local database
db shell Open an interactive DuckDB shell
db path Print the database file path

db load takes the same filter flags as table.

convert

ccrawl convert <file|dir> [flags]

--to parquet|jsonl (default parquet). -O/--out sets the output file or directory. --markdown converts HTML bodies on the way.

library

ccrawl library list
ccrawl library du
ccrawl library verify
ccrawl library gc --older-than 90d
ccrawl library scan

The dataset library is the tree --library downloads into and processes from, ~/notes/ccrawl by default and moved with --library-dir or CCRAWL_LIBRARY. It is separate from the data dir so scratch state and the files you keep never mix, and library is how you find out what is in it.

library.json at the root of the tree records every artifact: its path, crawl, kind, format, size, sha256, when it was written, and which version of ccrawl wrote it. Every command that materialises into the library updates it, and the checksum for a download is computed as the bytes stream past rather than by reading the file back. Concurrent runs take a lock on library.lock for the read-change-write, so two ccrawl processes filling one library do not lose each other's records.

Subcommand Does
library list List the artifacts the manifest records
library du Report library size, per crawl
library verify Rehash every artifact and report what does not match
library gc Delete artifacts older than a cutoff
library scan Record what is on disk into the manifest

library list

Flag Default Does
--kind all Only this kind: warc, wet, wat, and so on
--format all Only this format: raw, parquet, jsonl

-c narrows to one crawl and -n caps the rows.

library du

--by crawl|kind|format picks the grouping, crawl by default. A total row follows the groups when there is more than one, as a row rather than a footer so it survives -o json and a pipe.

library verify

Reads every artifact and compares it against the manifest. Four kinds of trouble are reported: missing for a file that is gone, resized for one whose size changed, corrupt for one whose bytes changed, and untracked for a file on disk the manifest has never heard of. The first three exit 1, so library verify can gate a publish run; an untracked file is reported but is not a failure, since library scan is the fix.

Flag Default Does
--quick false Check existence and size only, no rehash
--kind all Only this kind

library gc

Deletes artifacts and drops them from the manifest. It needs something to select on: --older-than, -c, or --kind. --older-than takes 30d, 8w, or any Go duration such as 12h.

It is a dry run unless you pass --yes, and the dry run prints exactly the list the real run deletes. Directories the collection emptied are removed with it.

library scan

Walks the tree, hashes what it finds, and writes the manifest. This is how a library built by a ccrawl that predated the manifest, or one you copied files into, comes under management. It reads every byte the first time and is a no-op after that: an artifact already recorded at the same size is left alone unless you pass --rehash.

Only files that fit the library layout are recorded, <crawl>/<kind>/<file> for a raw archive and <crawl>/<format>/<kind>/<file> for processed output, so a README you left in the tree is not mistaken for a corrupt artifact.

dedup

ccrawl dedup <parquet-file|dir>... [flags]

Reports duplicate documents in a dataset written by markdown export or markdown refetch. It reads and prints, it never rewrites the input, so it is safe to point at a published dataset.

Flag Default Does
--distance 3 Hamming distance between fingerprints that still counts as a near duplicate, 0 to 64
--top 10 How many of the largest clusters to list, 0 for none
--json false Emit the report as JSON instead of a table 
ccrawl dedup ./md
ccrawl dedup ./md --distance 6 --top 20
ccrawl dedup ./md/part-000.parquet ./md/part-001.parquet --json

20,861 rows in 1 files
  exact duplicates          165 in 113 clusters, 139.7 kB
  near duplicates            24 in 23 clusters, 171.0 kB  (distance <= 3)
  redundant                 189  (0.9% of rows)
  no fingerprint              2  (too short to hash, left alone)

Exact clusters are documents with identical Markdown. Near clusters are grouped by the simhash column, and a file written before that column existed is fingerprinted on the fly, so an older dataset still works.

Only three columns are read, which is why a 20k row shard takes well under a second. Raising --distance widens the net and chains clusters together, so 6 finds more real template duplicates and also more junk. Documents under 512 bytes are left out of the near pass because a 64 bit fingerprint over that little text is decided by noise. See Deduplication for what the numbers mean and how they were measured.

serve

Serve the record-stream operations over HTTP as NDJSON. Every kit operation gets an endpoint, so anything the CLI can list, the server can stream.

ccrawl serve --addr :8080
ccrawl serve --addr :8080 --allow-writes
Flag Meaning
--addr Listen address (default :8080)
--allow-writes Expose the write operations, which are hidden by default

serve is the generic operation server; api is the purpose-built v2 REST API with the host store and search.

mcp

Run as an MCP server over stdio, exposing the same operations as tools to an MCP client.

ccrawl mcp

There are no command-specific flags. The global flags apply, so -c, --data-dir, and --no-cache all set the server's defaults.

version

ccrawl version              # the one-line form
ccrawl version --short      # just the version number
ccrawl version -o json      # the same six fields as data

The default line is six facts glued into a sentence: ccrawl 0.10.1 (commit 664360e, built 2026-08-13, darwin/arm64, go1.26.5). Pass -o and you get them separately, which is what a CI job wants when it needs one of them and would otherwise write a regular expression:

ccrawl version -o json | jq -r .commit

-o auto, which is the default, keeps the sentence whether the output is a terminal or a pipe. Every other command switches to JSONL when piped; this one does not, because that line is what ccrawl version has always printed and asking for a format is how you say you want data instead.

config

ccrawl config show

config show prints every setting a run resolved, the value it ended up with, and where that value came from. The source column is the reason to run it: a run that behaves oddly is nearly always a setting arriving from somewhere you did not look, and workers 7 config [bulk] ends that search in one line.

key          value                        source
crawl        CC-MAIN-2026-30              flag --crawl
workers      7                            config [bulk]
retries      2                            config [default]
user_agent   my-crawler/1                 env CCRAWL_USER_AGENT
timeout      2m0s                         default
profile      bulk                         flag --profile
config_file  ~/.config/ccrawl/config.toml derived from config_dir

The config file

ccrawl reads ~/.config/ccrawl/config.toml if it is there. CCRAWL_CONFIG_DIR names the directory outright, and XDG_CONFIG_HOME moves it the usual way. There is no file by default and nothing needs one.

The [default] table applies to every run. Any other table is a profile, and --profile <name> layers it on top of [default] for that run.

[default]
workers = 8
global_rate = "500ms"

[bulk]
workers = 64
global_rate = "50ms"
library_dir = "/data/ccrawl"

[polite]
global_rate = "5s"
retries = 8

ccrawl --profile bulk markdown build -c 2026-30
ccrawl --profile polite search '*.gov' --limit 1000

CCRAWL_PROFILE selects a profile from the environment, for a shell or a systemd unit that should run everything one way.

Precedence is flag, then environment, then profile, then [default], then the built-in default. A profile cannot undo an export in the shell that started the run, and a flag on the command line always wins.

The settings, with the environment variable that beats each one:

Setting Env What it sets
crawl CCRAWL_CRAWL Default crawl, same values as -c
source CCRAWL_SOURCE https or s3
data_dir CCRAWL_DATA_DIR Root data directory
cache_dir CCRAWL_CACHE_DIR Cache directory, follows data_dir when unset
library_dir CCRAWL_LIBRARY Dataset library root
db_path CCRAWL_DB_PATH Local DuckDB file
workers CCRAWL_WORKERS Concurrency
rate CCRAWL_RATE Per-process delay between requests
global_rate CCRAWL_GLOBAL_RATE Host-wide gap between Common Crawl requests
timeout CCRAWL_TIMEOUT Per-request timeout
retries CCRAWL_RETRIES Retry attempts
backoff CCRAWL_BACKOFF Base wait before the first retry
backoff_max CCRAWL_BACKOFF_MAX Cap on a single retry wait
user_agent CCRAWL_USER_AGENT User agent sent to Common Crawl
urls_repo CCRAWL_URLS_REPO HuggingFace dataset for urls publish
domains_repo CCRAWL_DOMAINS_REPO HuggingFace dataset for domains publish
news_repo CCRAWL_NEWS_REPO HuggingFace dataset for news publish and news search
collinfo_endpoint CCRAWL_COLLINFO_ENDPOINT Where the crawl list comes from
data_endpoint CCRAWL_DATA_ENDPOINT Where manifests, WARC files and the columnar index come from
cdx_endpoint CCRAWL_CDX_ENDPOINT Where the URL index comes from

Durations are written the way the flags are written, "500ms" or "5s", and have to be quoted. The endpoints are there for a mirror or a local proxy; the defaults are Common Crawl's own hosts.

A setting ccrawl does not read stops the run and names the line, since the alternative is a config file that looks like it is doing something and is not. So does --profile naming a table the file does not declare, and the error lists the ones it does. A value of the right key and the wrong type, workers = "lots", is reported on stderr and the run continues on the default: that is read while the flags are being registered, where there is nowhere to return an error.

Global flags

These apply to every command.

Flag Short Meaning Default
--crawl -c Crawl ID, year (all crawls of that year), latest, all, an integer for the newest N, or a comma list latest
--output -o Output format: auto, table, json, jsonl, csv, tsv, url, raw, parquet auto
--limit -n Maximum records (0 = unlimited) 0
--workers -j Concurrency for downloads and scans 8
--source Bulk data source: https or s3 https
--rate Minimum delay between requests, for this process alone 0s
--global-rate Minimum gap between Common Crawl requests across every ccrawl process on this host (0 disables) 200ms
--timeout Per-request timeout 0s
--no-cache Bypass the on-disk cache false
--fields Comma-separated columns to show
--template Go template applied per record
--library Read and write under the dataset library false
--library-dir Library root ~/notes/ccrawl
--data-dir Root data directory
--dry-run Print actions, do not perform them false
--quiet -q Suppress progress output false
--verbose -v Increase verbosity (repeatable)
--color Color output: auto, always, never auto
--no-header Omit the header row in table output false
--db Tee every record into a store (e.g. out.db, postgres://...)
--profile Named profile to load
--progress Progress reporting for long runs: text, json, none text on a terminal, json otherwise
--journal Append run events as JSON Lines to this file run.jsonl beside the ledger
--metrics-addr Serve Prometheus metrics for the run on this address, e.g. :9090 off

See run journal for the event schema, the metric names, and the queries worth keeping.

The crawl list outlives the index server

Turning latest into a crawl ID needs collinfo.json from index.commoncrawl.org. It is cached for six hours, and when the fetch fails the cached copy is used at any age, with a line on stderr saying how old it is:

crawls: the index server is unreachable, using the crawl list cached 19h0m0s ago; pass -c to name a crawl instead of resolving it

Common Crawl publishes about six crawls a year, so a day-old list is almost always the list a fresh fetch would return, and the index server has gone away for three days at a stretch. Most of what ccrawl does reads data.commoncrawl.org, which stays up through those outages, and only touches the index server to resolve that one word. Without the fallback paths, columnar and download all failed on a crawl ID sitting in the cache.

Naming a crawl with -c CC-MAIN-2026-30 skips the lookup entirely, and is the right move for a scheduled job that should not depend on it. With no cached copy and the server unreachable the run still fails with exit 8: guessing a crawl list is worse than saying nothing.

--no-cache turns the whole cache off, the fallback with it.

The shared request budget

--rate spaces the requests one ccrawl process makes. That is not the number Common Crawl sees. Running the URL publish, the domain publish, and a Markdown export at once means three processes each pacing themselves politely and three times the traffic arriving at a nonprofit that serves this for free.

--global-rate is the gap between requests summed over every ccrawl process on the host. The processes coordinate through a small lock file at <data-dir>/ratelimit.lock: taking a slot means locking the file, reading the time the next slot comes free, pushing it forward by one interval, and sleeping until the slot you were handed. The lock is held for sixteen bytes of read and write, so processes queue on the timestamps rather than on the lock.

It covers index.commoncrawl.org, data.commoncrawl.org, commoncrawl.org, and the commoncrawl S3 bucket. Requests a recrawl makes to arbitrary sites are not Common Crawl's bandwidth, so they pay --rate and nothing else. Columnar scans are also exempt, for the reason --rate does not apply to them either: a scan is thousands of few-kilobyte footer reads, and pushing those through a five per second budget turns a thirty second query into an hour.

The default is 200ms, five requests per second, which is what a single process used to take on its own. So one process behaves exactly as it did before, and three processes now split that budget instead of tripling it. Raise the gap to be gentler, or pass --global-rate 0 to switch the shared limiter off and go back to a per process delay. CCRAWL_GLOBAL_RATE sets it from the environment.

Processes sharing a budget must share a data dir, since that is where the lock file lives. When the file cannot be created or locked, which is what a read-only or exotic filesystem looks like, ccrawl prints one warning and falls back to the per process delay rather than failing the run. Pass -v to have any run print the rate it is actually working under.

Measured on one host, three concurrent pipelines walking twenty crawls each and fetching real path manifests: --global-rate 2s served 60 requests at 0.499 per second combined against a configured 0.500, and the same 60 requests with --global-rate 0 went out at 2.316 per second.

Exit codes

0 success, 1 error, 2 usage error, 3 the query matched nothing, 4 a credential is needed and is not set, 8 transport failure so Common Crawl could not be reached at all, 75 temporary failure so run it again. See exit codes for what to branch on and how to supervise a publish run.

76 min read