Reference
Every command, flag, output format, and setting, plus a troubleshooting guide.
The complete surface, for when you know what you want and just need the exact spelling: the CLI command tree and flags, the Markdown pipelines and their schemas, the CC-NEWS index ccrawl builds because Common Crawl publishes none, the columnar engines behind ccrawl columnar, what each command needs at runtime, configuration and environment, the run journal a long run writes, output formats, exit codes, how far each part goes, and troubleshooting.
CLI
Every command and subcommand, with the flags that matter.
Recrawl datasets
The two recrawl repos: the capture schema column by column, how the shards and the ledger are laid out, and DuckDB queries that run against the hub as written.
Markdown pipelines
The markdown export and refetch commands: schemas, HuggingFace layout, resume behaviour, and the three knobs that set throughput.
CC-NEWS index
The index Common Crawl does not publish for CC-NEWS: how it is built, what the columns mean, and how a search reads it.
Columnar engines
The two engines behind ccrawl columnar, what each one can answer, and which one runs when.
What each command needs
The runtime dependencies outside the binary: network, DuckDB, tokens, credentials, disk, and ports.
Configuration
The data directory, environment variables, and global flags, with their defaults.
Run journal
The JSON Lines event log a long run writes, the metrics it exposes, and the queries to ask of both.
Output formats
Every output format, how to narrow columns, and how to template rows.
Exit codes
What ccrawl returns to the shell, and which codes are worth branching on.
Troubleshooting
The handful of things that trip people up, and how to fix each one.
How far each part goes
Which subsystems are finished, which are running in production, and which are reference implementations.