Bulk and archives

Single-record fetching covers most questions. When you want volume, you work with whole archive files: list their paths, download them, parse them locally, and convert them into something your pipeline can read.

Listing paths

ccrawl paths <kind> prints the file paths for a crawl, one per line, ready to pipe into download. The kinds are warc, wat, wet, robotstxt, non200responses, cc-index, cc-index-table, and segment:

ccrawl paths warc -c 2024-51            # every WARC path for the crawl
ccrawl paths wet -n 1                   # the first WET path
ccrawl paths robotstxt -o url           # full https URLs, not relative paths

Downloading files

ccrawl download <kind> pulls whole files from the crawl manifest, concurrently and resumably. Pass - instead of a kind to read an explicit list of paths on stdin:

ccrawl download warc -n 5               # first 5 WARC files of the crawl
ccrawl download robotstxt -n 1          # one robots.txt archive
ccrawl paths wet -n 100 | ccrawl download -   # download a chosen list

Files land under <data-dir>/raw in the source directory tree by default; use --out to choose a directory and --flat to drop the tree. Concurrency is set by --workers (-j).

Parsing a local archive

ccrawl parse <file> decodes a WARC, WAT, or WET file into structured records. The format is detected from the file name and can be forced with --format:

ccrawl parse file.warc.gz -o jsonl
ccrawl parse file.warc.gz --type response --status 200 -o table -n 20
ccrawl parse file.warc.wet.gz --lang eng -o jsonl

The same content flags as get apply, so you can extract while you parse:

ccrawl parse file.warc.gz --links -o url       # every outbound link in the file
ccrawl parse file.warc.gz --text -o jsonl      # text of every response
ccrawl parse file.warc.gz --markdown -o jsonl  # Markdown of every response

parse reads stdin with -, so it pairs with download for a stream that never hits disk:

ccrawl paths wet -n 1 | ccrawl download - --out - | ccrawl parse - --lang eng

Converting to Parquet or JSONL

ccrawl convert turns archive files into columnar Parquet or line-delimited JSON for your own tooling. It defaults to Parquet:

ccrawl convert file.warc.gz --to parquet -O out.parquet
ccrawl convert file.warc.wet.gz --to jsonl
ccrawl convert ./warc/ --to parquet --out ./parquet   # a whole directory

Add --markdown to convert HTML response bodies to Markdown as they are written. Parquet output makes a crawl slice queryable in DuckDB, Athena, or Spark without any further plumbing.

Exporting a WARC subset

Whole-file download gives you the crawl's files as published. When you want only the captures that match a query, ccrawl export writes them into fresh .warc.gz files instead. Each file opens with a warcinfo record carrying provenance (the tool and version, the prefix, and the exact command line), so the output is self-describing and reproducible.

ccrawl export 'example.com/*' --status 200 --prefix example
ccrawl export '*.gov/*' --mime application/pdf --size 100000000   # rotate every 100 MB

It takes the same filters as search (--match, --from, --to, --status, --mime, --lang, --filter) plus URL filters --url-fgrep and --url-fgrepv. Files rotate once they pass --size bytes (1 GB by default), named <prefix>-NNNNNN.extracted.warc.gz. You can also feed it locations on stdin, exactly what search --locations and columnar locations emit:

ccrawl search example.com --locations | ccrawl export - --prefix example

The exported records are the original gzip members fetched by byte range, so they round-trip through ccrawl parse unchanged.

Building a dataset library

For volume work you want the files in one tidy, browsable place rather than scattered across ad-hoc --out dirs. The --library flag gives the archive files a home and extends the same four commands to list, download, and process them in place. The library lives under ~/notes/ccrawl by default; override it with --library-dir or CCRAWL_LIBRARY. It is a separate tree from the data dir, so clearing scratch state never touches the corpus you keep.

ccrawl download wet -n 20 --library -c 2024-51    # pull 20 WET files into the library
ccrawl paths    wet --library -c 2024-51          # list the WET files you have locally
ccrawl parse    wet --library --lang eng -o jsonl # decode every local WET file, eng only
ccrawl convert  wet --library --to parquet        # write parquet beside the raw files

In library mode the argument is a kind, not a file. Raw archives land under <crawl>/<kind>/ and processed output under <crawl>/<format>/<kind>/, so a directory listing is the index:

~/notes/ccrawl/CC-MAIN-2024-51/
  wet/                 raw WET archives
  parquet/wet/         the same files as Parquet
  jsonl/wet/           and as JSONL

A few things worth knowing:

• download skips files already present, so re-running only fetches what is missing and the corpus grows incrementally.
• paths --library lists what you have on disk for a kind, the local mirror of the remote manifest.
• parse --library streams every archive of a kind through one output, so a global -n caps the whole run rather than each file.
• convert --library picks the destination subtree from --to (parquet/jsonl); pass -O/--out to send it elsewhere.

ccrawl config show prints the active library_dir so you can confirm where a run reads and writes.

Knowing what you have

A directory listing tells you the file names. It does not tell you whether the bytes are still the ones that were downloaded, which crawl a file came from once it has been moved, or what any of it is costing you. library.json at the root of the tree does: every artifact with its size, sha256, creation time, and the version of ccrawl that produced it, updated by every command that writes into the library.

ccrawl library list                  # what is in the library
ccrawl library du                    # what it costs, per crawl
ccrawl library verify                # rehash it all, report what moved
ccrawl library gc --older-than 90d   # free the crawls you are done with

verify is the one worth putting in a cron. A WARC that lost a block to a bad disk still parses, still converts, and quietly produces wrong rows; the checksum is what turns that into a line of output. It exits 1 when anything failed, so it can gate a publish run.

gc is a dry run unless you pass --yes, and the dry run prints exactly the list the real run deletes.

ccrawl library gc --older-than 90d          # what would go
ccrawl library gc --older-than 90d --yes    # and now it goes

If your library predates all of this, or you copied files into it by hand, ccrawl library scan reads the tree and writes the manifest. It hashes every byte the first time and is a no-op after that.

Counting files

ccrawl stats shows the shape of a crawl: how many files of each kind it ships.

ccrawl stats -c 2024-51

For row-level questions over a whole crawl, reach for the columnar index rather than downloading and parsing everything.