v0.10.1

v0.10.0 shipped with a documentation set that had drifted away from the binary in six separate places. None of them were caught by a test, because none of them are the kind of thing a test looks at: every one is a sentence that was true when it was written.

This release closes that gap and, where the sentence was the better idea, moves the binary rather than the page.

Chasing those six turned up seven more of the same family, and they are here too. They have one thing in common worth naming: not one of them was a crash. A stale crawl list, a cache that did not follow its data dir, a zero standing in for a number nobody measured, a misspelled output format rendered as JSON Lines, a thirteen-minute stream that died on the network and reported the code for a bad command. Every one of them is a wrong answer that looks exactly like a right one, which is why they lasted.

A guide that described a command nobody wrote

The content signals guide showed content quality - reading a JSONL stream. There was no stream form. It listed six output columns, none of which QualityReport emits, and documented a --warc flag on three subcommands and a --lang flag on a fourth, none of which were ever registered. Every pipeline on the page failed.

The page was right about what the commands should do, so all four content commands now take - in place of the URL and read a list from stdin. A line is either a bare URL or a JSON object with a url field, which is what search, columnar and crawl fetch write with -o jsonl, so a query feeds a content command with nothing in between:

ccrawl content outlinks https://go.dev/blog/ -o jsonl -n 3 | jq -r .url | ccrawl content quality - -o jsonl

A URL that cannot be fetched is named on stderr and the rest of the list carries on. A run that scored at least one page exits 0, a run where every URL failed exits 1, and empty stdin exits 3.

A dataset card naming the wrong dataset

markdown export --repo defaults to open-index/open-markdown-v3. The README generator hardcoded open-index/open-markdown-v2 in eight places, so the card committed into the v3 repo told readers to download v2, and a run publishing to somebody else's repo advertised ours on the one page a reader comes to for the download line.

Every snippet and link on the card is now built from the repo the run is committing to, and the card and the export flag read one constant, so the next version bump is one edit rather than nine.

Errors and help printed flags that do not exist

The CLI renders through fang, which title-cases the first word of a message before drawing the box. On an ordinary sentence that capitalizes the first letter. On a message that opens with a flag it rewrites the flag:

$ ccrawl rank host wikipedia.org
  ERROR
  --Table is required (URL of a gzipped rank table).

$ ccrawl sched diff
  ERROR
  --Crawl-A and --crawl-b are required.

The renderer belongs to the framework, so the rule is ours to keep: a message may not open with a word that title-casing damages. 291 error messages were checked and the ones that broke were reworded.

The same renderer runs over the help line beside every flag, which is worse, because help is where a user looks up what to type. --engine advertised Auto|Duckdb|Native|Print, four values it rejects all four of. --subset advertised Warc|Crawldiagnostics|Robotstxt. Eleven columnar filters opened with a bare column name and printed it capitalized, so --mime offered Content_mime_detected, which is not a column any query has.

42 help lines were reworded, 252 are now checked on every run, and both rules are guarded by a test that walks the source, because the framework builds its command tree privately and a test cannot ask the binary what it registered.

Two framework flags are still damaged and cannot be fixed here: --fields prints "Comma-Separated columns to show" and --timeout prints "Per-Request timeout". They are recorded in the test that cannot reach them.

JSON keys that matched nothing else

ccrawl parse on a WET or WAT file wrote Go field names:

{"RecordID":"...","URL":"...","ContentLanguage":"eng","Text":"..."}

The parquet column for that language is content_language, the columnar index calls the WARC location warc_filename, warc_record_offset and warc_record_length, and ccrawl fetch reads exactly those three names. So a script moving records between two ccrawl commands had to rename the keys in between.

Both record types now carry the names the parquet schema uses, and the WAT location fields carry the names fetch reads, so parse wat feeds fetch unchanged apart from the crawl-data prefix, which Common Crawl does not put in a WAT record. convert and parse write the same object for all three kinds, verified record for record against CC-MAIN-2025-05.

index build --input accepts all three spellings of the language key: language, which is what a hand-written file says, content_language, and ContentLanguage for a file written by an older ccrawl. Run three ways over the same 500 records, all three give {"docs_added":499,"docs_skipped":1,"terms":84001}.

Table and CSV column names stay short. A terminal column is read by a person and a JSON key is read by a program, and those are different jobs.

A rank lookup that demanded a URL the program already knew

$ ccrawl rank host wikipedia.org
  ERROR
  Point --table at a gzipped rank table.

$ ccrawl host get wikipedia.org
  {"host":"wikipedia.org","harmonic_pos":864,...}

The same question, the same web graph, the same RankLookup underneath, and only one of them went and found the table.

All four rank commands now resolve it the way host does: --table wins, then the release named by --graph, then the newest release published. The domain ranks resolve separately, because a release is listed as soon as its host tables land and its domain table follows later, so the newest release is often the wrong answer for a domain lookup.

ccrawl rank host wikipedia.org      # harmonic_pos 864, 1.65s
ccrawl rank domain wikipedia.org    # harmonic_pos 14, 1.61s
ccrawl rank all --tld gov -n 3

864 against 14 for the same name is why the host and domain split has a test of its own: they are separate tables with separate positions in them, and reading a host rank out of the domain table is the kind of wrong answer that looks right.

A network outage that exited 1

ccrawl --help has always listed exit 8 as "transport failure, the bytes did not arrive". download was the only command that returned it.

index.commoncrawl.org refused connections for two days while this release was being written, which made the cost obvious:

$ ccrawl search 'example.com/*'; echo $?
  ERROR
  Fetch collinfo: all 6 attempts failed ... connect: connection refused.
1

Exit 1 is the code for anything at all, so a cron job could not tell an unreachable Common Crawl from a bad crawl ID.

The retry loop now marks its exhaustion as a transport failure when the last attempt never reached a server, and the CLI classifies that as exit 8. An exhausted 503 stays exit 1: the server was there and said no, and telling a supervisor otherwise sends it into a loop against a source that is up and refusing.

ccrawl search "$url" -o jsonl > captures.jsonl
case $? in
  0) ;;                                  # captures on disk
  3) echo "nothing crawled for $url" ;;  # a real answer
  8) sleep 3600; exec "$0" "$@" ;;       # Common Crawl is unreachable, come back later
  *) exit 1 ;;                           # the command itself is wrong
esac

The classification happens where commands are registered rather than in each of the 34 handlers, and a test walks the source and fails on a registration that goes around it.

One host being down took out the commands that did not need it

index.commoncrawl.org refused connections for three days while this release was being written. data.commoncrawl.org answered 200 the whole time.

Most of what ccrawl does reads the second host: the path manifests, the WARC files, the columnar Parquet index, the web graph. The first host is needed for the URL index, and for turning the word latest into a crawl ID. That second job was enough to take everything down:

ccrawl crawls latest                      # exit 8
ccrawl paths warc -n 1                    # exit 8
ccrawl columnar urls --tld gov -n 1       # exit 8
ccrawl download warc -n 1 --dry-run       # exit 8

The crawl list was in the cache the entire time. ListCrawls cached it for six hours and then refused to look at it, so past the TTL the fetch was the only path and there was nothing behind it.

A failed fetch now falls back to the cached list at any age and says how old it is:

$ ccrawl columnar urls --tld gov -n 1 -o url
crawls: the index server is unreachable, using the crawl list cached 19h0m0s ago; pass -c to name a crawl instead of resolving it
https://www.13thjudicialil.gov/

Common Crawl publishes about six crawls a year, so a day-old list is almost always the list a fresh fetch would return. Naming a crawl with -c skips the lookup and is the better move for a scheduled job. With nothing in the cache the run still fails with exit 8, because guessing a crawl list is worse than saying nothing.

That fix uncovered a second one. search keeps going past index pages it cannot read, and a run that emitted nothing exited 1 whatever the reason. During a full outage the crawl list used to fail first, so that path was rarely reached; once the list resolved from cache, a total outage landed there and came back as exit 1, undoing the guarantee above. A search that lost everything to the transport now exits 8, and one where any part of the loss was a status the server sent stays 1, because a supervisor that backs off against a server which is up and refusing never stops.

A flag that asked for the impossible and worked for half an hour anyway

crawl seed reads the web-graph host rank table and assigns each host a crawl tier. A tier is a function of two things, how central the host is and how often it changes, and the rank table only carries the first, so every seed is tiered as if it changed at the middle rate. Tier 1 needs a change rate above 0.8. No seed has ever reached it and no seed ever will.

--max-tier 1 accepted that request and went looking:

$ ccrawl crawl seed --max-tier 1 -n 5 --progress none; echo $?
  No results.
3

That took 28 minutes and read all 262 million rows to find out what the tier function says in one line. Exit 3 means the query ran and matched nothing, which read as "Common Crawl has no important hosts".

It now answers before it opens the stream:

$ ccrawl crawl seed --max-tier 1 -n 5; echo $?
  ERROR
  Seeds carry no measured change rate, so no host reaches tier 1; use --max-tier 2
  for the top million hosts, or feed real change rates in with ccrawl sched diff.
2

Tier 2 is the top million hosts and is what the flag is for. The recrawl guide already said tier 1 was out of reach from a seed; the CLI reference and the flag help now say it too, and a test pins the floor to what the tier function actually returns.

One question, two commands, two answers

crawls info accepted -o and ignored it. Every other read command writes JSONL, CSV, or a table on request; this one printed its own key: value block whatever you asked for, so a script that piped it to jq got nothing usable and no error telling it why.

It had also drifted apart from stats, which asks the same question from the top level. stats counted warc,wat,wet,robotstxt,non200responses and crawls info counted warc,wat,wet,robotstxt,cc-index-table, one kind different in each direction, so which totals you got depended on which name you happened to type.

Both are now the same operation over one list of kinds, which puts the renderer back in the path and makes the two spellings agree by construction:

$ ccrawl crawls info -o jsonl
{"crawl":"CC-MAIN-2026-30","kind":"warc","files":100000}
{"crawl":"CC-MAIN-2026-30","kind":"wat","files":100000}
{"crawl":"CC-MAIN-2026-30","kind":"wet","files":100000}
{"crawl":"CC-MAIN-2026-30","kind":"robotstxt","files":900}
{"crawl":"CC-MAIN-2026-30","kind":"non200responses","files":900}
{"crawl":"CC-MAIN-2026-30","kind":"cc-index-table","files":900}

crawls info keeps the positional crawl it has always taken, so ccrawl crawls info CC-MAIN-2026-25 still works, and it gains --kinds to narrow the list. stats still reads -c. A kind whose manifest cannot be fetched comes back as a row with files of -1 rather than being dropped, so a run never quietly returns fewer rows than there are kinds.

A flag that moved half the tree

--data-dir moved the downloads, the Parquet, and the DuckDB file, and left the cache pointing at the old tree. The reference has said since the beginning that the cache dir follows the data dir when nothing names it, and the config file did that; the flag did not, because the framework works out its default cache dir from its default data dir before any flag is parsed.

It is worse than untidy. It turned up while measuring the stale crawl list above: a run pointed at an empty directory, to prove a command fails when it has no cache, came back exit 0. It had been reading the ordinary cache the whole time. A flag that looks like isolation and is not gives you a wrong answer with nothing on screen to say so.

$ ccrawl cache dir --data-dir /tmp/dd1
/tmp/dd1/cache

$ ccrawl crawls latest --data-dir /tmp/emptydata; echo $?
Fetch collinfo: all 6 attempts failed ... connect: connection refused.
8

Naming the cache dir still pins it, with cache_dir in the config file or CCRAWL_CACHE_DIR in the environment, and the flag leaves a named one alone.

ccrawl config show was part of the same problem. It reported default as the source of cache_dir and db_path even when both had moved with the data dir, and that column exists to end questions about where a value came from rather than start them. Both now say derived from data_dir when that is what happened, the way raw_dir and parquet_dir always have.

A number nobody measured, reported as zero

host top and host get read the rank table. The rank table has ranks in it and nothing else, so those two commands never learn how many inbound links a host has or how many URLs it has in the index. They printed 0 for all of it anyway:

$ ccrawl host top --limit 1 -o jsonl
{"host":"www.facebook.com","harmonic_pos":1,"in_degree":0,"url_count":0,"status_2xx":0,...}

Facebook has more than zero inbound links. Zero is an answer, and this was the absence of one, wearing the same clothes.

Every count on a host record is now a pointer, so a number nothing measured is left out of the JSON rather than filled in with a guess:

$ ccrawl host top --limit 1 -o jsonl
{"host":"www.facebook.com","host_rev":"com.facebook.www","tld":"com","registered_domain":"facebook.com","harmonic_pos":1,"harmonic_val":34375268,"pagerank_pos":3,"pagerank_val":0.0055143537769673755}

The column is still there in -o csv and -o table, and the cell is simply empty, so the shape of the output does not change with what a run happened to measure:

$ ccrawl host top --limit 3 -o csv
host,tld,registered_domain,harmonic_pos,harmonic_val,pagerank_pos,pagerank_val,in_degree,out_degree,url_count,...
www.facebook.com,com,facebook.com,1,3.4375268e+07,3,0.0055143537769673755,,,,...
fonts.googleapis.com,com,googleapis.com,2,3.369161e+07,1,0.007332122627094865,,,,...
www.google.com,com,google.com,3,3.1091204e+07,2,0.006303097428163461,,,,...

host degrees, host enrich, and host cdx are the commands that measure those columns, and they fill them. A count one of them took and that came out zero still prints as 0, which is the whole point of separating the two cases.

host top also fills registered_domain now, which it had been leaving blank for no reason: the registered domain is the host with its labels dropped, so there was never anything to fetch.

A stream that died halfway exited 1

The exit 8 work above covered a request that never got an answer. It did not cover a request that got one and then lost it, because the transport marker is attached by the retry loop, and once the response headers arrive the body read has left that loop behind.

The commands this hits are the long ones. The host rank table is 262 million rows and the edge files are 7.7 GB, so they hold a connection open for minutes:

$ time ccrawl rank host example.invalid; echo $?
read tcp [2001:ee0:...]:54947->[2600:9000:...]:443: read: no route to host
1

real	1m17.093s

Seventy-seven seconds of streaming, ended by the network, reported with the code that means the command is wrong. A supervisor watching that stops instead of backing off.

Losing the connection mid-response is now exit 8 like any other transport failure. It caught a real one the same week, unstaged: a host enrich --degrees run started to measure something else died 800 seconds into the vertex table.

{"ts":"2026-08-12T23:53:08.127Z","event":"start","pipeline":"host enrich"}
{"ts":"2026-08-13T00:06:28.800Z","event":"end","phase":"vertices","elapsed_s":800.670773084}

  Phase 2 vertices: read tcp [2001:ee0:40e1:...]:60647->[2600:9000:26cc:...]:443: read: no route to host.

exit 8

Thirteen minutes of streaming, ended by the network. Exit 8 is a supervisor backing off and running it again; exit 1, which is what it was, is a supervisor stopping. A status the server sent is still exit 1, so an exhausted 503 does not send anything into a retry loop against a host that is up and refusing.

Two edges are worth naming, because getting them wrong is easy. os.PathError has Timeout and Temporary methods and therefore satisfies the net.Error interface, so classifying on the interface reports a missing local file as a network outage; this asks for the concrete socket errors instead, and ccrawl parse /nope.warc.gz stays exit 1. And content quality against a host that will not answer moves from 1 to 8, which is the same rule applied consistently, while a URL that does not parse stays at 1 because no amount of waiting will fix it.

A format nobody could encode, rendered as something else

The renderer picks an encoder by name and its switch ends in a default that writes JSON Lines. Nothing rejected the name on the way in, so a typo went all the way to the encoder and fell off the end:

$ ccrawl search 'example.com/*' -o csvv > captures.csv; echo $?
0

captures.csv is full of JSON Lines and the run reported success. -o is the one flag whose entire job is to say what the bytes will look like, which made this the quietest way in the program to get the wrong bytes.

An unknown format is now a usage error, exit 2, with nothing written to stdout, so a redirect leaves an empty file rather than an error inside one:

$ ccrawl search example.com -o csvv > captures.csv; echo $?
ccrawl: -o csvv: unknown output format, use one of auto, table, markdown, list, json, jsonl, csv, tsv, url, raw, md, section, sections, template, parquet
2

The valid set belongs to the framework rather than to ccrawl, and a copy of it here would start refusing values that work the day the framework adds a format. So the copy is checked: a test reads the --output help off the built command tree, which the framework generates from the formats it implements, and fails on any difference. Formats a binary registers itself, which is how parquet gets in, are read at runtime rather than written down.

It caught one on the first run. A test in this repository had been passing -o crawl,url since it was written, meaning --fields, and passed anyway because the fallthrough handed it JSON Lines and JSON Lines contain a crawl field. That is the shape of the bug: not a crash, an answer that looks right.

Six facts glued into a sentence

ccrawl version prints the version, the commit, the build date, the platform, and the Go toolchain, all in one line, and -o did nothing at all:

$ ccrawl version -o csv
ccrawl 0.10.0 (commit 77a6294, built 2026-08-12, darwin/arm64, go1.26.5)

A CI job that wants the commit had to write a regular expression against a line whose format is documented nowhere. It is a record wearing a sentence, so now it can be either:

$ ccrawl version -o json | jq -r .commit
77a6294

-o auto, the default, keeps the sentence, and keeps it in a pipe as well as on a terminal. Everywhere else auto means a table for a person and JSON Lines for a pipe, and following that rule here would change what ccrawl version | grep returns for everyone who already has it in a script. Asking for a format is how you say you want data. --short is untouched.

This turned up in a sweep of the local commands after the crawls info fix, and it is worth saying what the sweep left alone. cache dir, library path, and config get each print one string. Wrapping one string in a JSON object helps nobody and would break cd "$(ccrawl cache dir)".

One thing this release promised and does not deliver

v0.10.0's notes said the CDX filter push ships in v0.10.1 with the two runs that prove it. It does not, and the reason is worth more than a quiet omission.

The change translates --url-contains and its family into server side filter= expressions, so a domain wildcard query stops downloading pages in order to throw them away. Its entire claim is a byte count, and the byte count has to come from index.commoncrawl.org, which has refused connections for 22 days:

$ curl -sS --max-time 20 https://index.commoncrawl.org/collinfo.json
curl: (7) Failed to connect to index.commoncrawl.org port 443 after 432 ms
$ dig +short index.commoncrawl.org
54.237.141.66

DNS resolves and nothing answers. data.commoncrawl.org is up, which is why everything else here could be measured against live data.

The code is written and green on every check. It is not merged, because merging a performance change with nothing behind it is how a claim turns into folklore, and a release note saying "an order of magnitude fewer bytes" with no number under it is exactly the kind of sentence the rest of this release was spent deleting. It moves to v0.10.2 with the measurement to run written down on the issue.

The outage is also why the crawl-list caching above matters more than it looks. Three weeks of one host being down is what turned that from tidiness into the difference between a working binary and a broken one.

CI was red

docs-drift had been failing on main since the content commands landed: the requirements page is checked against a hardcoded list of the commands it covers, and crawl fetch and crawl run were not on it, so two flags that exist were reported as drift.

It merged red because main has no branch protection and --auto merges as soon as GitHub considers a PR mergeable, whether or not the checks passed. That is a repository setting rather than a code change and is open for the maintainer.

Install

brew install tamnd/tap/ccrawl
scoop install ccrawl

The release attaches the prebuilt archives, the deb, rpm, and apk packages, and the container image at ghcr.io/tamnd/ccrawl, and refreshes the apt and dnf repositories.