v0.10.2

Two fixes, and both of them are the same kind of bug the last release was spent on: a run that finishes, reports success, and hands back something other than what was asked for.

The second one is the measurement v0.10.1 promised and could not take, because index.commoncrawl.org had been refusing connections for 22 days when that release went out. It came back on 2026-08-18, after 26 days, and the numbers are below. They are not the numbers that were expected, and the reason is worth more than the speedup.

A misspelled subcommand wrote help into your data file

Every group command answered a subcommand it did not have by printing its own help to stdout and exiting 0:

$ ccrawl host lst -o jsonl > hosts.jsonl; echo $?
0
$ head -3 hosts.jsonl
  Enumerate and enrich hosts from the CC web graph

  USAGE

That is 2808 bytes of help text in a file a script believes holds host records, with nothing in the exit code and nothing on stderr to say otherwise. All seventeen groups did it: cache, columnar, config, content, crawl, crawls, db, domains, extract, host, index, library, markdown, news, publish, sched, and urls.

	before	after
exit code	0	2
bytes on stdout	2073 to 5591	0

The word that was wrong is now named on stderr, with a guess at what was meant:

$ ccrawl host lst; echo $?
  ERROR
  Unknown command "lst" for "ccrawl host", did you mean "get".
2

The fix is in the framework, which is what builds those commands, as any-cli v0.4.13. Args: cobra.NoArgs on the parent is the obvious answer and it does not work: a command with no handler of its own is not runnable, and cobra returns flag.ErrHelp for a command that is not runnable before it ever validates the arguments, so the validator never runs. Giving the command a RunE is what gets it far enough to have an opinion.

A misspelling at the very top still exits 1 rather than 2. It was briefly 2, at the price of ccrawl serch --help printing the root help and exiting 0, because the help flag is read before any argument check. scripts/docs-drift.sh is what caught that: it works out which command a heading scopes to by running it with --help, so every heading became a valid command and 40 documented flags were suddenly reported as undocumented. The lookup that catches a bad top-level name runs earlier than the help flag and is worth more than a consistent exit code. The reference now says which code you get where.

A flag the parser rejects also exits 2 now, for the same reason it did not before: its error never reached the exit-code taxonomy.

A URL filter that ran in the wrong place

--url-contains and --url-not-contains were post filters. A wildcard query pulled every page of the result down and then dropped most of it on the floor. They are substrings, and a substring is a regex the index server can apply, so they now go on the wire as filter=~url:.*sub.* and filter=!~url:.*sub.*. export sends --url-fgrep and --url-fgrepv the same way.

--explain says where each part of a query runs, and now says what the index cost:

$ ccrawl search '*.senate.gov' --url-contains /budget --explain -o url
search: 1 crawl: CC-MAIN-2026-30
search: the index server answers https://index.commoncrawl.org/CC-MAIN-2026-30-index?filter=~url%3A.%2A%2Fbudget.%2A&matchType=domain&output=json&url=senate.gov
search: pushed to the server: --url-contains /budget
search: applied here: --url-contains /budget (again, on what the server sent)
search: read 49.4 KB from the index (50576 bytes)

That count is taken in the one place an index response is read, so a page that arrived and was thrown away counts, and so does an attempt that came back short and was retried. Those bytes moved. Both substrings are still applied here on the way past, which costs nothing and means a server whose filtering disagrees with ours cannot widen a result. --no-push-filters keeps them off the wire, for a server that disagrees in the other direction.

Every character of those two filter strings was paid for once

The first version sent url:.*budget.*, and the whole feature read 1978 bytes across 43 index pages and returned nothing at all. It looked like a spectacular saving right up until the results were compared against the client side path. The 1978 bytes were 46 per page, and 46 bytes is {"message": "No Captures found for: ca.gov"} with a newline.

The index server compares a bare field:value as a literal string, not a regex, so that filter asked for a URL that is those nine characters. The ~ is what makes the value a regex. The regex is then anchored at the start, because the server matches it with Python's re.match, so the leading and trailing .* are what make a substring match a URL at all. And the ! goes outside the ~, because ~!url: names a field no row has. Each of those three is a way to get an empty answer from the real server without being told. Measured on CC-MAIN-2026-30 over abag.ca.gov, 788 rows of which 2 hold budget:

filter	rows
url:.*budget.*	0
~url:budget	0
~!url:.*budget.*	0
~url:.*budget.*	the 2 that match
!~url:.*budget.*	the other 786

The fake Common Crawl the tests run against read a filter the same way the first version wrote one, treating a bare field:value as a regex. A fixture more generous than the server it stands in for is how the first three rows of that table got as far as a green build, so it now copies all three rules exactly.

The measurement

ccrawl search '*.senate.gov' -c 1 --url-contains /budget, ten index pages of CC-MAIN-2026-30, no limit on either side:

	bytes read from the index	rows
pushed	50,576	77
pushed, again	50,576	77
pushed, a third time	50,576	77
--no-push-filters	73,740,022	75
--no-push-filters, again	81,066,396	75

About 1,500 times fewer bytes for the same question. The issue asked for an order of magnitude.

The issue also asked that both paths return identical rows, and the honest answer is that the pushed path is a superset. On a query small enough to read whole, *.abag.ca.gov --url-contains budget, the two are byte for byte the same output, 2 rows each, 1308 bytes against 786,086. On the wide query the two client side runs each returned 75 of the 77, and a different two were missing each time.

What that missing pair turned out to be

Neither client side run reported a lost page, and both paginate over the same ten pages. The index truncates a large page under load and closes the connection as if it had finished. The same page fetched by hand came back at 573,440 bytes and at 7,386,419 bytes minutes apart, both HTTP 200.

ccrawl already retries a page whose read breaks partway. A page that ends cleanly on a record boundary cannot be told from a complete one, by ccrawl or by anything else. So moving 75 MB to answer a query costs rows as well as time, and which rows is not knowable from the client. Filtering on the server is the difference between a complete answer and a nearly complete one, not only a faster one. The reference page says this under --no-push-filters, with these numbers and the date they were taken.

Two other things fell out of running this against the live server, and both are on the issue. *.gov, which is what the issue names, is 786 index pages on one crawl rather than the 250 estimated when it was written; a run started on 2026-08-18 took 13 minutes to reach page 54 and lost two pages to 504 Gateway Time-out, which is what the server does when it has to scan a 7 MB page behind a filter. Both directions would be about six hours today, so the measurement uses *.senate.gov, which is the same shape, a wide wildcard and a rare substring, and small enough to run five times and mean something. And /budget/ with the trailing slash matches nothing at all, 0 of 12,460 rows on page 0 of *.ca.gov, so the measurements use /budget.

Go 1.26.6

The toolchain moved from 1.26.5, which carried six standard library advisories published on 2026-08-18. The workflows read the version out of go.mod, so the one line is the whole fix.

golangci-lint is pinned to latest in CI, and 2.12.2 landed a staticcheck that no longer treats t.Fatal as the end of a path, which turned main red without a commit touching it. Two nil checks in ccrawl/robots_test.go were reworded to suit it.

Install

brew install tamnd/tap/ccrawl
scoop install ccrawl

The release attaches the prebuilt archives, the deb, rpm, and apk packages, and the container image at ghcr.io/tamnd/ccrawl, and refreshes the apt and dnf repositories.