v0.9.0

ccrawl could read Common Crawl and it could fetch a page. It could not crawl. The frontier, the robots cache and the WARC writer were all in the tree, all tested, and none of them reachable from a command. v0.9.0 fixes each of the three and then wires them together.

ccrawl crawl seed -n 100000 -o jsonl > seeds.jsonl
ccrawl crawl run --seeds seeds.jsonl --out warc/ --state crawl.db --delay 1s -j 64

That is a crawl. It obeys robots.txt, keeps one request per host per delay, writes standards compliant WARC, and resumes where it stopped if you kill it.

crawl run

The loop is small on purpose. Workers pop from the frontier, check the host's robots.txt, fetch, write the request and response pair to WARC, extract links, and admit what is in scope back into the frontier.

Flag	Does
--seeds	crawl seed JSONL or one URL per line, - for stdin
--state	the frontier file, which is what makes a run resumable
--out	WARC directory
--delay	minimum spacing between two requests to the same host
--max-depth	how far from a seed to follow links, 0 is the seeds only
--max-pages	stop after this many fetches
--same-host	stay on the hosts the seeds named
--no-robots	skip the robots check, which you had better have a reason for
--warc-size	rotate to a new WARC file past this many bytes

100,000 pages against a 200 host local harness at --delay 10ms -j 64 finish in 50.5 seconds with nothing failed, which is 1980 pages per second. Against the open web from 120,000 real seeds it settles at about 3.2 pages per second with a mean body of 389 KB, because 96 workers saturate roughly 1 MB/s of downstream and the crawler spends its life waiting on sockets. The archive passes warcio check either way, and warcio index over the 100k run counts 100,000 requests, 100,000 responses and one warcinfo.

Kill a run and start it again against the same state file and it picks the queue back up. kill -9 at 22,042 of 100,000 pages, then a restart that fetched the other 78,496 and stopped on its own: the union is exactly 100,000 URLs and the overlap is 538 pages. Those 538 are the completion buffer that had not been flushed yet plus what was in flight when the signal landed, which is the documented cost of buffering completions rather than committing each one.

The frontier is a file now

seen was a map[string]struct{} and the queue was a slice, which is fine until the frontier holds what a frontier is for. The seed set this was built for is around 280M URLs, and a Go map with 280M string keys is tens of gigabytes of live heap and a collector that never catches up.

It is SQLite now, through modernc.org/sqlite, so the binary is still pure Go with nothing to install beside it. One table for the queue keyed on a 16 byte URL hash, one for per host politeness, and a staging table in front of the queue.

The staging table is the interesting part. Inserting discovered URLs straight into the frontier writes to a random page of a B-tree much larger than any page cache, and the admit rate collapses as the table grows: 170k/s at 200k rows, 2.9k/s at 3M, still falling at 4M. Admissions go to a rowid table written in increasing rowid order instead, where an append costs the same at four million rows as at four hundred, and they are merged into the frontier with a single ordered INSERT OR IGNORE that walks the tree left to right once. Completions and politeness clocks are buffered and written in key order for the same reason.

There is no Bloom filter, and the issue asked for one. A Bloom filter's error is a false positive, which in front of a seen check means calling a brand new URL already known and dropping it. Sized at the conventional one percent it lost 52 of 50,000 URLs in the test run, and one page in a hundred missing from a crawl with nothing downstream able to tell is not a trade a data structure gets to make for the operator. The cache is exact for what it holds and forgets the coldest keys instead of lying about them, and a forgotten key costs one wasted row in the next merge.

robots.txt to RFC 9309

The old parser matched literal path prefixes, knew two user agent names, and read a 5xx on /robots.txt as permission to crawl. That last one is a bug with consequences: a site whose robots endpoint is failing cannot tell you to stop, and crawling it anyway is how a crawler ends up in a block list. Section 2.3.1.4 calls an unreachable robots.txt a complete disallow, and so does ccrawl now, remembered for five minutes so the host is crawled again as soon as it recovers.

What the parser handles: * and $ wildcards with longest match winning and allow breaking a tie, case insensitive product token matching where the most specific group wins and the wildcard group never merges with a specific one, an empty group naming a crawler still counting as that crawler's group, percent encoding normalised on both sides, Sitemap lines collected without interrupting the group they sit in, and the 500 KB parse limit. Crawl-delay is honoured too, which is not in RFC 9309 but is still what sites write. The RFC's own section 2.2.2 match table and section 5.2 example file are both transcribed into the test suite, including the four per bot verdicts the RFC spells out.

Status handling had to get more careful than a bool. A 403 means there is no robots.txt to read and the site is open, a dead host means assume everything is off limits, and the HTTP client was collapsing both into the same error because it retries 403 and 5xx alike. Robots can tell a refusal from a silence now.

A Crawl-delay that nothing enforces is worth nothing, so a parsed delay pushes the host's next eligible time out in the frontier and never brings it forward.

WARC that another tool will read

The crawl side had a WARC writer in name only: five headers, no digests, no request record, no warcinfo, a Content-Length copied off the wire, and no command that could reach it.

Each file now opens with a warcinfo record naming the tool and the exact command, and every record carries that record's ID in WARC-Warcinfo-ID. A fetch becomes a request and a response linked both ways with WARC-Concurrent-To, written as one unit so a rotation never splits a pair across files. Every record carries a WARC-Block-Digest and every response also a WARC-Payload-Digest, both sha1: and base32. WARC-IP-Address comes from an httptrace hook, since that is the only place the remote address is observable. The 10 MB body cap flags the record with WARC-Truncated: length rather than storing a short page as if it were whole.

Header blocks are rebuilt rather than captured, because net/http hands back a decoded body and a parsed header and never the wire bytes. That is what makes the rewriting mandatory: Content-Length is recomputed against the stored body, Transfer-Encoding is dropped because Go dechunks on the way in, and Content-Encoding is dropped when we decoded it. The record describes the body stored with it.

crawl fetch --warc-dir reaches the writer for a one off fetch, and crawl run uses the same writer for a whole crawl.

Politeness is measured, not asserted

Spacing the pops out of a frontier is not the same as spacing the requests that leave the machine, and a server counting arrivals says so. The crawler holds its own per host clock, waits on it before it dispatches, and restamps it from httptrace.WroteRequest when the bytes actually go out.

Measured on the harness: a sequential control of 300 pages against one host at a 10 ms delay sees a 10.65 ms minimum gap and never goes under. With 64 workers the same delay measures 8.72 ms minimum and 10.58 ms median, and a 50 ms delay measures 48.25 ms minimum. The 1.3 ms to 1.8 ms shortfall is the server's own handler scheduling jitter at 2000 requests per second, which the sequential control rules out as a crawler fault.

Robots is fetched once per host and no more, 200 fetches for 200 hosts under 64 workers. A harness serving Disallow: /private/ saw zero requests for a /private/ path across 20,000 fetches, with 1,000 URLs refused before the wire.

Also in this release

The frontier throughput tests no longer assert absolute rates. A floor that passes on a laptop and fails on a shared CI runner tests the runner, not the code, so the tests check the shape of the curve and the ordering guarantees and leave the numbers to the benchmarks.

Install

brew install tamnd/tap/ccrawl
scoop install ccrawl

The release attaches the prebuilt archives, the deb, rpm, and apk packages, and the container image at ghcr.io/tamnd/ccrawl, and refreshes the apt and dnf repositories.