v0.5.3

v0.5.3 fixes how domains publish chooses which web-graph release to mirror.

It used to take the first release id off Common Crawl's web-graphs index and trust it was both the newest and complete. Common Crawl lists a release as soon as the host-level graph is out, sometimes days before the domain-ranks table lands, so that single id could point the publisher at a domain table that does not exist yet and 404s.

Now the resolver enumerates every advertised release, sorts them by parsed release date so the choice does not depend on how the index page happens to order things, and probes each release's domain-ranks table newest first with a one-byte range request. It publishes the nearest release that actually has a table. A missing table 404s and is not retried, so probing a release that is not ready is cheap, and the common case where the newest release is ready is a single probe.

domains recount uses the same resolution, so a repair run lands on the same release a publish run chose. Passing --graph <id> still pins a release explicitly and skips discovery.

Install

brew install tamnd/tap/ccrawl
scoop install ccrawl

The release attaches the prebuilt archives, the deb, rpm, and apk packages, and the container image at ghcr.io/tamnd/ccrawl, and refreshes the apt and dnf repositories.