Skip to content
ccrawl

Getting started

Install ccrawl and pull your first page out of Common Crawl in under a minute.

Three short pages: what Common Crawl is and how ccrawl thinks about it, how to install the binary, and a guided first run that ends with you reading the text of a real captured page on your terminal.

Introduction What Common Crawl is, how it is laid out, and the model ccrawl uses to make it feel small. Installation Install ccrawl from a release, with go install, or from source. DuckDB is optional. Quick start From an empty terminal to reading a real captured page, in a handful of commands.