Build log · 6 August 2026
A name is not a destination: hardening our crawler
Written up in November 2026 from our engineering log · 2 min read
A crawler fetches addresses that other people choose. When it runs inside a private network, the question is not only where a URL says it goes, but where it actually lands.
What was missing
Our crawler validated the hostname of each URL. But a name is not an address: a perfectly public-looking name can resolve to a private one, and a public URL can redirect somewhere it should never go. Checking the name alone, and following redirects automatically, left both doors open. Both were exploitable, and both were closed on 6 August.
The rule now
Anything that opens a connection to an address someone else influenced resolves it first and refuses unless every resulting address is public — and does that again on every redirect hop, never with automatic redirect-following. The check is the same lookup the connection uses, so there is no second resolution to race.
A rule only holds where it is applied, so a test now fails the build if code that imports the guard makes a request without it. Its first version stayed green while it never scanned the worker's folder at all; it now asserts its own coverage.