Build log · 31 July 2026
Splitting the crawler from the website
Written up in November 2026 from our engineering log · 2 min read
Until the end of July, the website and the crawler shared one container. A long crawl and a page view were competing for the same machine.
One container, two very different jobs
Serving a dashboard is short and bursty. Crawling a site is long and steady, and it needs the CPU the whole time. On a platform that throttles CPU between requests, putting both in one process meant the crawler was starved exactly when nobody was visiting — and the website was slowed exactly when a big crawl was running.
Same image, two roles
We split them into two Cloud Run services built from the same image: the web service starts no worker, and the worker runs as one always-on instance with CPU always allocated. A single environment variable, SERVICE_ROLE, decides which half a container plays.
The same day, a runaway clean-up query sat on the production database for over two hours. The database role now has a five-minute statement timeout, so no single statement can do that again.