Build log · 22 August 2026
The query that took production down
Written up in November 2026 from our engineering log · 2 min read
On 22 August the site went down, and the cause was not a bug in the usual sense. The code was correct. It was just far too expensive.
A correct rule, an expensive shape
Teams that share a website can read each other's work on it, through a roster. The first version expressed that as one branch per site, each with a deeply nested subquery to prove both teams were on the roster. For an account with seats on about 55 sites, every site-scoped read became a query with about 55 correlated subqueries.
Those reads sat on the endpoints the Chrome extension polls continuously. With a nine-connection database pool, they filled it. Then even the sign-in check could not get a connection, requests failed, the extension retried, and the load fed itself.
Same rule, resolved in code
The fix kept the rule exactly and changed its shape: the roster is read once, the pairs of site and team are worked out in code, and the database receives plain equality conditions it can answer from an index. Caps on the number of branches keep it bounded, and a log line says so whenever a cap is reached, rather than silently dropping access.