A group engineering blog
Real bugs, real fixes, runnable code
Postmortems and tutorials from four developers who run production systems. Every post shows the commands, the output and the diff we shipped.
Browse by topic
9
stories so far, by 4 writers
Latest posts
Renaming a column without downtime: expand, migrate, contract
ALTER TABLE ... RENAME COLUMN takes a millisecond and breaks every running pod. A step-by-step expand and contract rename in Postgres, with the trigger we use.
TIL: our trial countdown was wrong one day a year, and only in Europe
A date test that passed for months failed on a CI runner in Berlin. What we learned about DST, local midnight and testing code that reads the clock.
Feature flags without a vendor: one table, one function, one rule
For most teams, feature flags are a Postgres table, a hashing function and a policy about expiry dates. Here is ours, and when we would pay for a product instead.
The index that did nothing: reading an EXPLAIN plan line by line
We added an index to fix a 1.2 second query and the plan did not change. Here is how EXPLAIN ANALYZE told us why, and the partial index that worked.
A retry helper with full jitter, in about 40 lines of Go
Exponential backoff without jitter turns one outage into synchronized waves of retries. Here is a small Go helper that gets backoff, jitter and cancellation right.
Get the next post in your inbox
About two emails a month. Each one is a full post with code, written by someone who ran it in production first.
Plain text. No tracking. Unsubscribe any time.