Buggy Code on Production, Survived
Areca is the name of the billing engine I am working on for Turk Telekom. Funny enough, it is also the name of the flowers we bought to freshen the office. We wanted the office...
Production and Reliability Series
What software engineering looks like under production pressure: debugging hard problems, handling incidents, managing overload, building for reliability.
Local vs Production Debugging
I have been debugging this data workflow tool we built in house lately. It has an Angular UI and a Java backend, and it moves data between different systems like Postgres to Hiv...
Update Statements on Production
Executing update statements on a production database is always a big challenge. It’s one of those tasks that looks deceptively simple until something breaks in ways you didn’t i...
Message Brokers Are Modern Grids
While working on my book on , I keep noticing the same pattern. Some systems look simple while they belong to one team and become something else after everybody starts using the...
Service Overload Strategies
Service overload happens a lot. If you haven't seen one, count yourself lucky. The first time I watched it take a system down, I realized how serious it’s to get the basics righ...
Balancing Act of Reliability
Once something is in production, you are no longer just building software. You are also keeping it alive. That sounds obvious, but teams forget it all the time. We get excited a...
Silent Guardians of Quality
Testers are the silent guardians of software. Their work is easy to miss when they do it well. If no critical or highpriority issues reach production, people assume there was no...
Why Metrics Don’t Equal Quality
In 1902, Hanoi was drowning in rats. The government was getting nervous about plague. Hence, the city put a bounty per rat tail. Suddenly, the system had a way to measure, somet...
Promoting Learnings in Incidents
Incidents are the negative consequences of an action that fails to produce the expected outcome. For instance, code might be deployed to production to add a new feature or impro...
Operational Skills Needed
Over the years, I've interviewed many candidates. One crucial skill that often gets overlooked is operational reflexes during oncalls. Surprisingly, few companies test for this,...
The Mirror Is Part of the Machine
The most fun telemetry problems I've seen usually start with an incident where we could not see enough, so we added the missing field after the postmortem. Then that justincase...