A system only reveals what it's made of when it starts changing. At first, they often look better than they are because the code is still new. The data is still fairly clean. There is not much history yet. Most of the ugly parts only start showing up once the system has been in production for a while and people begin changing it.
What changed my mind was seeing how diffrent the same system looks once it starts changing under production load. A new version rolls out slowly. Some services are upgraded, some are not. A few clients keep old assumptions alive. Background workers process messages produced hours ago. Cached values stay around longer than you hoped for. Hope is not a good thing in this case. At that point, architecture has to answer whether old and new parts of the system can live together without causing trouble, or whether everything starts breaking in stupid ways.
That is why I think compatibility is a feature. I mean it in a very practical sense. If a system only works when everything upgrades together, then it is brittle. It may still look elegant in code review, but it will create stress the moment real change begins. A strong system has enough tolerance for mixed versions, delayed consumers, and partial rollouts.
Rollback Is Not a Safety Net
Time is Part of the Design
The design is shown as if version N cleanly becomes version N+1. Production does not work like that. Rollouts take time. Queues introduce delay. Mobile apps stay old. Services restart at different moments. Data written by the new version is read by the old version.
While upgrading, the system has to deal with history, and that is usually where things get messy as hell. It has old data, old clients, and consumers that lag behind the producer. It also carries a bunch of undocumented assumptions that nobody gave a damn about when everything was still small and fresh. Plenty of designs look good when nothing is moving. The moment the system starts changing, you find out which parts were solid and which parts were bullshit.
Rollback Sounds Easier Than It Is
State is different. Once a schema changes, once records are stored in a new structure, or once downstream systems consume a new event format, the path backward becomes much less friendly. Sometimes it is possible. Sometimes it is slow. Sometimes it is dangerous.
If version N and version N-1 cannot work against the same state, then the rollback story is weak. The system has already crossed a one-way door. I remember working on a scheduling system and migrating it from a job-based model to a task-based one, where everything became a job and a job could have multiple tasks. I wrote the implementation in a week. Then I spent the next three weeks testing whether we could roll it back safely. The model change itself was not the expensive part. The expensive part was making sure the system would behave well during the overlap, when old and new logic had to survive each other.
Different parts Move At Different Speeds
Compatibility also matters because producers and consumers are usually not on the same clock. The team running the service wants to simplify things and get rid of old baggage. The teams consuming it have their own deadlines and their own mess. For them, upgrading means work, risk, and a lot of effort for something users may never even notice. So it slips. Then it slips again. That is normal.
The same thing happens inside message-driven systems. A producer starts sending a new payload. A lagging consumer is still catching up on the old backlog. Now both worlds are present in the same queue. If the consumer only understands one of them, failures start appearing in strange places. Retry loops, blocked partitions, and stuck processing pipelines are often signs that the compatibility window was not taken seriously enough.
I saw a similar problem while working on identity and tracking across multiple consumer products. Different apps, sites, and devices all had their own identifiers. Each one made sense locally. The hard part was keeping a stable mapping while the surrounding systems changed at different speeds. Cookies, device IDs, login IDs, and tracking IDs all had different lifetimes and different levels of trust. Once you try to unify them, compatibility becomes part of the design. The identity layer has to cope with delayed consumers, partial adoption, and merge rules without asking every other system to move at once. When that works, analytics and personalization get much easier. When it does not, the edges of the system start drifting apart.
Trouble Usually Starts at the Boundary
A lot of failures start right at the seam between old and new. The field is still there, but it means something slightly different now. The payload is technically valid, but the ordering changed and some downstream code starts choking on it. A new optional field shows up, and some consumer falls over because its parser was more brittle than anyone realized. Sometimes even the telemetry gets messed with during the migration, so the dashboard stays green while the system is quietly going to shit.
This is one reason production systems are harder than they look. The official contract is only part of the picture. Observable behavior matters too. If enough people depend on a behavior, it becomes part of the real interface whether you documented it or not.
Safe Change Usually Takes Extra Steps
Compatibility has a cost. You may need additive schema changes before removal. You may need dual writes for a while. You may need shadow reads, feature flags, or better telemetry on who is still using the old path. You may need to delay cleanup until you are sure the system has actually moved on. None of this feels especially elegant. It does, however, make production change much less dramatic. I am usually happy to pay that cost. A slower and safer transition is often cheaper than saving a week and discovering later that the whole release depended on perfect timing.
Before Changing a Live Boundary
Before making a structural change to a live system, I find it worth slowing down long enough to answer a few uncomfortable questions honestly.
Can the previous version still run safely against the new state? Not in theory, but in practice, with real data written by the new code. Can delayed consumers survive the new payload, including the ones sitting behind a backlog that is hours deep by the time they catch up?
I also want to know whether additive changes have been separated from destructive ones. Mixing them is how a cautious migration becomes a fragile one. And before removing anything, I want to know who still uses the old path, based on telemetry rather than memory.
The last question is the most important. If the rollout goes badly, do I have a real rollback, or only a code rollback? A code rollback that cannot safely read existing state is not a safety net unfortunately. I have seen many systems where the rollback was not possible because the new code changed the state in a way that the old code could not read.
None of this is especially fast. But I would rather answer these questions before a release than during an incident.
In Consequence
Compatibility is one of the things that makes production change manageable. It gives you room for old clients, delayed consumers, mixed versions, and rollback decisions that still work against real state. Once you start designing for that overlap, the system usually becomes easier to trust, because you are finally designing for the way it actually changes.