The bug report that started this is one every product with notifications eventually receives, and it is almost impossible to act on as written. A user gets a push on their phone, opens the app, and the thing they were notified about is not in their notification list. Or the reverse: an item sits in the list that never reached their phone, so they found it three days late.
You cannot reproduce it on demand. It does not correlate with a device, a platform or a release. Every individual instance has a local explanation, such as a race, a retry, or a filter applied on one side and not the other, so it gets fixed individually for as long as anyone is prepared to keep fixing it.
I owned the notification and real-time system at Cendra, across four repositories, and the design decision I would most want to be questioned about is the one that made that class of report stop existing.
Two producers is the shape of the problem
The original arrangement was the natural one. A domain event happens. Something decides a push should go out and sends it. Something else decides an in-app notification should exist and writes it. Two code paths, each reasonable, each with its own view of who should be told and when.
Two independent producers of the same fact will diverge, and it happens through ordinary drift rather than carelessness. A preference check gets added to one path because that is where the complaint came from. A recipient filter gets refined on the other. A retry succeeds on one side and not the other. Someone adds a new event type and wires it into the path they were already looking at. Each of those changes is locally correct, and together they guarantee the two channels will eventually describe different worlds.
That is why fixing the instances never converges. Every fix closes one route to disagreement while leaving the capacity for disagreement in place.
One row
So: one outbox record per notification, driving both push and the in-app notification centre. The event-processing consumer classifies and routes a domain event, and writes one row. The realtime hub reads it to push a live update to the web client, and downstream of the hub the same row drives delivery to iOS and Android. The in-app notification centre is a view of that same row.
There is no second producer, so there is nothing for the first one to disagree with. Divergence stops being a bug that gets fixed and becomes a state the data model cannot represent.
I have come to think that distinction, a bug you fix versus a state that cannot exist, is the most useful lens I have for judging a design. Two implementations that agree because someone keeps them in step are a maintenance commitment with a person's attention as its dependency. One implementation that cannot disagree with itself is a property. The property survives the team changing. The commitment does not.
What it cost
A shared record is a constraint, and I would rather state the constraint than present the decision as free.
Once one row serves every channel, every channel-specific concern has to be expressed as data on that record rather than as logic inside a private producer. Push has a payload shape, a collapse identity and a size limit. The in-app centre has read state, grouping and a lifetime measured in weeks rather than seconds. Those are genuinely different requirements meeting on one schema, and the schema has to carry enough for both without becoming a bag of channel-specific columns.
That makes the outbox a contract, and contracts are slower to change than private code. Adding a channel means thinking about the record rather than writing a new producer, which is more work on the day and much less work every day after.
I still think it is the right trade. The alternative buys its speed with a permanent capacity for the two channels to lie about each other.
Everything around the row
The record is the spine. The behaviour that makes a notification system tolerable lives in the machinery around it: fan-out with deduplication and stable grain keys, burst suppression, preference gating, rate limiting, collapse keys, recipient filtering, and device-token lifecycle management.
Two of those are worth pulling out, because they are the ones people discover late.
Stable grain keys are the decision about what counts as one notification. Ten field updates to the same underlying record inside ninety seconds add up to one situation, and a system that sends ten notifications for it has made a modelling error before it has made a throttling error. Choosing the grain is where a notification system either respects a person's attention or does not. Getting the grain right removes volume that no rate limiter should have had to remove.
Collapse keys are the same idea at the delivery edge: a later update replaces an earlier undelivered one for the same subject rather than stacking behind it. Without them, a user who has had their phone in a pocket for an hour is punished for it on unlock.
I wrote the entire test suite for this system, seventy-nine of seventy-nine commits to the notification test directory. I mention it for a reason other than volume. Notification behaviour is mostly conditional logic over preferences, state and time, which is expensive to verify by hand on a real device and cheap to pin in tests. Anything I want to be structurally true needs a test that fails when it stops being true. Otherwise it is a preference I happen to hold.
The ordering I got wrong
My initial design treated delivery as the problem and suppression as a later refinement.
That ordering was mine and it was wrong. A notification system needs a suppression model before it needs a delivery model, because volume is easy to add and very hard to take back once users have adjusted to it. Attention trained away from a channel does not come back when the channel improves. I had built the satisfying half first, and the half that decides whether anybody still has notifications enabled in a month second.
Correcting it was a three-phase programme: event classification to separate genuinely notifiable events from telemetry, then preference gating, then rate limiting, recipient filtering and collapse keys. The result was approximately nineteen thousand unnecessary push notifications a day eliminated. That figure needs its method attached, because a round number invites the assumption that somebody watched a counter for a day. It was measured in production log telemetry over a ten-minute observation window against the prior baseline and then scaled to a daily rate. Nobody observed a twenty-four-hour count directly. Alongside it, zero degraded events and zero authentication failures, which is the part I cared about most. A volume reduction that also drops something real is an incident with a good headline.
When I later built a proactive assistant of my own, I built the suppression model first and made it a single ordered gate with every rejection logged. That essay is the argument for doing it in that order, and it exists because I did it in the other order once.
Two boundaries I want to be precise about
The event-processing consumer lives in a .NET backend I joined roughly six months into a twelve-month tenure. That codebase had been under active development for eighteen months before I touched it, by engineers who remain there. I was a substantial contributor to it, and I was not its architect. My share of commits since joining runs highest in the test suite and the event-processing consumer, which is a fair description of where I worked. It is not a claim to have designed the system those parts sit inside. The outbox decision above is a design decision I made and argued for within a codebase that was not mine to have designed, and both halves of that sentence matter.
The delivery edge is a different story. I owned both mobile applications entirely, including Firebase push, the over-the-air update chain and release through App Store Connect and Google Play. The most instructive thing that happened there was a small feature I built and then deliberately removed, which says more about how I work than the outbox does.
The question I would want asked
Why one row rather than two producers, and what would have to be true for the answer to be different?
I think the answer changes when the two channels carry different facts rather than the same fact through different transport. A marketing send and a transactional alert are two different notifications and should not share a record. Push and in-app were always the same fact, and every hour spent reconciling them was rent on a decision nobody had made deliberately.
More on that role, including the release process and what I did not own: Cendra.