Real-time notifications for a VR game

A real-time notification pipeline inside the game's backend, one workstream in a larger platform engagement. It pushes events to players while they are in session: a friend coming online, an invite, an account or session notice, an alert when someone signs in on more devices than they are allowed. Producers do not deliver anything themselves. They write a message to a Redis Stream and return, and a background consumer group drains that stream across however many backend instances are running, sending over AWS API Gateway's WebSocket API, which holds the socket state so the application stays stateless.

• A Redis Stream between the producer and the delivery, so whatever raises a notification writes a message and returns rather than waiting on socket fan-out.
• A background loop started on module init reads the stream as a consumer group, so several backend instances drain the same stream in parallel without two of them sending the same notification.
• Messages left pending by a consumer that crashed or was rolled are auto-claimed after 30 seconds. At-least-once, deliberately: a duplicate notification is a shrug, a missing one is a support ticket.
• Every producer wired in behind one message format: webhooks from upstream platform services, internal events raised by the backend itself, and admin and automation endpoints for pushing a message by hand.
• Live connections tracked in Redis sorted sets keyed by player and scored by connection time, with a reverse map from connection id, so finding a player's oldest session is a range read and disconnect cleanup is O(1).
• Delivery over AWS API Gateway's WebSocket API, with $connect and $disconnect routed to their own controller endpoints, leaving the application holding connection ids and nothing else.
• Redis and the WebSocket API provisioned as CDK constructs, so the infrastructure ships and versions with the service rather than beside it.

• A queue between the producer and the delivery

  Sending inline would have put socket fan-out on the response time of whatever raised the event, and made every delivery failure the caller's problem. Writing to a stream keeps the producer fast and turns delivery into something that can be retried, observed and scaled on its own terms.

• Let a managed service hold the sockets

  Connection state is what makes a service stateful and awkward to deploy. API Gateway's WebSocket API owns the sockets, the application owns connection ids in Redis, and the result scales horizontally with no sticky sessions and no draining connections at every release.

• At-least-once, and say which way it fails

  Consumer groups give each message a single owner across every running instance, plus a pending list to reclaim from. An instance dying mid-delivery then costs a delay rather than a message. That is a choice about which failure is acceptable, and it is better made in the design than discovered in an incident.

• Shape the connection state around the question

  The question the connection limits keep asking is which of a player's sessions is oldest. Scoring connections by timestamp in a sorted set answers it with a range read instead of a scan, and the reverse map keeps the disconnect path constant time no matter how many sessions a player has open.

The WebSockets were never the interesting part. A notification that arrives after the session has ended may as well not have been sent, so the whole design is about the awkward cases rather than the happy path: the player who is offline, the instance that dies holding a message, the player signed in on three headsets at once. Putting a stream between the producer and the delivery is what turned those from incidents into states, because each one becomes a message that is still pending and a consumer that can pick it up. The other half was refusing to own the sockets. Connection state is the thing that stops a service scaling, and handing it to a managed service kept the problems worth solving in our code and left the tedious one somewhere else.