What it was
Players needed to hear about friends coming online, invites and account or session warnings while they were in the game. I built the pipeline inside the backend, using a Redis Stream to hold the work and AWS API Gateway's WebSocket API to deliver it. The code raising an event writes a message and carries on. Consumers on the backend instances handle sending it. API Gateway keeps the sockets open, so the application only needs to track connection ids.
Build
- A Redis Stream sits between the code raising a notification and the code sending it. Writing to the stream doesn't wait for every socket delivery.
- A consumer group distributes messages across the backend instances. Each instance starts a background delivery loop when its module initialises.
- Startup reclaims messages pending for more than 30 seconds. Graceful shutdown drains the delivery loop. Delivery is at least once, so retries can produce duplicates.
- Upstream webhooks, backend events and the admin automation endpoints all use the same message format.
- Redis tracks live connections and a reverse lookup by connection id. A teammate later moved session lists into sorted sets scored by connection time, which made it easy to find the oldest session for device limits.
- API Gateway owns the WebSocket connections. The service handles its connect and disconnect events and sends through the stored ids.
- The Redis and WebSocket infrastructure is defined in CDK alongside the service.
Key decisions
WebSockets
We recommended long-polling because it was ordinary HTTP with less to operate. The client preferred WebSockets and wanted the option of bidirectional messages later. We went with that and used a managed service to hold the connections.
Buffering delivery
Sending inline would make the producer wait for socket fan-out and deal with any failures. The stream lets the delivery work be retried and scaled independently.
Connection state
Keeping only connection ids in the application means a release doesn't have to move or drain all the sockets. API Gateway holds those while the backend instances come and go.
Retries
Consumer groups keep a pending list when an instance fails. I accepted the possibility of a duplicate so a failed send could be retried.
Lesson learned
A stranded message could wait until the next deployment.the rest of it ↓fold it back up ↑
The recovery loop is the thing I'd change. A consumer reclaimed old pending messages when it started, but didn't keep checking for them after that. If another instance crashed while holding a message, that message could sit there until the next deployment. I'd add periodic reclaim and a dead-letter stream for deliveries that keep failing. I'd also prune a connection as soon as API Gateway reports that it's gone.