A series of successful stumbles.

Lesson learned

I wrote architecture decision records alongside this. Trying to explain why I'd chosen something changed a few of the decisions, which I hadn't expected when I started writing them. They also leave the rejected options in the repo, so someone reading it can see what I considered. I didn't get the role. Still, it gave me a week to get into Unity's build tooling and licensing, both of which had rather more going on than I'd realised.

Lesson learned

I underestimated the time we'd need with those teams. The ETL changes were manageable, agreeing what 'the same number' meant after the switch took much longer. Different people were using the outputs to answer different questions. I'd bring those definitions into the planning much earlier next time.

Lesson learned

By this point the teams had seen the numbers disagree often enough that they were hesitant to use them. Fixing a conversion didn't answer all their questions about the rest of the report. I spent about as long making the corrections traceable as I did making them, so somebody could follow a figure back through the pipeline and see how we'd arrived at it.

Lesson learned

The recovery loop is the thing I'd change. A consumer reclaimed old pending messages when it started, but didn't keep checking for them after that. If another instance crashed while holding a message, that message could sit there until the next deployment. I'd add periodic reclaim and a dead-letter stream for deliveries that keep failing. I'd also prune a connection as soon as API Gateway reports that it's gone.

Lesson learned

Moving timings to histograms and percentiles showed bad frames that the averages had hidden. We had to be just as careful with the collector's own timings, including the occasional slow tick. There was a lot to agree across the three SDKs too. A metric with the same name needs the same units and behaviour in each language, or a studio looking at the shared dashboard can end up comparing different things.

Lesson learned

I initially went looking for one place to enforce access. The launcher is on the player's machine though, so they can change what it does. A backend check doesn't stop somebody going straight to the file URL. And if the CDN refuses a request, the player still needs to understand why their download didn't start. There was some pushback on repeating the checks. Explaining the job of each one helped us get past that, particularly the difference between telling a player they can download and actually protecting the file.

Lesson learned

I'd assumed the platform would get adopted once it made the common jobs easier. Teams still needed to see how it fitted their own services though. The Champions Network gave them a colleague to ask, someone who could work through the standards in the context of that team's code.

Lesson learned

About half of this work was teaching. I needed to build the relationship and earn some credibility before turning up with a different way of doing everything. We started with the SQL the analysts already knew, moved into Python that felt familiar and brought in the command line when they had a reason to use it. The setup had to work well too. An experienced engineer might work around a strange error, somebody learning can lose an afternoon to it. We spent time on the development environment and put the tests and quality checks into the normal workflow. The repository is public and still being developed years later. It includes tooling choices the team made after our engagement.

Lesson learned

The public sector design and style guides are rock solid. They're intuitive and straightforward to build against, I came away with a lot of respect for them. The risk culture was harder going though. End-to-end tests took 90 minutes and the rollback path was cumbersome enough that people were reluctant to use it. Mistakes still got through. I wanted us to be able to deploy a fix in ten minutes when that happened.

Lesson learned

The weekly demos made this the best delivery I'd worked on. We gathered feedback and made decisions with the client in the room, and I've kept that habit since. Late on they mentioned wanting a 'dual-ledger' system. We took the phrase at face value and spent six weeks building double-entry accounting, including journals and tax handling. It turned out they'd read the term somewhere and weren't sure what it meant. They wanted a simple finance ledger, about a week's work. The extra accounting machinery stayed in the product. Every later finance feature had to work through it, and we felt that for months. We'd skipped the demo and the questions on this part of the system.

Lesson learned

Hermes started with a gap I wanted fixed. Once we'd got something going, it caught senior management's attention and became part of the official roadmap. Seeing an idea I'd argued for become work the wider team wanted to keep was new to me. I also learnt more on this than on a lot of the projects I'd been assigned. Containers, distributed processing and observability were all things I had to get working together.

Lesson learned

I had to defend the research to people whose job was to question it. I'd been used to checking whether something I'd built worked. This time I had to explain what the evidence supported and where my conclusions were limited.