Data pipelines for the adult social care workforce

Extensible data pipelines and machine-learning models supporting the UK adult social care workforce: AWS infrastructure in Terraform, models developed in Jupyter across a small Spark cluster and then productionised properly, with a full suite of unit, integration and quality tests. The repository is public and still being worked on.

• There was no AWS account when I arrived and no infrastructure to speak of: the data moved around as flat files in Dropbox. The first job was real cloud infrastructure defined in Terraform, reviewable and rebuildable rather than clicked together in a console.
• Spark jobs organised as modular Python under a project-per-pipeline layout, each with its own unit tests, so a new pipeline had an obvious shape to copy.
• A deliberate two-track workflow: Jupyter over a small Spark cluster for exploration and model development, modular tested Python for anything that ran in production, and a clear crossing point between them.
• Governance wired into the daily loop rather than written into a policy document: pre-commit hooks, a style guide, unit and integration tests, data quality checks and CI on every change.
• The engagement ran roughly half build and half teaching, with the pipeline and the team's ability to run it treated as one deliverable.

• Teach the tools they would keep, not the ones I liked

  The analysts arrived fluent in Excel and SQL and had not used a terminal. The temptation is to treat that as a gap to be corrected, but their SQL was real modelling skill and the domain knowledge was theirs entirely. So the path ran from what they already knew: SQL first, then Python that looked like the SQL they were writing, then the command line as the thing that made the rest possible. Starting from their strengths made the new tools feel like more reach rather than a replacement.

• Developer experience was a requirement, not a nicety

  For a team new to software, every rough edge is a reason to go back to a spreadsheet. Setup that fails silently, a test suite that takes an unclear incantation to run, an error message with no next step: each of those is a small tax on a confident engineer and a hard stop on someone learning. Time spent on the paved path was the highest-leverage work in the engagement.

• Governance had to be automatic to survive

  Standards that depend on someone remembering them do not outlast the consultant who introduced them. Putting the checks in pre-commit and CI meant the rules enforced themselves and the feedback arrived while the change was still in hand, which is also when it teaches the most. The aim was a team that had absorbed the habits, not one following a document.

• The measure of success was what happened after I left

  Three analysts with no prior software experience ended up owning a large multifaceted pipeline. Years on, the repository is public, actively developed and has moved well beyond what we built, with tooling choices the team made on their own. It still produces the workforce estimates the adult social care sector in England relies on, which is the outcome that mattered.

Tuition was as much the deliverable as the pipeline. What made it work was building the relationship first and earning credibility before writing anything, then taking the team on the journey - why SQL, why Python, why this shape - rather than handing it over. They could barely use a command line at the start, so the developer experience and the process had to be genuinely strong: Jupyter notebooks for exploration, modular Python for anything going to production, and a clear line between the two. Three analysts went from zero software experience to owning a large, multifaceted pipeline.