What it was
When I arrived there wasn't an AWS account yet. The analysts were moving data around as flat files in Dropbox. They knew the adult social care data inside out, and they were good with Excel and SQL. Software development was new to them. We built the AWS infrastructure in Terraform and wrote Python and Spark pipelines for the workforce models. Models could be explored in Jupyter, then moved into tested jobs when they were ready to run in production. The three analysts ended up running and changing those pipelines themselves.
Build
- Terraform defined the AWS infrastructure so changes could be reviewed and the environment rebuilt.
- Spark jobs were written as modular Python, with one project per pipeline and unit tests alongside it.
- Jupyter ran over a small Spark cluster for exploration and model development. Repeatable production work moved into the Python jobs.
- Pre-commit hooks, a style guide, unit and integration tests, data quality checks and CI were introduced as part of the daily workflow.
Key decisions
Starting from SQL
The analysts already used SQL comfortably. We worked from that knowledge when introducing Python instead of making every tool and concept new at once.
Notebooks and production jobs
Jupyter was useful for pulling a dataset apart and trying a model. We kept a clear handoff into modular Python with tests when that work needed to become a production pipeline.
Development setup
The team needed to be able to install, test and change the jobs after I'd left. We worked on those steps while teaching them, which showed us where the confusing parts were.
Lesson learned
Half the work was teaching the team who would own it.the rest of it ↓fold it back up ↑
About half of this work was teaching. I needed to build the relationship and earn some credibility before turning up with a different way of doing everything. We started with the SQL the analysts already knew, moved into Python that felt familiar and brought in the command line when they had a reason to use it. The setup had to work well too. An experienced engineer might work around a strange error, somebody learning can lose an afternoon to it. We spent time on the development environment and put the tests and quality checks into the normal workflow. The repository is public and still being developed years later. It includes tooling choices the team made after our engagement.