Neil Crofts is a Principal Software Engineer at the BBC. Currently he works for the Audience Platform Data team responsible for securely processing billions of events generated when audience members sign into the BBC, play videos on iPlayer or interact with any page on the BBC web and mobile sites. Recently he has been behind the development of a flexible, model-driven ETL framework that works at scale across both clustered Big Data and serverless platforms.
The BBC’s Audience Platform Data team collects, transforms and delivers billions of events each day from audience interactions with mobile apps and web sites such as BBC News, BBC Sport, iPlayer and Sounds.
Last year we migrated to a new analytics provider and we took this as an opportunity to re-architect our distributed monolith. We will share the lessons learnt from operating it for nearly 3 years, how we designed our new microservices architecture so that it is easier to test, scale to cater for increasing demand, keep track of the message flow and replay errors without stopping the rest of the messages from being processed.
We will also discuss the ideas behind the tooling we have developed which helps us operate our pipeline and has helped new members of the team share the understanding required to troubleshoot problems.
We have been in production for over a year and as demand from our big data platform increases we are beginning to discuss what our platform may look like in the future and the steps we will go through to achieve it.