The BBC’s Audience Platform Data team collects, transforms and delivers billions of events each day from audience interactions with mobile apps and web sites such as BBC News, BBC Sport, iPlayer and Sounds.
Last year we migrated to a new analytics provider and we took this as an opportunity to re-architect our distributed monolith. We will share the lessons learnt from operating it for nearly 3 years, how we designed our new microservices architecture so that it is easier to test, scale to cater for increasing demand, keep track of the message flow and replay errors without stopping the rest of the messages from being processed.
We will also discuss the ideas behind the tooling we have developed which helps us operate our pipeline and has helped new members of the team share the understanding required to troubleshoot problems.
We have been in production for over a year and as demand from our big data platform increases we are beginning to discuss what our platform may look like in the future and the steps we will go through to achieve it.
Blanca Garcia Gil is a principal systems engineer at BBC. She currently works on a team whose aim is to provide a reliable platform at petabyte scale for data engineering and machine learning. She provides leadership on ensuring that the development team has the correct infrastructure and tooling required for the entire delivery and support cycles of the project. Prior to the BBC she has had a variety of roles from developing web applications for an agency, mobile prototyping (before smartphones came about!), developing a content management system or writing highly scalable APIs. She always enjoyed working closer to the backend and since she started developing in the Cloud she's taken on the challenge of learning about infrastructure. This has led to her deciding to move recently to an infrastructure automation and reliability role.
Neil Crofts is a Principal Software Engineer at the BBC. Currently he works for the Audience Platform Data team responsible for securely processing billions of events generated when audience members sign into the BBC, play videos on iPlayer or interact with any page on the BBC web and mobile sites. Recently he has been behind the development of a flexible, model-driven ETL framework that works at scale across both clustered Big Data and serverless platforms.