In this talk I will be exploring how to use Apache Kafka’s APIs to build transactional microservices, allowing distributed systems to meet processing guarantees.
If there's one thing we know about software, it is that eventually something will fail, and you many lose data.
This is not the end. Designing for failure has brought us many useful innovations like the TCP protocol, the Erlang programming language, and even Apache Kafka itself.
It's so important, that databases have enshrined these properties as ACID compliance.
But what happens when there is more than one system in your transaction? Sometimes it's more than just a two phase commit. When dealing with payment systems, being able to act upon a stream continuously while maintaining ACID & exactly once semantics is mandatory.
Follow along as we see what happens when systems start to degenerate, and what we can and can’t trust at scale. Learn about when to use Kafka as the transaction controller, what can and can’t be stateless, and what are the tradeoffs.
We'll explore completion criteria, the routing slip pattern, the outbox pattern, and others as we go on a trip through the various methods of ensuring ACID (atomicity, consistency, isolation, and durability) compliance and exactly once processing, in an asynchronous distributed system. Leave with a few extra tools and patterns about how to make large scale systems reliable
If there's one thing we know about software, it is that eventually something will fail, and you many lose data.
This is not the end. Designing for failure has brought us many useful innovations like the TCP protocol, the Erlang programming language, and even Apache Kafka itself.
It's so important, that databases have enshrined these properties as ACID compliance.
But what happens when there is more than one system in your transaction? Sometimes it's more than just a two phase commit. When dealing with payment systems, being able to act upon a stream continuously while maintaining ACID & exactly once semantics is mandatory.
Follow along as we see what happens when systems start to degenerate, and what we can and can’t trust at scale. Learn about when to use Kafka as the transaction controller, what can and can’t be stateless, and what are the tradeoffs.
We'll explore completion criteria, the routing slip pattern, the outbox pattern, and others as we go on a trip through the various methods of ensuring ACID (atomicity, consistency, isolation, and durability) compliance and exactly once processing, in an asynchronous distributed system. Leave with a few extra tools and patterns about how to make large scale systems reliable
Ben Gamble
Aiven
Ben has spent over 10 years leading engineering in startups and high-growth companies. As a founder, a CTO, a producer and a product leader he's bridged the gap between research and product development. Having worked with the cutting edge of augmented reality, scaling 3D gaming, and same day logistics, he’s no stranger to taking on technical challenges, and the commercial realities that entails. Ben now works to make real-time data a reality for anyone who needs it with open source tools and shared ideas.