Speaker Details

Pedro Tavares


Pedro is a passionate Software Engineer. Distributed Systems and Software Engineering best practices bring a big smile on his face. He likes to read books and scientific papers on Computer Science and he's currently running the Porto's chapter of Papers We Love.

Tales from running Kafka Streams in Production

Data & AI

Have you chosen Kafka as your distributed streaming platform? You went for the extra mile and started processing your data with Kafka Streams? You made the right choice. Kafka Streams gets handy for several reasons, still, some specific problems only appear with heavy load in production. After a full year of using Kafka Streams in Production on top of Kubernetes, and making heavy usage of the Processor API and State Stores, I've documented the incidents we had over the year and now you can prevent it from happening in your workplace. In this talk you'll learn about planning the partitions in advance to increase the platform throughput, how persistent schemas might cause you an headache, how to handle the computed changelog properly, how to make external requests using the StateStore interface and specially how to manage the beast, off-heap memory.

Apache Kafka
Apache Kafka Streams