Pushing Java to the Limits: Processing a Billion Rows in under 2 Seconds
Conference (INTERMEDIATE level)
Room B
Last January a challenge was posted online by Gunnar Morling:
How fast can you parse a file with one billion rows of weather data using Java?
Little did I know this deceivingly simple question would lead me down a path that taught me all about: parallelism, memory mapped files, SWAR techniques (SIMD as a register), bit twiddling, branchless code, mechanical sympathy, Graal native compilation and finally... I even turned to the dark side: using sun.misc.Unsafe.
Join me in this deep dive where I'll explain all the code changes and tricks that took me from the reference implementation which processes the billion records in >4 minutes, to processing everything in under two seconds.
Who knew Java could be _this_ fast?
How fast can you parse a file with one billion rows of weather data using Java?
Little did I know this deceivingly simple question would lead me down a path that taught me all about: parallelism, memory mapped files, SWAR techniques (SIMD as a register), bit twiddling, branchless code, mechanical sympathy, Graal native compilation and finally... I even turned to the dark side: using sun.misc.Unsafe.
Join me in this deep dive where I'll explain all the code changes and tricks that took me from the reference implementation which processes the billion records in >4 minutes, to processing everything in under two seconds.
Who knew Java could be _this_ fast?
Roy van Rijn
OpenValue
Roy van Rijn is a director at OpenValue Rotterdam and a Java Champion. He worked on numerous projects all over the Netherlands as developer, architect and agile coach. He talks at conferences all around the world, CodeOne (Rockstar), Devoxx BE & UK & PL, GOTO, Joy of Coding, and local JUG events. You can read more at his blog (http://www.royvanrijn.com) or follow him on Twitter (@royvanrijn).