Video: Containerizing Distributed Pipes

Print Friendly, PDF & Email

Hagen Toennies, Gaikai

In this video from the 2017 HPC Advisory Council Stanford Conference, Hagen Toennies from Gaikai Inc. presents: Best Practices – Containerizing Distributed Pipes.

“In this talk we will present how we enable distributed, Unix style programming using Docker and Apache Kafka. We will show how we can take the famous Unix Pipe Pattern and apply it to a Distributed Computing System. We will demonstrate the development of two simple applications with the focus on “Do One Thing and Do It Well.” Afterwards we demonstrate how we make these two programs work to together using Apache Kafka. By encapsulating our applications in containers we will also show how that enables us to go from the limited resources of a development machine to cluster of computers in a data center without changing our applications or containers.”

Hagen Toennies joined Gaikai Inc in 2016 where he is focused on resilient Data Pipeline services to deliver the right data to the right place in the right time. For more than five years, from 2011 until joining Gaikai, Hagen gained experience in search engine scaling and applied his expertise in building custom retrieval systems to deliver search applications for Germany’s news publishing sites. His work with the German Federal Statistical Office introduced challenges in scaling database systems and maintaining a clean codebase led to the 2005 launch of Podcast Ready Inc. where the startup co- founder gained practical experience in software development and entrepreneurialism.

Highly specialized in Information-Retrieval and Web-Technology, Hagen holds a Master of Systems Science from the Bauhaus University in Weimar, Germany. In addition to enjoying most of his free time with his family his remaining time is spent dedicated to the Gödel-Escher-Bach book, being a (lisp) fan and a Clojure enthusiast – a JVM based (lisp).

Download the Slides (PDF)

See more talks in the Stanford Conference Video Gallery

Sign up for our insideHPC Newsletter