Sign upSign InSign upSign InMember-only storyEdward HuangFollowLevel Up Coding--ShareA group of brilliant engineers in Google led by Paul Nordstrom wants to create a system that does the streaming data process that MapReduce did for Batch data processing. They wanted to provide a robust abstraction and scale to a massive size.Building MillWheel was no easy feat. Testing and ensuring correctness in the streaming system was especially challenging because it couldn’t be rerun like a batch pipeline to produce the same output. As if that wasn’t enough, the Lambda architecture complicated matters further, making it difficult to aggregate and reconcile streaming and batch results. Out of such adversity, Google Dataflow was born- a solution combining the best of both worlds into one unified system serving batch and streaming pipelines.Creating and designing pipelines is a different thought process and framework from writing custom applications. For the past few months, I have spent numerous days and weeks learning the fundamentals and concepts of Apache Beam and Dataflow job to build a dataflow pipeline for my projects.There aren’t as many articles that briefly introduce Dataflow, Apache Beam, and Scio that you can read while commuting by train or bus to work. Thus, I hope this article helps all beginners like me to wrap their heads around these concepts.Dataflow is a serverless, fast, cost-effective service that supports stream and batch processing. It provides portability with processing jobs written using the open-source Apache Beam libraries. Automating infrastructure provisioning and cluster management removes operational overhead from your data engineering teams.A lot of the data processing usually works by source input, transformation, and a sink. Engineers developed the pipeline and the transformation in the data flow template. They can use the template to deploy and execute a Dataflow job pipeline. Dataflow then assigns the worker virtual machines to execute the data processing, and you can customize the shape and size of these machines.For instance, to do a batch processing pipeline for the daily user score in a game, the source will be an…----Level Up CodingDocument my journey in technology, functional programming, and careers in tech. Read my articles for free @ https://pathtosenior.substack.com/Edward HuanginBetter Programming--14Arslan MirzainLevel Up Coding--5Jacob BennettinLevel Up Coding--67Edward HuanginLevel Up Coding--7Michael LonginBetter Programming--20Dominik PolzerinTowards Data Science--6Love SharmainDev Genius--31Kai Waehner--JP Brown--267Tech Wisdom--HelpStatusWritersBlogCareersPrivacyTermsAboutText to speechTeams