Rui ZhouFollowBetter Programming--ListenShareA practice to Ingest the data in real-time from Kafka cluster to the Hadoop/HDFS platformIt is quite a common requirement to ingest the data from the microservice cluster to the big data platform for further analytics. Depending on the data platform architecture, it can ingest data to either the object store(s3) or Hadoop/HDFSThis tech stack is a bit outdated, but lots of companies are still using it nowadays.object store/s3 as data lake is more popular when the data platform is deployed on the public cloud. No matter which deployment, there is not much difference in the data-sink(ingestion to data platform) part. Both of them will use Kafka-Connect to ingest data to data platform. I will give an example based on the Hadoop solution.I created a docker-compose file run Hadoop and Kafka. refer togithub.comand https://github.com/big-data-europe/docker-hadoopdocker-compose.yamlhadoop.envopen localhost:9021 , Click Add topiccreate topic data-sink-hdfsopen localhost:9021 then goto HOME/Connect-Cluster/Connectorsadd a new connectorin this case we choose HdfsSinkConnector, and configure the connector as below. “key.converter.schemas.enable” and “value.converter.schemas.enable” are additional property that needs to be added at the bottom of the pageproduce a messagedocker attach to namenode then checkIn the next article, I will explore how to save the message in parquet format and use data visualization tools to query on parquet file.Happy Coding!----Better Programminghttps://www.linkedin.com/in/rui-zhou-sg/Rui ZhouinBetter Programming--VinitainBetter Programming--38Benoit RuizinBetter Programming--222Rui ZhouinJavarevisited--Mahesh SainiinLevel Up Coding--11Shawn.Yang--7Dmitrii EliuseevinTowards Data Science--1Mahdi MallakiinITNEXT--6Gavin Fong--2Analytics at Meta--19HelpStatusAboutCareersBlogPrivacyTermsText to speechTeams