Apache Flume – Ingesting log data into Hadoop and Kafka
Free Download Udemy Apache Flume – Ingesting log data into Hadoop and Kafka. With the help of this course you can Detailed workshop about using Flume to ingest web server logs into live Hadoop and Kafka Cluster..
This course was created by Durga Viswanatha Raju Gadiraju. It was rated 4.7 out of 5 by approx 4027 ratings. There are approx 49011 users enrolled with this course, so don’t wait to download yours now. This course also includes 3 hours on-demand video, Full lifetime access, Access on mobile and TV & Certificate of Completion.
What Will You Learn?
Understand basics of Flume
Implement simple flume agent
Understand multiple flume agent flows
Setup multiple agents using Avro as connector
Setup agent to get log data into HDFS
Understand sources like netcat, avro and exec in detail
Understand channels like memory and file in detail
Understand sinks like logger, avro and HDFS in detail
As part of this session we will understand how we can use Apache Flume to ingest streaming real time data in detail.
Overview of Flume
Setting up gen_logs
Develop first Flume Agent
Understand Source, Sink and Channel
Flume Multi Agent Flows
Get data into HDFS using Flume
Limitations and Conclusion
For this demo we will be using our Big Data developer labs. You need to have access to existing big data cluster or sign up to our labs.
Hands On demos:
Developing simple Flume agent to get data from netcat to agent logs
Develop multi agent flow where data from web server logs go to avro sink and then from avro source to logger
Develop multiplex flow where data from web server logs is written to HDFS and Kafka