Data should be written as text to hdfs

Data should be written as text to hdfs

View Answer

Answer: Solution:

Step 1: Create directory mkdir /tmp/nrtcontent

Step 2: Create flume configuration file, with below configuration for source, sink and channel and save it in flume6.conf.

agent1 .sources = source1

agent1 .sinks = sink1

agent1.channels = channel1

agent1 .sources.source1.channels = channel1

agent1 .sinks.sink1.channel = channel1

agent1 . sources.source1.type = spooldir

agent1 .sources.source1.spoolDir = /tmp/nrtcontent

agent1 .sinks.sink1 .type = hdfs

agent1 . sinks.sink1.hdfs .path = /tmp/flume

agent1.sinks.sink1.hdfs.filePrefix = events

agent1.sinks.sink1.hdfs.fileSuffix = .log

agent1 .sinks.sink1.hdfs.inUsePrefix = _

agent1 .sinks.sink1.hdfs.fileType = Data Stream

Step 4: Run below command which will use this configuration file and append data in hdfs.

Start flume service:

flume-ng agent -conf /home/cloudera/flumeconf -conf-file /home/cloudera/fIumeconf/fIume6.conf –name agent1

Step 5: Open another terminal and create a file in /tmp/nrtcontent

echo "I am preparing for CCA175 from ABCTech m.com " > /tmp/nrtcontent/.he1.txt

mv /tmp/nrtcontent/.he1.txt /tmp/nrtcontent/he1.txt

After few mins

echo "I am preparing for CCA175 from TopTech .com " > /tmp/nrtcontent/.qt1.txt

mv /tmp/nrtcontent/.qt1.txt /tmp/nrtcontent/qt1.txt