Data should be written as text to hdfs

Data should be written as text to hdfs

View Answer

Answer: Solution:

Step 1: Create directory mkdir /tmp/spooldir2

Step 2: Create flume configuration file, with below configuration for source, sink and channel and save it in flume8.conf.

agent1 .sources = source1

agent1.sinks = sink1a sink1b agent1.channels = channel1a channel1b

agent1.sources.source1.channels = channel1a channel1b

agent1.sources.source1.selector.type = replicating

agent1.sources.source1.selector.optional = channel1b

agent1.sinks.sink1a.channel = channel1a

agent1 .sinks.sink1b.channel = channel1b

agent1.sources.source1.type = spooldir

agent1 .sources.sourcel.spoolDir = /tmp/spooldir2

agent1.sinks.sink1a.type = hdfs

agent1 .sinks, sink1a.hdfs. path = /tmp/flume/primary

agent1 .sinks.sink1a.hdfs.tilePrefix = events

agent1 .sinks.sink1a.hdfs.fileSuffix = .log

agent1 .sinks.sink1a.hdfs.fileType = Data Stream

agent1 . sinks.sink1b.type = hdfs

agent1 . sinks.sink1b.hdfs.path = /tmp/flume/secondary

agent1 .sinks.sink1b.hdfs.filePrefix = events

agent1.sinks.sink1b.hdfs.fileSuffix = .log

agent1 .sinks.sink1b.hdfs.fileType = Data Stream

agent1.channels.channel1a.type = file

agent1.channels.channel1b.type = memory

step 4: Run below command which will use this configuration file and append data in hdfs.

Start flume service:

flume-ng agent -conf /home/cloudera/flumeconf -conf-file /home/cloudera/flumeconf/flume8.conf –name age

Step 5: Open another terminal and create a file in /tmp/spooldir2/

echo "IBM, 100, 20160104" » /tmp/spooldir2/.bb.txt

echo "IBM, 103, 20160105" » /tmp/spooldir2/.bb.txt mv /tmp/spooldir2/.bb.txt /tmp/spooldir2/bb.txt

After few mins

echo "IBM.100.2, 20160104" »/tmp/spooldir2/.dr.txt

echo "IBM, 103.1, 20160105" » /tmp/spooldir2/.dr.txt mv /tmp/spooldir2/.dr.txt /tmp/spooldir2/dr.txt