apache-flume1.6 sink默认支持kafka

• [] - Flume Sink and Source for Apache Kafka

官方给的例子很贴心，可以直接运行=，=，详细配置之后慢慢看。

a1.channels = channel1

a1.sources = src-1

a1.sinks = k1

a1.sources.src-1.type = spooldir  #实时性要求不高的话，可以用这种方式，tail实际效率也挺低的

a1.sources.src-1.channels = channel1

a1.sources.src-1.spoolDir = /opt/flumeSpool/

a1.sources.src-1.fileHeader = false

#a1.sources.src-1.type = exec

#a1.sources.src-1.command = tail -F /opt/flumeSpool/abc.log

#a1.sources.src-1.channels = channel1

a1.sinks.k1.type = org.apache.flume.sink.kafka.KafkaSink

a1.sinks.k1.topic = testlog

a1.sinks.k1.brokerList = 1.1.1.11:9092,1.1.1.12:9092

a1.sinks.k1.requiredAcks = 1

a1.sinks.k1.batchSize = 20

a1.sinks.k1.channel = channel1

a1.channels.channel1.type   = org.apache.flume.channel.kafka.KafkaChannel

a1.channels.channel1.capacity = 10000

a1.channels.channel1.transactionCapacity = 1000

a1.channels.channel1.brokerList=1.1.1.11:9092,1.1.1.12:9092

a1.channels.channel1.topic=testlog

a1.channels.channel1.zookeeperConnect=1.1.1.13:2181

flume启动命令：

../bin/flume-ng agent --conf-file flume-conf.properties --name myagent -Dflume.monitoring.type=http -Dflume.monitoring.port=34545