flume負載均衡，容錯，監控

文章目錄

• • 串聯flume
  • exec-avro-agent.conf 用戶端
  • avro-logger-agent.conf 服務端
• 負載均衡
• • 用戶端
  • 兩個server端
• 容錯
• • 監控

串聯flume

用avro串聯

exec-avro-agent.conf 用戶端

#agent
exec-avro-agent.sources = exec-source
exec-avro-agent.channels = memory-channel
exec-avro-agent.sinks = avro-sink

#source
exec-avro-agent.sources.exec-source.type = exec
exec-avro-agent.sources.exec-source.command = tail -F /home/hadoop/app/flume-1.6.0-cdh5.7.1/logs/1.log

#channel
exec-avro-agent.channels.memory-channel.type = memory

#sink
exec-avro-agent.sinks.avro-sink.type = avro
exec-avro-agent.sinks.avro-sink.hostname = localhost
exec-avro-agent.sinks.avro-sink.port = 55555


#bind source and sink to channel
exec-avro-agent.sources.exec-source.channels = memory-channel
exec-avro-agent.sinks.avro-sink.channel = memory-channel
           

啟動腳本

./flume-ng agent --conf /home/hadoop/app/flume-1.6.0-cdh5.7.1/conf --conf-file /home/hadoop/app/flume-1.6.0-cdh5.7.1/conf/exec-avro-agent.conf --name exec-avro-agent -Dflume.root.logger=INFO,console

           

avro-logger-agent.conf 服務端

#agent
avro-logger-agent-1.sources = avro-source
avro-logger-agent-1.channels = memory-channel
avro-logger-agent-1.sinks = logger-sink

#source
avro-logger-agent-1.sources.avro-source.type = avro
avro-logger-agent-1.sources.avro-source.bind = localhost
avro-logger-agent-1.sources.avro-source.port = 55555

#channel
avro-logger-agent-1.channels.memory-channel.type = memory

#sink
avro-logger-agent-1.sinks.logger-sink.type = logger

#bind source and sink to channel
avro-logger-agent-1.sources.avro-source.channels = memory-channel
avro-logger-agent-1.sinks.logger-sink.channel = memory-channel
           

啟動

./flume-ng agent --conf /home/hadoop/app/flume-1.6.0-cdh5.7.1/conf --conf-file /home/hadoop/app/flume-1.6.0-cdh5.7.1/conf/avro-logger-agent.conf --name avro-logger-agent-1 -Dflume.root.logger=INFO,console
           

先啟動服務端，再啟動用戶端

往1.log寫東西，發現服務端有資料，用戶端沒資料

負載均衡

用戶端

exec-avro-agent-Load-balance.conf

#agent
exec-avro-agent-Load-balance.sources = exec-source
exec-avro-agent-Load-balance.channels = memory-channel
exec-avro-agent-Load-balance.sinks = avro-sink-1 avro-sink-2

#負載平衡
exec-avro-agent-Load-balance.sinkgroups = g1
exec-avro-agent-Load-balance.sinkgroups.g1.sinks = avro-sink-1 avro-sink-2
exec-avro-agent-Load-balance.sinkgroups.g1.processor.type =load_balance
exec-avro-agent-Load-balance.sinkgroups.g1.processor.backoff =true
exec-avro-agent-Load-balance.sinkgroups.g1.processor.selector=round_robin

#source
exec-avro-agent-Load-balance.sources.exec-source.type = exec
exec-avro-agent-Load-balance.sources.exec-source.command = tail -F /home/hadoop/app/flume-1.6.0-cdh5.7.1/logs/1.log

#channel
exec-avro-agent-Load-balance.channels.memory-channel.type = memory

#sink
exec-avro-agent-Load-balance.sinks.avro-sink-1.type = avro
exec-avro-agent-Load-balance.sinks.avro-sink-1.hostname = localhost
exec-avro-agent-Load-balance.sinks.avro-sink-1.port = 55555

exec-avro-agent-Load-balance.sinks.avro-sink-2.type = avro
exec-avro-agent-Load-balance.sinks.avro-sink-2.hostname = localhost
exec-avro-agent-Load-balance.sinks.avro-sink-2.port = 55556

#bind source and sink to channel
exec-avro-agent-Load-balance.sources.exec-source.channels = memory-channel
exec-avro-agent-Load-balance.sinks.avro-sink-1.channel = memory-channel
exec-avro-agent-Load-balance.sinks.avro-sink-2.channel = memory-channel
           

啟動

./flume-ng agent --conf /home/hadoop/app/flume-1.6.0-cdh5.7.1/conf --conf-file /home/hadoop/app/flume-1.6.0-cdh5.7.1/conf/exec-avro-agent-Load-balance.conf --name exec-avro-agent-Load-balance -Dflume.root.logger=INFO,console
           

兩個server端

avro-logger-agent-1.conf

#agent
avro-logger-agent-1.sources = avro-source
avro-logger-agent-1.channels = memory-channel
avro-logger-agent-1.sinks = logger-sink

#source
avro-logger-agent-1.sources.avro-source.type = avro
avro-logger-agent-1.sources.avro-source.bind = localhost
avro-logger-agent-1.sources.avro-source.port = 55555

#channel
avro-logger-agent-1.channels.memory-channel.type = memory

#sink
avro-logger-agent-1.sinks.logger-sink.type = logger

#bind source and sink to channel
avro-logger-agent-1.sources.avro-source.channels = memory-channel
avro-logger-agent-1.sinks.logger-sink.channel = memory-channel

           

啟動

./flume-ng agent --conf /home/hadoop/app/flume-1.6.0-cdh5.7.1/conf --conf-file /home/hadoop/app/flume-1.6.0-cdh5.7.1/conf/avro-logger-agent-1.conf --name avro-logger-agent-1 -Dflume.root.logger=INFO,console
           

avro-logger-agent-2.conf

#agent
avro-logger-agent-2.sources = avro-source
avro-logger-agent-2.channels = memory-channel
avro-logger-agent-2.sinks = logger-sink

#source
avro-logger-agent-2.sources.avro-source.type = avro
avro-logger-agent-2.sources.avro-source.bind = localhost
avro-logger-agent-2.sources.avro-source.port = 55556

#channel
avro-logger-agent-2.channels.memory-channel.type = memory

#sink
avro-logger-agent-2.sinks.logger-sink.type = logger

#bind source and sink to channel
avro-logger-agent-2.sources.avro-source.channels = memory-channel
avro-logger-agent-2.sinks.logger-sink.channel = memory-channel

           

啟動

./flume-ng agent --conf /home/hadoop/app/flume-1.6.0-cdh5.7.1/conf --conf-file /home/hadoop/app/flume-1.6.0-cdh5.7.1/conf/avro-logger-agent-2.conf --name avro-logger-agent-2 -Dflume.root.logger=INFO,console
           

先起兩個服務端，再起用戶端

然後可以看到兩個服務端都可以接受資料，總數為log檔案的内容，資料配置設定的政策，就是我們配置的round_robin算法

容錯

這裡的10，5是優先級，越大越優先，maxpenalty是冷卻時間上限

可參考http://www.aboutyun.com/thread-9170-1-1.html

監控

在server端啟動腳本最後面加上

-Dflume.monitoring.type=http -Dflume.monitoring.port=34545
           

啟動然後可以在網頁上34545端看到資料