天天看点

Flume:3.输出模式处理器【sink-processor】

Flume配置的三个“器”:

  • 1.消息源拦截器【source-Interceptor】
  • 2.通道选择器【source-selector】
  • 3.输出模式处理器【sink-processor】(本文)
#agent1 name
agent1.channels = c1
agent1.sources = r1
agent1.sinks = k1 k2


## ---------------------------
## 开启sinks的processor功能,必须要有sinkgroup
## ---------------------------
#set gruop
agent1.sinkgroups = g1
#set sink group
agent1.sinkgroups.g1.sinks = k1 k2


#set channel
agent1.channels.c1.type = memory
agent1.channels.c1.capacity = 1000
agent1.channels.c1.transactionCapacity = 100
 

agent1.sources.r1.channels = c1
agent1.sources.r1.type = exec
agent1.sources.r1.command = tail -F /root/log/test.log


# set sink1
agent1.sinks.k1.channel = c1
agent1.sinks.k1.type = avro
agent1.sinks.k1.hostname = itcast02
agent1.sinks.k1.port = 52020


# set sink2
agent1.sinks.k2.channel = c1
agent1.sinks.k2.type = avro
agent1.sinks.k2.hostname = itcast03
agent1.sinks.k2.port = 52020

 
## ---------------------------
## 处理器 - 故障转移:按优先级传输
## 优先级 -- priority,从高到低排序
## 最大尝试时间 -- maxpenalty
## ---------------------------
agent1.sinkgroups.g1.processor.type = failover
agent1.sinkgroups.g1.processor.priority.k1 = 10
agent1.sinkgroups.g1.processor.priority.k2 = 5
agent1.sinkgroups.g1.processor.maxpenalty = 10000

## ---------------------------
## 处理器 - 负载均衡:组内的sink都收到
## 模式选择:selector -- 默认是round_robin(轮训),还可以选择random(随机)
## 如果backoff被开启,则 sink processor会屏蔽故障的sink
## ---------------------------
agent1.sinkgroups.g1.processor.type = load_balance
agent1.sinkgroups.g1.processor.selector = round_robin
agent1.sinkgroups.g1.processor.backoff = true