在1.6版本中,Thrift的source和sink支持kerberos认证。客户端需要使用secureRpcClientFactory的getThriftInstance方法获得SecureThriftRpcClient对象。SecureThriftClient继承ThriftRpcClient(实现了RpcClient接口)。使用SecureRpcClientFactory依赖于Flume-ng-auth模块。客户端的principal以及keytab都需要通过参数的形式传入,他们作为kerberos KDC的证书。另外,目标服务器的principal也需要提供。下面就是secureRpclientFacotry的例子:
ThriftSource则需要配置成kerberos模式。
a1.sources = r1
a1.sinks = k1
a1.channels.c1.type = memory
a1.sources.r1.channels = c1
a1.sources.r1.type = thrift
a1.sources.r1.bind = 0.0.0.0
a1.sources.r1.port = 41414
a1.sources.r1.kerberos = true
a1.sources.r1.agent-principal = flume/server.example.org@EXAMPLE.ORG
a1.sources.r1.agent-keytab = /tmp/flume.keytab
a1.sinks.k1.channel = c1
a1.sinks.k1.type = logger
为了更灵活一些,failover flume client实现FailoverRpcClient,可以基于下面的配置:
client.type = default_failover
hosts = h1 h2 h3 # at least one is required, but 2 or
# more makes better sense
hosts.h2 = host2.example.org:41414
hosts.h3 = host3.example.org:41414
# specified, 3 in this case). A '0'
# value doesn't make much sense because
# it will just cause an append call to
# immmediately fail. A '1' value means
# that the failover client will try only
# once to send the Event, and if it
# fails then there will be no failover
# to a second client, so this value
# causes the failover client to
# degenerate into just a default client.
# It makes sense to set this value to at
# least the number of hosts that you
# specified.
batch-size = 100 # Must be >=1 (default: 100)
connect-timeout = 20000 # Must be >=1000 (default: 20000)
request-timeout = 20000 # Must be >=1000 (default: 20000)
Flume客户端SDK也支持在多个主机中负载均衡。client使用<host>:<port>
的形式组成一个负载均衡组。client端会配置负载均衡的策略,可能是随机选择配置的主机,也可能是基于轮询的模式。你可以通过实现LoadBalancingRpcClient$HostSelector接口,指定自定义的类。在这种情况下,FQCN需要指定成特定的host selector.负载均衡RPC目前不支持Thrift.
最大的backoff事件可以通过maxBackoff进行配置。默认是30S(在OrderSelector中指定)。backoff 参数会以指数级增长。最大限制为65536秒,即18.2小时。
也可以直接如下配置:
client.type = default_loadbalance
hosts.h2 = host2.example.org:41414
hosts.h3 = host3.example.org:41414
backoff = false # Specifies whether the client should
# back-off from (i.e. temporarily
# blacklist) a failed host
# (default: false).
maxBackoff = 0 # Max timeout in millis that a will
# remain inactive due to a previous
# failure with that host (default: 0,
# which effectively becomes 30000)
host-selector = round_robin # The host selection strategy used
# when load-balancing among hosts
# (default: round_robin).
# Other values are include "random"
# or the FQCN of a custom class
# that implements
# LoadBalancingRpcClient$HostSelector
batch-size = 100 # Must be >=1 (default: 100)
connect-timeout = 20000 # Must be >=1000 (default: 20000)
配置嵌入式节点与配置普通节点类似。下面是额外的配置: