Apache Kafka Plugin

Apache Kafka™ is a distributed, partitioned, replicated commit log service. It provides the functionality of a messaging system, but with a unique design.

So, at a high level, producers send messages over the network to the Kafka cluster which in turn serves them up to consumers like this:

We are providing PhoenixConsumer to recieves the messages from Kafka Producer.

Prerequisites:

• Phoenix 4.10.0+

Installation & Setup:

Use our binary artifacts for Phoenix 4.10.0+ directly or download and build Phoenix yourself (see instructions here)

Phoenix Consumer for RegexEventSerializer Example:

Phoenix Consumer for JsonEventSerializer Example:

Create a kafka-consumer-json.properties file with below properties

Phoenix Consumer Execution Procedure:

Start the Kakfa Producer then send some messages

Learn more about Apache Kafka here

Start the PhoenixConsumer using below command

Configuration:

Note: This Plugin supports all Phoenix Flume Event Serializers.

RegexEventSerializer which primarily breaks the Kafka Message based on the regex specified in the configuration file.

JsonEventSerializer which primarily breaks the Kafka Message based on the schema specified in the configuration file.