Add Kafka as a Receiver

    • Deploy strimzi-kafka-operator and then create a Kafka cluster and a Kafka topic by creating and KafkaTopic CRDs.
    • Add Kafka as a log receiver to receive logs sent from Fluent Bit.
    • Verify whether the Kafka cluster is receiving logs using .
    • You need a user granted a role including the permission of Cluster Management. For example, you can log in to the console as admin directly or create a new role with the permission and assign it to a user.
    • Before adding a log receiver, you need to enable any of the logging, events or auditing components. For more information, see Enable Pluggable Components. logging is enabled as an example in this tutorial.

    You can use to create a Kafka cluster and a Kafka topic. If you already have a Kafka cluster, you can start from the next step.

    1. Install strimzi-kafka-operator in the default namespace:

      1. helm install --name kafka-operator -n default strimzi/strimzi-kafka-operator
      1. $ kubectl -n default get pod 
      2. NAME                                         READY   STATUS    RESTARTS   AGE
      3. my-cluster-entity-operator-f977bf457-s7ns2   3/3     Running   0          69m
      4. my-cluster-kafka-0                           2/2     Running   0          69m
      5. my-cluster-kafka-1                           2/2     Running   0          69m
      6. my-cluster-kafka-2                           2/2     Running   0          69m
      8. my-cluster-zookeeper-1                       1/1     Running   1          71m
      9. my-cluster-zookeeper-2                       1/1     Running   1          71m

      Run the following command to check the metadata of the Kafka cluster:

    1. Log in to KubeSphere as admin. Click Platform in the upper-left corner and select Cluster Management.

      Note

    2. On the Cluster Management page, go to Log Receivers in Cluster Settings.

    3. Click Add Log Receiver and select Kafka. Enter the Kafka service address and port number, and then click OK to continue.

    4. Run the following commands to verify whether the Kafka cluster is receiving logs sent from Fluent Bit:

      1. # Start a util container
      2. kubectl run --rm utils -it --generator=run-pod/v1 --image arunvelsriram/utils bash
      3. # Install Kafkacat in the util container
      4. apt-get install kafkacat
      6. kafkacat -C -b my-cluster-kafka-0.my-cluster-kafka-brokers.default.svc:9092,my-cluster-kafka-1.my-cluster-kafka-brokers.default.svc:9092,my-cluster-kafka-2.my-cluster-kafka-brokers.default.svc:9092 -t my-topic