Apache Kafka

    Currently, we provide a simpler way to integrate by combining two APIs, ListOffsets and Fetch, to quickly implement the ability to pull Kafka messages. Still, they do not support Apache Kafka’s consumer group feature for now and cannot be managed for offsets by Apache Kafka.

    • Offsets need to be managed manually

    They can be stored by a custom backend service or obtained via the list_offset command before starting to fetch the message, which can use timestamp to get the starting offset, or to get the initial and end offsets.

    • Unsupported batch data acquisition

    A single instruction can only obtain the data of a Topic Partition, does not support batch data acquisition through a single instruction

    The sequence field in the protocol is used to associate the request with the response, they will correspond one to one, the client can manage it in the way they want, APISIX will not modify it, only pass it back to the client through the response body.

    The following commands are currently used by Apache Kafka connect：

    • CmdKafkaFetch
    • CmdKafkaListOffset

    Possible response body: When an error occurs, ErrorResp will be returned, which includes the error string; the rest of the response will be returned after the execution of the particular command.

    • KafkaFetchResp
    • KafkaListOffsetResp

    Create route

    Create a route, set the upstream scheme field to kafka, and configure nodes to be the address of the Kafka broker.

    After configuring the route, you can use this feature.

    Enabling TLS and SASL/PLAIN authentication

    2.     -H 'X-API-KEY: <api-key>' \
    3.     -H 'Content-Type: application/json' \
    4.     -d '{
    5.     "uri": "/kafka",
    6.     "plugins": {
    7.         "kafka-proxy": {
    8.             "sasl": {
    9.                 "username": "user",
    10.                 "password": "pwd"
    11.             }
    13.     "upstream": {
    14.         "nodes": {
    15.             "kafka-server1:9092": 1,
    16.             "kafka-server2:9092": 1,
    17.             "kafka-server3:9092": 1
    18.         },
    19.         "type": "none",
    20.         "scheme": "kafka",
    21.         "tls": {
    22.             "verify": true
    23.         }