Kafka Mesh filter

    Attention

    The Kafka mesh filter is only included in

    Attention

    Attention

    The kafka_mesh filter is does not work on Windows (the blocker is getting librdkafka compiled).

    Below example shows us typical filter configuration that proxies 3 Kafka clusters. Clients are going to connect to ‘127.0.0.1:19092’, and their messages are going to be distributed to cluster depending on topic names.

    It should be noted that Kafka broker filter can be inserted before Kafka mesh filter in the filter chain to capture the request processing metrics.

    Notes

    Given that this filter does its own processing of received requests, there are some changes in behaviour compared to explicit connection to a Kafka cluster:

    2. Python producers need to set API version of at least 1.0.0, so that the produce requests they send are going to have records with magic equal to 2.

    3. Downstream handling of Kafka producer ‘acks’ property is delegated to upstream client. E.g. if upstream client is configured to use acks=0 then the response is going to be sent to downstream client as soon as possible (even if they had non-zero acks!).

    4. Because of the splitting mentioned above, records are not necessarily appended one after another (as they do not get sent as single request to upstream). Users that want to avoid this scenario might want to take a look into downstream producer configs: ‘linger.ms’ and ‘batch.size’.

    5. Produce requests that reference to topics that do not match any of the rules are going to close connection and fail. This usually should not happen (clients request metadata first, and they should then fail with ‘no broker available’ first), but is possible if someone tailors binary payloads over the connection.