To use message deduplication in Pulsar, you need to configure your Pulsar brokers and clients.
You can enable or disable message deduplication on a per-namespace basis. By default, it is disabled on all namespaces. You can enable it in the following ways:
- Enable for specific namespaces with the interface
You can configure message deduplication in Pulsar using the configuration file. The following deduplication-related parameters are available.
By default, message deduplication is disabled on all Pulsar namespaces. To enable it by default on all namespaces, set the brokerDeduplicationEnabled
parameter to true
and re-start the broker.
Though message deduplication is disabled by default at broker-level, you can enable message deduplication for specific namespaces using the pulsar-admin namespace set-deduplication
command. You can use the --enable
/-e
flag and specify the namespace. The following is an example with <tenant>/<namespace>
:
Even if you enable message deduplication at broker-level, you can disable message deduplication for a specific namespace using the command. Use the --disable
/-d
flag and specify the namespace. The following is an example with :
If you enable message deduplication in Pulsar brokers, you need complete the following tasks for your client producers:
- Specify a name for the producer.
The instructions for Java, Python, and C++ clients are different.
Python clients
C++ clients
To enable message deduplication on a Java producer, set the producer name using the producerName
setter, and set the timeout to 0
using the sendTimeout
setter.
To enable message deduplication on a , set the producer name using producer_name
, and set the timeout to 0
using send_timeout_millis
.