Pulsar Encryption

    Pulsar uses dynamically generated symmetric AES key to encrypt messages(data). The AES key(data key) is encrypted using application provided ECDSA/RSA key pair, as a result there is no need to share the secret with everyone.

    Key is a public/private key pair used for encryption/decryption. The producer key is the public key, and the consumer key is the private key of the key pair.

    The application configures the producer with the public key. This key is used to encrypt the AES data key. The encrypted data key is sent as part of message header. Only entities with the private key(in this case the consumer) will be able to decrypt the data key which is used to decrypt the message.

    Pulsar does not store the encryption key anywhere in the pulsar service. If you lose/delete the private key, your message is irretrievably lost, and is unrecoverable

    Producer

    alt text

    Here are the steps to get started:

    1. Create your ECDSA or RSA public/private key pair.
    1. Add the public and private key to the key management and configure your producers to retrieve public keys and consumers clients to retrieve private keys.
    2. Implement CryptoKeyReader::getPublicKey() interface from producer and CryptoKeyReader::getPrivateKey() interface from consumer, which will be invoked by Pulsar client to load the key.
    3. Add encryption key to producer configuration: conf.addEncryptionKey(“myapp.key”)
    4. Add CryptoKeyReader implementation to producer/consumer config: conf.setCryptoKeyReader(keyReader)
    5. Sample producer application:
    1. Sample Consumer Application:

    Enabling encryption at the producer application:

    If you produce messages that are consumed across application boundaries, you need to ensure that consumers in other applications have access to one of the private keys that can decrypt the messages. This can be done in two ways:

    1. You grant access to one of the private keys from the pairs used by producer

    In some cases, the producer may want to encrypt the messages with multiple keys. For this, add all such keys to the config. Consumer will be able to decrypt the message, as long as it has access to at least one of the keys.

    E.g: If messages needs to be encrypted using 2 keys myapp.messagekey1 and myapp.messagekey2,

    Handling Failures:

    • Producer/ Consumer loses access to the key
      • Producer action will fail indicating the cause of the failure. Application has the option to proceed with sending unencrypted message in such cases. Call conf.setCryptoFailureAction(ProducerCryptoFailureAction) to control the producer behavior. The default behavior is to fail the request.
      • If consumption failed due to decryption failure or missing keys in consumer, application has the option to consume the encrypted message or discard it. Call conf.setCryptoFailureAction(ConsumerCryptoFailureAction) to control the consumer behavior. The default behavior is to fail the request. Application will never be able to decrypt the messages if the private key is permanently lost.
    • Batch messaging
      • If decryption fails and the message contain batch messages, client will not be able to retrieve individual messages in the batch, hence message consumption fails even if conf.setCryptoFailureAction() is set to CONSUME.
    • If decryption fails, the message consumption stops and application will notice backlog growth in addition to decryption failure messages in the client log. If application does not have access to the private key to decrypt the message, the only option is to skip/discard backlogged messages.