Pulsar Perf

    This example shows how the Pulsar Perf produces messages with default options. For all configuration options available for the pulsar-perf produce command, see configuration options.

    After the command is executed, the test data is continuously output on the Console.

    输出

    1. 19:53:31.459 [pulsar-perf-producer-exec-1-1] INFO org.apache.pulsar.testclient.PerformanceProducer - Created 1 producers
    2. 19:53:31.482 [pulsar-timer-5-1] WARN com.scurrilous.circe.checksum.Crc32cIntChecksum - Failed to load Circe JNI library. Falling back to Java based CRC32c provider
    3. 19:53:40.861 [main] INFO org.apache.pulsar.testclient.PerformanceProducer - Throughput produced: 93.7 msg/s --- 0.7 Mbit/s --- failure 0.0 msg/s --- Latency: mean: 3.575 ms - med: 3.460 - 95pct: 4.790 - 99pct: 5.308 - 99.9pct: 5.834 - 99.99pct: 6.609 - Max: 6.609
    4. 19:53:50.909 [main] INFO org.apache.pulsar.testclient.PerformanceProducer - Throughput produced: 100.0 msg/s --- 0.8 Mbit/s --- failure 0.0 msg/s --- Latency: mean: 3.437 ms - med: 3.328 - 95pct: 4.656 - 99pct: 5.071 - 99.9pct: 5.519 - 99.99pct: 5.588 - Max: 5.588
    5. 19:54:10.940 [main] INFO org.apache.pulsar.testclient.PerformanceProducer - Throughput produced: 100.0 msg/s --- 0.8 Mbit/s --- failure 0.0 msg/s --- Latency: mean: 3.298 ms - med: 3.220 - 95pct: 4.474 - 99pct: 4.926 - 99.9pct: 5.645 - 99.99pct: 5.654 - Max: 5.654
    6. 19:54:20.956 [main] INFO org.apache.pulsar.testclient.PerformanceProducer - Throughput produced: 100.1 msg/s --- 0.8 Mbit/s --- failure 0.0 msg/s --- Latency: mean: 3.308 ms - med: 3.199 - 95pct: 4.532 - 99pct: 4.871 - 99.9pct: 5.291 - 99.99pct: 5.323 - Max: 5.323
    7. 19:54:30.972 [main] INFO org.apache.pulsar.testclient.PerformanceProducer - Throughput produced: 100.0 msg/s --- 0.8 Mbit/s --- failure 0.0 msg/s --- Latency: mean: 3.249 ms - med: 3.144 - 95pct: 4.437 - 99pct: 4.970 - 99.9pct: 5.329 - 99.99pct: 5.414 - Max: 5.414
    8. 19:54:40.987 [main] INFO org.apache.pulsar.testclient.PerformanceProducer - Throughput produced: 100.0 msg/s --- 0.8 Mbit/s --- failure 0.0 msg/s --- Latency: mean: 3.435 ms - med: 3.361 - 95pct: 4.772 - 99pct: 5.150 - 99.9pct: 5.373 - 99.99pct: 5.837 - Max: 5.837
    9. ^C19:54:44.325 [Thread-1] INFO org.apache.pulsar.testclient.PerformanceProducer - Aggregated throughput stats --- 7286 records sent --- 99.140 msg/s --- 0.775 Mbit/s

    From the above test data, you can get the throughput statistics and the write latency statistics. The aggregated statistics is printed when the Pulsar Perf is stopped. You can press Ctrl+C to stop the Pulsar Perf. After the Pulsar Perf is stopped, the formatted test result appears under your directory. The document looks like perf-producer-1589370810837.hgrm. You can also check the test result through HdrHistogram Plotter. For details about how to check the test result through , see HdrHistogram Plotter.

    You can get all options by executing the bin/pulsar-perf produce -h command. Therefore, you can modify these options as required.

    The following table lists configuration options available for the pulsar-perf produce command.

    This example shows how the Pulsar Perf consumes messages with default options.

    1. bin/pulsar-perf consume my-topic

    输出

    From the output test data, you can get the throughput statistics and the end-to-end latency statistics. The aggregated statistics is printed after the Pulsar Perf is stopped. You can press Ctrl+C to stop the Pulsar Perf.

    Configuration options for pulsar-perf consume

    You can get all options by executing the command. Therefore, you can modify these options as required.

    The following table lists configuration options available for the pulsar-perf consume command.

    选项说明默认值
    acks-delay-millis设置确认分组的延迟时间,以毫秒为单位100 ms
    auth-params设置认证参数,格式由认证插件类中configure方法的实现决定的,如”key1:val1,key2:val2” 或 “{“key1”:”val1”,”key2”:”val2”}”。N/A
    auth_plugin设置认证插件类名N/A
    auto_ack_chunk_q_full配置如果队列满时,是否自动确认接收器队列中最旧的消息。false
    listener-name为broker指定监听器名称。N/A
    batch-index-ack开启或禁用批量索引确认false
    busy-wait在 Pulsar 客户端启用或禁用 Busy-Wait.false
    conf-file设置配置文件N/A
    encryption-key-name设置用于加密有效载荷的公钥名称N/A
    encryption-key-value-file设置包含用于加密有效载荷的公钥文件N/A
    help配置帮助信息false
    expire_time_incomplete_chunked_messagesSet the expiration time for incomplete chunk messages (in milliseconds).0
    max-connections设置单个broker的最大TCP连接数100
    max_chunked_msgSet the max pending chunk messages.0
    num-consumers设定每个主题的消费者数量1
    num-io-threadsSet the number of threads to be used for handling connections to brokers.1
    num-subscriptionsSet the number of subscriptions (per topic).1
    num-topic设置主题数量1
    pool-messagesConfigure whether to use the pooled message.true
    rate模拟一个慢速消费者消费数据(速率 消息/秒)0.0
    receiver-queue-size设置接收队列大小1000
    receiver-queue-size-across-partitionsSet the max total size of the receiver queue across partitions.50000
    replicated配置是否复制订阅状态。false
    service-url设置 Pulsar 服务的 URL
    stats-interval-seconds设置统计间隔。 如果置为0,则不统计。0
    subscriber-name设置订阅者名称前缀sub
    subscription-positionSet the subscription position. Valid values are Latest, Earliest.Latest
    subscription-type设置订阅类型
  • Exclusive
  • Shared(共享)
  • Failover(灾备)
  • Exclusive
    test-durationSet the test duration (in seconds). If the value is 0 or smaller than 0, it keeps consuming messages.0
    tls-allow-insecureSet the allowed insecure TLS connection.N/A
    trust-cert-file设置受信的TLS证书文件所在路径

    By default, the Pulsar Perf uses conf/client.conf as the default configuration and uses conf/log4j2.yaml as the default Log4j configuration. If you want to connect to other Pulsar clusters, you can update the brokerServiceUrl in the client configuration.

    You can use the following commands to change the configuration file and the Log4j configuration file.

    1. export PULSAR_CLIENT_CONF=<你的配置文件>
    2. export PULSAR_LOG_CONF=<你的日志配置文件>

    In addition, you can use the following command to configure the JVM configuration through environment variables:

    1. export PULSAR_EXTRA_OPTS='-Xms4g -Xmx4g -XX:MaxDirectMemorySize=4g'

    To check test results through the HdrHistogram Plotter, follow these steps:

    1. 从 GitHub 上克隆 HdrHistogram 仓库到本地

    2. 切换到HdrHistogram文件夹

      1. cd HdrHistogram
    3. 安装HdrHistogram 绘图器

      1. mvn clean install -DskipTests
    4. 转换 Pulsar Perf 生成的文件

    5. 你将获得两个文件。 将扩展名为 .hgrm 的输出文件上传到HdrHistogram Plotter