Set up a standalone Pulsar locally

This tutorial guides you through every step of installing Pulsar locally.

Currently, Pulsar is available for 64-bit macOS, Linux, and Windows. To use Pulsar, you need to install 64-bit JRE/JDK 8 or later versions

tip

By default, Pulsar allocates 2G JVM heap memory to start. It can be changed in file under PULSAR_MEM. This is extra options passed into JVM.

note

Broker is only supported on 64-bit JVM.

Install JDK on M1

In the current version, Pulsar uses a BookKeeper version which in turn uses RocksDB. RocksDB is compiled to work on x86 architecture and not ARM. Therefore, Pulsar can only work with x86 JDK. This is planned to be fixed in future versions of Pulsar.

One of the ways to easily install an x86 JDK is to use SDKMan as outlined in the following steps:

  1. Install .
  • Method 1: follow instructions on the SDKMan website.
  • Method 2: if you have Homebrew installed, enter the following command.
  1. Turn on Rosetta2 compatibility for SDKMan by editing ~/.sdkman/etc/config and changing the following property from false to true.
  2. sdkman_rosetta2_compatible=true
  1. Close the current shell / terminal window and open a new one.
  2. Make sure you don’t have any previously installed JVM of the same version by listing existing installed versions.
  2. sdk list java|grep installed

Example output:

  1.                | >>> | 17.0.3.6.1   | amzn    | installed  | 17.0.3.6.1-amzn

If you have any Java 17 version installed, uninstall it.

  2. sdk uinstall java 17.0.3.6.1
  1. Install any Java versions greater than Java 8.
  2.  sdk install java 17.0.3.6.1-amzn

Install Pulsar using binary release

To get started with Pulsar, download a binary tarball release in one of the following ways:

  2. $ tar xvfz apache-pulsar-2.10.0-bin.tar.gz
  3. $ cd apache-pulsar-2.10.0

What your package contains

The Pulsar binary package initially contains the following directories:

These directories are created once you begin running Pulsar.

DirectoryContains
dataThe data storage directory used by RocksDB and BookKeeper.
logsLogs created by the installation.
tip

If you want to use builtin connectors and tiered storage offloaders, you can install them according to the following instructions：

Since 2.1.0-incubating release, Pulsar releases a separate binary distribution, containing all the connectors. To enable those builtin connectors, you can download the connectors tarball release in one of the following ways:

  • download from the Apache mirror

  • download from the Pulsar downloads page

  • download from the Pulsar

  • use wget:

    2. $ wget https://archive.apache.org/dist/pulsar/pulsar-2.10.0/connectors/{connector}-2.10.0.nar

After you download the nar file, copy the file to the connectors directory in the pulsar directory. For example, if you download the pulsar-io-aerospike-2.10.0.nar connector file, enter the following commands:

  2. $ mkdir connectors
  3. $ mv pulsar-io-aerospike-2.10.0.nar connectors
  5. $ ls connectors
  6. pulsar-io-aerospike-2.10.0.nar
  7. ...
note
  • If you are running Pulsar in a bare metal cluster, make sure connectors tarball is unzipped in every pulsar directory of the broker (or in every pulsar directory of function-worker if you are running a separate worker cluster for Pulsar Functions).
  • If you are running Pulsar in Docker or deploying Pulsar using a docker image (e.g. or DC/OS, you can use the apachepulsar/pulsar-all image instead of the apachepulsar/pulsar image. apachepulsar/pulsar-all image has already bundled .

Install tiered storage offloaders (optional)

tip
  • Since 2.2.0 release, Pulsar releases a separate binary distribution, containing the tiered storage offloaders.

To get started with tiered storage offloaders, you need to download the offloaders tarball release on every broker node in one of the following ways:

  • download from the Apache mirror

  • download from the Pulsar downloads page

  • download from the Pulsar

  • use wget:

    2. $ wget https://archive.apache.org/dist/pulsar/pulsar-2.10.0/apache-pulsar-offloaders-2.10.0-bin.tar.gz
  2. $ tar xvfz apache-pulsar-offloaders-2.10.0-bin.tar.gz
  4. // you will find a directory named `apache-pulsar-offloaders-2.10.0` in the pulsar directory
  5. // then copy the offloaders
  7. $ mv apache-pulsar-offloaders-2.10.0/offloaders offloaders
  9. $ ls offloaders

For more information on how to configure tiered storage, see .

note
  • If you are running Pulsar in a bare metal cluster, make sure that offloaders tarball is unzipped in every broker’s pulsar directory.
  • If you are or deploying Pulsar using a docker image (e.g. K8S or DC/OS), you can use the apachepulsar/pulsar-all image instead of the apachepulsar/pulsar image. apachepulsar/pulsar-all image has already bundled tiered storage offloaders.

Once you have an up-to-date local copy of the release, you can start a local cluster using the command, which is stored in the bin directory, and specifying that you want to start Pulsar in standalone mode.

If you have started Pulsar successfully, you will see INFO-level log messages like this:

  2. 21:59:29.327 [DLM-/stream/storage-OrderedScheduler-3-0] INFO  org.apache.bookkeeper.stream.storage.impl.sc.StorageContainerImpl - Successfully started storage container (0).
  3. 21:59:34.576 [main] INFO  org.apache.pulsar.broker.authentication.AuthenticationService - Authentication is disabled
  4. 21:59:34.576 [main] INFO  org.apache.pulsar.websocket.WebSocketService - Pulsar WebSocket Service started
tip
  • The service is running on your terminal, which is under your direct control. If you need to run other commands, open a new terminal window.

You can also run the service as a background process using the bin/pulsar-daemon start standalone command. For more information, see .

  • By default, there is no encryption, authentication, or authorization configured. Apache Pulsar can be accessed from remote server without any authorization. Please do check Security Overview document to secure your deployment.

  • When you start a local standalone cluster, a public/default is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces. For more information, see Topics.

Pulsar provides a CLI tool called . The pulsar-client tool enables you to consume and produce messages to a Pulsar topic in a running cluster.

The following command consumes a message with the subscription name first-subscription to the my-topic topic:

  2. $ bin/pulsar-client consume my-topic -s "first-subscription"

If the message has been successfully consumed, you will see a confirmation like the following in the pulsar-client logs:

  2. 22:17:16.781 [main] INFO  org.apache.pulsar.client.cli.PulsarClientTool - 1 messages successfully consumed
tip

As you have noticed that we do not explicitly create the my-topic topic, to which we consume the message. When you consume a message to a topic that does not yet exist, Pulsar creates that topic for you automatically. Producing a message to a topic that does not exist will automatically create that topic for you as well.

Produce a message

The following command produces a message saying hello-pulsar to the my-topic topic:

  2. $ bin/pulsar-client produce my-topic --messages "hello-pulsar"

If the message has been successfully published to the topic, you will see a confirmation like the following in the pulsar-client logs:

  1. 22:21:08.693 [main] INFO  org.apache.pulsar.client.cli.PulsarClientTool - 1 messages successfully produced

Press Ctrl+C to stop a local standalone Pulsar.

tip

If the service runs as a background process using the bin/pulsar-daemon start standalone command, then use the command to stop the service. For more information, see .