Getting started with cross-cluster replication

    Cross-cluster replication has the following prerequisites:

    • Both the leader and follower cluster must have the replication plugin installed.

    • If you’ve overridden in opensearch.yml on the follower cluster, make sure it also includes the remote_cluster_client role:

    Permissions

    Make sure the Security plugin is either enabled on both clusters or disabled on both clusters. If you disabled the Security plugin, you can skip this section. However, we strongly recommend enabling the Security plugin in production scenarios.

    If the Security plugin is enabled, make sure that non-admin users are mapped to the appropriate permissions so they can perform replication actions. For index and cluster-level permissions requirements, see Cross-cluster replication permissions.

    In addition, verify and add the distinguished names (DNs) of each follower cluster node on the leader cluster to allow connections from the followers to the leader.

    First, get the node’s DN from each follower cluster:

    1. curl -XGET -k -u 'admin:admin' 'https://localhost:9200/_opendistro/_security/api/ssl/certs?pretty'
    3. {
    4.    "transport_certificates_list": [
    5.       {
    6.          "issuer_dn" : "CN=Test,OU=Server CA 1B,O=Test,C=US",
    7.          "subject_dn" : "CN=follower.test.com", # To be added under leader's nodes_dn configuration
    8.          "not_before" : "2021-11-12T00:00:00Z",
    9.          "not_after" : "2022-12-11T23:59:59Z"
    10.       }
    11.    ]
    12. }

    Then verify that it’s part of the leader cluster configuration in opensearch.yml. Otherwise, add it under the following setting:

    1. plugins.security.nodes_dn:
    2.   - "CN=*.leader.com, OU=SSL, O=Test, L=Test, C=DE" # Already part of the configuration
    3.   - "CN=follower.test.com" # From the above response from follower

    To start two single-node clusters on the same network, save this sample file as docker-compose.yml and run docker-compose up:

    1. version: '3'
    2. services:
    3.   replication-node1:
    4.     image: opensearchproject/opensearch:2.7.0
    5.     container_name: replication-node1
    6.     environment:
    7.       - cluster.name=leader-cluster
    8.       - discovery.type=single-node
    9.       - bootstrap.memory_lock=true
    10.       - "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m"
    11.     ulimits:
    12.       memlock:
    13.         soft: -1
    14.         hard: -1
    15.     volumes:
    16.       - opensearch-data2:/usr/share/opensearch/data
    17.     ports:
    18.       - 9201:9200
    19.       - 9700:9600 # required for Performance Analyzer
    20.     networks:
    21.       - opensearch-net
    22.     image: opensearchproject/opensearch:2.7.0
    23.     container_name: replication-node2
    24.     environment:
    25.       - cluster.name=follower-cluster
    26.       - discovery.type=single-node
    27.       - bootstrap.memory_lock=true
    29.     ulimits:
    30.       memlock:
    31.         soft: -1
    32.         hard: -1
    33.     volumes:
    34.       - opensearch-data1:/usr/share/opensearch/data
    35.     ports:
    36.       - 9200:9200
    37.       - 9600:9600 # required for Performance Analyzer
    38.     networks:
    39.       - opensearch-net
    41. volumes:
    42.   opensearch-data1:
    43.   opensearch-data2:
    45. networks:
    46.   opensearch-net:

    After the clusters start, verify the names of each:

    1. curl -XGET -u 'admin:admin' -k 'https://localhost:9201'
    2. {
    3.   "cluster_name" : "leader-cluster",
    4.   ...
    5. }
    7. curl -XGET -u 'admin:admin' -k 'https://localhost:9200'
    8. {
    9.   "cluster_name" : "follower-cluster",
    10.   ...
    11. }

    For this example, use port 9201 (replication-node1) as the leader and port 9200 (replication-node2) as the follower cluster.

    1. docker ps
    2. CONTAINER ID    IMAGE                                       PORTS                                                      NAMES
    3. 3b8cdc698be5    opensearchproject/opensearch:2.7.0   0.0.0.0:9200->9200/tcp, 0.0.0.0:9600->9600/tcp, 9300/tcp   replication-node2
    4. 731f5e8b0f4b    opensearchproject/opensearch:2.7.0   9300/tcp, 0.0.0.0:9201->9200/tcp, 0.0.0.0:9700->9600/tcp   replication-node1

    Then get that container’s IP address:

    Set up a cross-cluster connection

    Cross-cluster replication follows a “pull” model, so most changes occur on the follower cluster, not the leader cluster.

    On the follower cluster, add the IP address (with port 9300) for each seed node. Because this is a single-node cluster, you only have one seed node. Provide a descriptive name for the connection, which you’ll use in the request to start replication:

    1. curl -XPUT -k -H 'Content-Type: application/json' -u 'admin:admin' 'https://localhost:9200/_cluster/settings?pretty' -d '
    2. {
    3.   "persistent": {
    4.     "cluster": {
    5.       "remote": {
    6.         "my-connection-alias": {
    7.         }
    8.       }
    9.     }
    10.   }
    11. }'

    To get started, create an index called leader-01 on the leader cluster:

    Then start replication from the follower cluster. In the request body, provide the connection name and leader index that you want to replicate, along with the security roles you want to use:

    1. curl -XPUT -k -H 'Content-Type: application/json' -u 'admin:admin' 'https://localhost:9200/_plugins/_replication/follower-01/_start?pretty' -d '
    2. {
    3.    "leader_alias": "my-connection-alias",
    4.    "leader_index": "leader-01",
    5.    "use_roles":{
    6.       "leader_cluster_role": "all_access",
    7.       "follower_cluster_role": "all_access"
    8.    }
    9. }'

    If the Security plugin is disabled, omit the use_roles parameter. If it’s enabled, however, you must specify the leader and follower cluster roles that OpenSearch will use to authenticate the request. This example uses all_access for simplicity, but we recommend creating a replication user on each cluster and mapping it accordingly.

    This command creates an identical read-only index named follower-01 on the follower cluster that continuously stays updated with changes to the leader-01 index on the leader cluster. Starting replication creates a follower index from scratch – you can’t convert an existing index to a follower index.

    Confirm replication

    After replication starts, get the status:

    1. curl -XGET -k -u 'admin:admin' 'https://localhost:9200/_plugins/_replication/follower-01/_status?pretty'
    3. {
    4.   "status" : "SYNCING",
    5.   "reason" : "User initiated",
    6.   "leader_alias" : "my-connection-alias",
    7.   "leader_index" : "leader-01",
    8.   "follower_index" : "follower-01",
    9.   "syncing_details" : {
    10.     "leader_checkpoint" : -1,
    11.     "follower_checkpoint" : -1,
    12.     "seq_no" : 0
    13.   }
    14. }

    Possible statuses are SYNCING, BOOTSTRAPPING, PAUSED, and REPLICATION NOT IN PROGRESS.

    The leader and follower checkpoint values begin as negative numbers and reflect the shard count (-1 for one shard, -5 for five shards, and so on). The values increment with each change and illustrate how many updates the follower is behind the leader. If the indexes are fully synced, the values are the same.

    1. curl -XPUT -k -H 'Content-Type: application/json' -u 'admin:admin' 'https://localhost:9201/leader-01/_doc/1?pretty' -d '{"The Shining": "Stephen King"}'

    Then validate the replicated content on the follower index:

    You can temporarily pause replication of an index if you need to remediate issues or reduce load on the leader cluster:

    1. curl -XPOST -k -H 'Content-Type: application/json' -u 'admin:admin' 'https://localhost:9200/_plugins/_replication/follower-01/_pause?pretty' -d '{}'

    To confirm that replication is paused, get the status:

    1. curl -XGET -k -u 'admin:admin' 'https://localhost:9200/_plugins/_replication/follower-01/_status?pretty'
    3. {
    4.   "status" : "PAUSED",
    5.   "reason" : "User initiated",
    6.   "leader_alias" : "my-connection-alias",
    7.   "leader_index" : "leader-01",
    8.   "follower_index" : "follower-01"
    9. }

    When you’re done making changes, resume replication:

    1. curl -XPOST -k -H 'Content-Type: application/json' -u 'admin:admin' 'https://localhost:9200/_plugins/_replication/follower-01/_resume?pretty' -d '{}'

    When replication resumes, the follower index picks up any changes that were made to the leader index while replication was paused.

    Note that you can’t resume replication after it’s been paused for more than 12 hours. You must , delete the follower index, and restart replication of the leader.

    Stop replication

    When you no longer need to replicate an index, terminate replication from the follower cluster:

    1. curl -XPOST -k -H 'Content-Type: application/json' -u 'admin:admin' 'https://localhost:9200/_plugins/_replication/follower-01/_stop?pretty' -d '{}'

    When you stop replication, the follower index un-follows the leader and becomes a standard index that you can write to. You can’t restart replication after stopping it.

    Get the status to confirm that the index is no longer being replicated:

    1. curl -XGET -k -u 'admin:admin' 'https://localhost:9200/_plugins/_replication/follower-01/_status?pretty'
    3. {
    4.   "status" : "REPLICATION NOT IN PROGRESS"

    You can further confirm that replication is stopped by making modifications to the leader index and confirming they don’t show up on the follower index.