Cross-cluster search

    When accessing a remote cluster from a coordinating cluster using cross-cluster search:

    1. The security plugin authenticates the user on the coordinating cluster.
    2. The security plugin fetches the user’s backend roles on the coordinating cluster.
    3. The call, including the authenticated user, is forwarded to the remote cluster.
    4. The user’s permissions are evaluated on the remote cluster.

    You can have different authentication and authorization configurations on the remote and coordinating cluster, but we recommend using the same settings on both.

    To query indices on remote clusters, users need to have the following permissions for the index, in addition to or SEARCH permissions:

    Sample roles.yml configuration

    1. humanresources:
    2.   cluster:
    3.     - CLUSTER_COMPOSITE_OPS_RO
    4.   indices:
    5.     'humanresources':
    6.       '*':
    7.         - READ
    8.         - indices:admin/shards/search_shards # needed for CCS

    Sample role in OpenSearch Dashboards

    Save this file as docker-compose.yml and run docker-compose up to start two single-node clusters on the same network:

    1. version: '3'
    2. services:
    3.   opensearch-ccs-node1:
    4.     image: opensearchproject/opensearch:2.1.0
    5.     container_name: opensearch-ccs-node1
    6.     environment:
    7.       - cluster.name=opensearch-ccs-cluster1
    8.       - discovery.type=single-node
    9.       - bootstrap.memory_lock=true # along with the memlock settings below, disables swapping
    10.       - "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m" # minimum and maximum Java heap size, recommend setting both to 50% of system RAM
    11.     ulimits:
    12.       memlock:
    13.         soft: -1
    14.         hard: -1
    15.     volumes:
    16.       - opensearch-data1:/usr/share/opensearch/data
    17.     ports:
    18.       - 9200:9200
    19.     networks:
    20.       - opensearch-net
    22.   opensearch-ccs-node2:
    23.     image: opensearchproject/opensearch:2.1.0
    25.     environment:
    26.       - cluster.name=opensearch-ccs-cluster2
    27.       - discovery.type=single-node
    28.       - bootstrap.memory_lock=true # along with the memlock settings below, disables swapping
    29.       - "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m" # minimum and maximum Java heap size, recommend setting both to 50% of system RAM
    30.     ulimits:
    31.       memlock:
    32.         soft: -1
    33.         hard: -1
    34.     volumes:
    35.       - opensearch-data2:/usr/share/opensearch/data
    36.     ports:
    37.       - 9250:9200
    38.       - 9700:9600 # required for Performance Analyzer
    39.     networks:
    40.       - opensearch-net
    42. volumes:
    43.   opensearch-data1:
    44.   opensearch-data2:
    46. networks:
    47.   opensearch-net:
    1. curl -XGET -u 'admin:admin' -k 'https://localhost:9200'
    2. {
    3.   "cluster_name" : "opensearch-ccs-cluster1",
    4.   ...
    5. }
    7. {
    8.   "cluster_name" : "opensearch-ccs-cluster2",
    9.   ...
    10. }

    Both clusters run on localhost, so the important identifier is the port number. In this case, use port 9200 () as the remote cluster, and port 9250 (opensearch-ccs-node2) as the coordinating cluster.

    To get the IP address for the remote cluster, first identify its container ID:

    Then get that container’s IP address:

    1. docker inspect --format='{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' 6fe89ebc5a8e
    2. 172.31.0.3

    On the coordinating cluster, add the remote cluster name and the IP address (with port 9300) for each “seed node.” In this case, you only have one seed node:

    1. curl -k -XPUT -H 'Content-Type: application/json' -u 'admin:admin' 'https://localhost:9250/_cluster/settings' -d '
    2. {
    3.   "persistent": {
    4.     "cluster.remote": {
    5.       "opensearch-ccs-cluster1": {
    6.         "seeds": ["172.31.0.3:9300"]
    7.       }
    8.     }
    9.   }
    10. }'

    On the remote cluster, index a document:

    1. curl -XPUT -k -H 'Content-Type: application/json' -u 'admin:admin' 'https://localhost:9200/books/_doc/1' -d '{"Dracula": "Bram Stoker"}'

    To continue testing, create a new user on both clusters:

    1. curl -XPUT -k -u 'admin:admin' 'https://localhost:9200/_plugins/_security/api/internalusers/booksuser' -H 'Content-Type: application/json' -d '{"password":"password"}'
    2. curl -XPUT -k -u 'admin:admin' 'https://localhost:9250/_plugins/_security/api/internalusers/booksuser' -H 'Content-Type: application/json' -d '{"password":"password"}'

    Then run the same search as before with booksuser:

    1. curl -XGET -k -u booksuser:password 'https://localhost:9250/opensearch-ccs-cluster1:books/_search?pretty'
    2. {
    3.   "error" : {
    4.     "root_cause" : [
    5.       {
    6.         "type" : "security_exception",
    7.         "reason" : "no permissions for [indices:admin/shards/search_shards, indices:data/read/search] and User [name=booksuser, roles=[], requestedTenant=null]"
    8.       }
    9.     ],
    10.     "type" : "security_exception",
    11.     "reason" : "no permissions for [indices:admin/shards/search_shards, indices:data/read/search] and User [name=booksuser, roles=[], requestedTenant=null]"
    12.   },
    13.   "status" : 403
    14. }

    Note the permissions error. On the remote cluster, create a role with the appropriate permissions, and map booksuser to that role:

    1. curl -XPUT -k -u 'admin:admin' -H 'Content-Type: application/json' 'https://localhost:9200/_plugins/_security/api/roles/booksrole' -d '{"index_permissions":[{"index_patterns":["books"],"allowed_actions":["indices:admin/shards/search_shards","indices:data/read/search"]}]}'
    2. curl -XPUT -k -u 'admin:admin' -H 'Content-Type: application/json' 'https://localhost:9200/_plugins/_security/api/rolesmapping/booksrole' -d '{"users" : ["booksuser"]}'

    Both clusters must have the user, but only the remote cluster needs the role and mapping; in this case, the coordinating cluster handles authentication (i.e. “Does this request include valid user credentials?”), and the remote cluster handles authorization (i.e. “Can this user access this data?”).

    Finally, repeat the search: