OpenSearch sink

    The plugin supports OpenSearch 1.0 and later and Elasticsearch 7.3 and later.

    To configure an opensearch sink, specify the opensearch option within the pipeline configuration:

    To configure an sink, specify the domain endpoint as the hosts option:

    1. pipeline:
    2.   ...
    3.   sink:
    4.     opensearch:
    5.       hosts: ["https://your-amazon-opensearch-service-endpoint"]
    6.       aws_sigv4: true
    7.       cert: path/to/cert
    8.       insecure: false
    9.       index_type: trace-analytics-service-map
    10.       bulk_size: 4

    The following table describes options you can configure for the opensearch sink.

    You can include the option in your pipeline configuration to control the number of times the source tries to write to sinks with exponential backoff. If you don’t include this option, pipelines keep retrying forever.

    If you specify max_retries and a pipeline has a dead-letter queue (DLQ) configured, the pipeline will keep trying to write to sinks until it reaches the maximum number of retries, at which point it starts to send failed data to the DLQ.

    If you don’t specify max_retries, only data that is rejected by sinks is written to the DLQ. Pipelines continue to try to write all other data to the sinks.

    In order to send data to an OpenSearch cluster using the opensearch sink plugin, you must specify your username and password within the pipeline configuration. The following example pipelines.yaml file demonstrates how to specify admin security credentials:

    1. sink:
    2.   - opensearch:
    3.       username: "admin"
    4.       ...

    Alternately, rather than admin credentials, you can specify the credentials of a user mapped to a role with the minimum permissions listed in the following sections.

    Cluster permissions

    • cluster_all
    • indices:admin/template/get
    • indices:admin/template/put
    • Index: otel-v1*; Index permission: indices_all
    • Index: .opendistro-ism-config; Index permission: indices_all
    • Index: *; Index permission: manage_aliases

    The opensearch sink plugin can send data to an domain, which uses IAM for security. The plugin uses the default credential chain. Run aws configure using the AWS Command Line Interface (AWS CLI) to set your credentials.

    Make sure the credentials that you configure have the required IAM permissions. The following domain access policy demonstrates the minimum required permissions:

    For instructions on how to configure the domain access policy, see in the Amazon OpenSearch Service documentation.

    Fine-grained access control

    If your OpenSearch Service domain uses , the opensearch sink plugin requires some additional configuration.

    IAM ARN as master user

    If you’re using an IAM Amazon Resource Name (ARN) as the master user, include the aws_sigv4 option in your sink configuration:

    1. ...
    2. sink:
    3.     opensearch:
    4.       hosts: ["https://your-fgac-amazon-opensearch-service-endpoint"]
    5.       aws_sigv4: true

    Run aws configure using the AWS CLI to use the master IAM user credentials. If you don’t want to use the master user, you can specify a different IAM role using the aws_sts_role_arn option. The plugin will then use this role to sign requests sent to the domain sink. The ARN that you specify must be included in the .

    Master user in the internal user database

    If your domain uses a master user in the internal user database, specify the master username and password as well as the aws_sigv4 option:

    1. sink:
    2.     opensearch:
    3.       hosts: ["https://your-fgac-amazon-opensearch-service-endpoint"]
    4.       aws_sigv4: false
    5.       username: "master-username"
    6.       password: "master-password"

    For more information, see in the Amazon OpenSearch Service documentation.

    Note: You can create a new IAM role or internal user database user with the all_access permission and use it instead of the master user.

    OpenSearch Serverless collection sinks have the following limitations:

    • You can’t write to a collection that uses virtual private cloud (VPC) access. The collection must be accessible from public networks.
    • The OTel trace group processor doesn’t currently support collection sinks.

    First, create an IAM role that the pipeline will assume in order to write to the collection. The role must have the following minimum permissions:

    The role must have the following trust relationship, which allows the pipeline to assume it:

    2.     "Version": "2012-10-17",
    3.     "Statement": [
    4.         {
    5.             "Effect": "Allow",
    6.             "Principal": {
    7.                 "AWS": "arn:aws:iam::<AccountId>:root"
    8.             "Action": "sts:AssumeRole"
    9.         }
    10.     ]
    11. }

    Creating a collection

    Next, create a collection with the following settings:

    • Public to both the OpenSearch endpoint and OpenSearch Dashboards.

    • The following data access policy, which grants the required permissions to the pipeline role:

      1. [
      2.  {
      3.     "Rules":[
      4.        {
      5.           "Resource":[
      6.              "index/collection-name/*"
      7.           ],
      8.           "Permission":[
      9.              "aoss:CreateIndex",
      10.              "aoss:UpdateIndex",
      11.              "aoss:DescribeIndex",
      12.              "aoss:WriteDocument"
      13.           ],
      14.           "ResourceType":"index"
      15.        }
      16.     ],
      17.     "Principal":[
      18.        "arn:aws:iam::<AccountId>:role/PipelineRole"
      19.     ],
      20.     "Description":"Pipeline role access"
      21.  }

      Important: Make sure to replace the ARN in the Principal element with the ARN of the pipeline role that you created in the preceding step.

      For instructions on how to create collections, see in the Amazon OpenSearch Service documentation.

    Within your pipelines.yaml file, specify the OpenSearch Serverless collection endpoint as the hosts option. In addition, you must set the serverless option to true. Specify the pipeline role in the option: