Dead-letter queues

    As of Data Prepper 2.3, only the source supports DLQs.

    To configure a DLQ writer for the s3 source, add the following to your pipeline.yaml file:

    1. dlq-v${version}-${pipelineName}-${pluginId}-${timestampIso8601}-${uniqueId}

    The following information is replaced in the name pattern:

    • version: The Data Prepper version.
    • pipelineName: The pipeline name indicated in pipeline.yaml.
    • pluginId: The ID of the plugin associated with the DLQ event.

    DLQ supports the following configuration options.

    DLQ supports the following metrics.

    • dlqS3RecordsSuccess: Measures the number of successful records sent to S3.
    • dlqS3RecordsFailed: Measures the number of records that failed to be sent to S3.
    • : Measures the number of failed S3 requests.
    • dlqS3RequestSizeBytes: Measures the distribution of the S3 request’s payload size in bytes.
    • dlqS3RequestLatency: Measures latency when sending each S3 request, including retries.

    DLQ supports the following DLQ objects:

    • pluginId: The ID of the plugin that originated the event sent to the DLQ.
    • pluginName: The name of the plugin.
    • failedData : An object that contains the failed object and its options. This object is unique to each plugin.