Telemetry

    For mesh level configuration, put the resource in root configuration namespace for your Istio installation without a workload selector.

    For any namespace, including the root configuration namespace, it is only valid to have a single workload selector-less Telemetry resource.

    For resources with a workload selector, it is only valid to have one resource selecting any given workload.

    The hierarchy of Telemetry configuration is as follows:

    1. Workload-specific configuration
    2. Namespace-specific configuration
    3. Root namespace configuration

    Examples:

    Policy to enable random sampling for 10% of traffic:

    Policy to disable trace reporting for the “foo” workload (note: tracing context will still be propagated):

    2. kind: Telemetry
    3. metadata:
    4.   name: foo-tracing
    5.   namespace: bar
    6. spec:
    7.   selector:
    8.     matchLabels:
    9.       service.istio.io/canonical-name: foo
    10.   tracing:
    11.   - disableSpanReporting: true

    Policy to select the alternate zipkin provider for trace reporting:

    1. apiVersion: telemetry.istio.io/v1alpha1
    2. kind: Telemetry
    3. metadata:
    4.   name: foo-tracing-alternate
    5.   namespace: baz
    6. spec:
    7.   selector:
    8.       service.istio.io/canonical-name: foo
    9.   tracing:
    10.   - providers:
    11.     - name: "zipkin-alternate"
    12.     randomSamplingPercentage: 10.00

    Policy to add a custom tag from a literal value:

    Policy to disable server-side metrics for Stackdriver for an entire mesh:

    1. apiVersion: telemetry.istio.io/v1alpha1
    2. kind: Telemetry
    3. metadata:
    4.   name: mesh-default
    5.   namespace: istio-system
    6. spec:
    7.   # no selector specified, applies to all workloads
    8.   metrics:
    9.   - providers:
    10.     - name: stackdriver
    11.     overrides:
    12.     - match:
    13.         metric: ALL_METRICS
    14.         mode: SERVER
    15.       disabled: true

    Policy to add dimensions to all Prometheus metrics for the foo namespace:

    1. apiVersion: telemetry.istio.io/v1alpha1
    2. kind: Telemetry
    3. metadata:
    4.   name: namespace-metrics
    5.   namespace: foo
    6. spec:
    7.   # no selector specified, applies to all workloads in the namespace
    8.   metrics:
    9.   - providers:
    10.     - name: prometheus
    11.     overrides:
    12.     # match clause left off matches all istio metrics, client and server
    13.     - tagOverrides:
    14.         request_method:
    15.           value: "request.method"
    16.         request_host:
    17.           value: "request.host"

    Policy to remove the response_code dimension on some Prometheus metrics for the bar.foo workload:

    Policy to enable access logging for the entire mesh:

    1. apiVersion: telemetry.istio.io/v1alpha1
    2. kind: Telemetry
    3.   name: mesh-default
    4.   namespace: istio-system
    5. spec:
    6.   # no selector specified, applies to all workloads
    7.   accessLogging:
    8.   - providers:
    9.     - name: envoy
    10.     # By default, this turns on access logging (no need to set `disabled:
    12.     except in # cases where a parent configuration has marked as `disabled:
    13.     true`. In # those cases, `disabled: false` must be set explicitly to
    14.     override.

    Policy to disable access logging for the foo namespace:

    1. apiVersion: telemetry.istio.io/v1alpha1
    2. kind: Telemetry
    3. metadata:
    4.   name: namespace-no-log
    5.   namespace: foo
    6. spec:
    7.   # no selector specified, applies to all workloads in the namespace
    8.   accessLogging:
    9.   - disabled: true

    Tracing

    Tracing configures tracing behavior for workloads within a mesh. It can be used to enable/disable tracing, as well as to set sampling rates and custom tag extraction.

    Tracing configuration support overrides of the fields providers, random_sampling_percentage, disable_span_reporting, and custom_tags at each level in the configuration hierarchy, with missing values filled in from parent resources. However, when specified, custom_tags will fully replace any values provided by parent configuration.

    FieldTypeDescriptionRequired
    match

    Allows tailoring of behavior to specific conditions.

    		No
    providersProviderRef[]

    Optional. Name of provider(s) to use for span reporting. If a provider is not specified, the [default tracing provider][istio.mesh.v1alpha1.MeshConfig.default_providers.tracing] will be used. NOTE: At the moment, only a single provider can be specified in a given Tracing rule.

    		No
    randomSamplingPercentage

    Controls the rate at which traffic will be selected for tracing if no prior sampling decision has been made. If a prior sampling decision has been made, that decision will be respected. However, if no sampling decision has been made (example: no x-b3-sampled tracing header was present in the requests), the traffic will be selected for telemetry generation at the percentage specified.

    Defaults to 0%. Valid values [0.00-100.00]. Can be specified in 0.01% increments.

    		No
    disableSpanReportingBoolValue

    Controls span reporting. If set to true, no spans will be reported for impacted workloads. This does NOT impact context propagation or trace sampling behavior.

    		No
    customTagsmap<string, >

    Optional. Configures additional custom tags to the generated trace spans.

    		No

    ProviderRef

    Used to bind Telemetry configuration to specific providers for targeted customization.

    FieldTypeDescriptionRequired
    namestring

    Required. Name of Telemetry provider in MeshConfig.

    		No

    Metrics

    Metrics defines the workload-level overrides for metrics generation behavior within a mesh. It can be used to enable/disable metrics generation, as well as to customize the dimensions of the generated metrics.

    FieldTypeDescriptionRequired
    providersProviderRef[]

    Optional. Name of providers to which this configuration should apply. If a provider is not specified, the [default metrics provider][istio.mesh.v1alpha1.MeshConfig.default_providers.metrics] will be used.

    		No
    overrides

    Optional. Ordered list of overrides to metrics generation behavior.

    Specified overrides will be applied in order. They will be applied on top of inherited overrides from other resources in the hierarchy in the following order: 1. Mesh-scoped overrides 2. Namespace-scoped overrides 3. Workload-scoped overrides

    Because overrides are applied in order, users are advised to order their overrides from least specific to most specific matches. That is, it is a best practice to list any universal overrides first, with tailored overrides following them.

    		No

    MetricSelector

    FieldTypeDescriptionRequired
    metric

    One of the well-known Istio Standard Metrics.

    		No
    customMetricstring (oneof)

    Allows free-form specification of a metric. No validation of custom metrics is provided.

    		No
    modeWorkloadMode

    Controls which mode of metrics generation is selected: CLIENT and/or SERVER.

    		No

    MetricsOverrides

    MetricsOverrides defines custom metric generation behavior for an individual metric or the set of all standard metrics.

    FieldTypeDescriptionRequired
    matchMetricSelector

    Match allows provides the scope of the override. It can be used to select individual metrics, as well as the workload modes (server and/or client) in which the metrics will be generated.

    If match is not specified, the overrides will apply to all metrics for both modes of operation (client and server).

    		No
    disabled

    Optional. Must explicitly set this to “true” to turn off metrics reporting for the listed metrics. If disabled has been set to “true” in a parent configuration, it must explicitly be set to “false” to turn metrics reporting on in the workloads selected by the Telemetry resource.

    		No
    tagOverridesmap<string, TagOverride>

    Optional. Collection of tag names and tag expressions to override in the selected metric(s). The key in the map is the name of the tag. The value in the map is the operation to perform on the the tag. WARNING: some providers may not support adding/removing tags. See also:

    		No

    Access logging defines the workload-level overrides for access log generation. It can be used to select provider or enable/disable access log generation for a workload.

    Tracing.TracingSelector

    TracingSelector provides a coarse-grained ability to configure tracing behavior based on certain traffic metadata (such as traffic direction).

    FieldTypeDescriptionRequired
    mode

    This determines whether or not to apply the tracing configuration based on the direction of traffic relative to the proxied workload.

    		No

    Tracing.CustomTag

    CustomTag defines a tag to be added to a trace span that is based on an operator-supplied value. This value can either be a hard-coded value, a value taken from an environment variable known to the sidecar proxy, or from a request header.

    NOTE: when specified, custom_tags will fully replace any values provided by parent configuration.

    FieldTypeDescriptionRequired
    literal

    Literal adds the same, hard-coded value to each span.

    		No
    Environment (oneof)

    Environment adds the value of an environment variable to each span.

    		No
    header

    RequestHeader adds the value of an header from the request to each span.

    		No

    Tracing.Literal

    FieldTypeDescriptionRequired
    valuestring

    The tag value to use.

    		No

    Tracing.Environment

    FieldTypeDescriptionRequired
    namestring

    Name of the environment variable from which to extract the tag value.

    		No
    defaultValuestring

    Optional. If the environment variable is not found, this value will be used instead.

    		No

    Tracing.RequestHeader

    FieldTypeDescriptionRequired
    namestring

    Name of the header from which to extract the tag value.

    		No
    defaultValuestring

    Optional. If the header is not found, this value will be used instead.

    		No

    TagOverride specifies an operation to perform on a metric dimension (also known as a label). Tags may be added, removed, or have their default values overridden.

    AccessLogging.LogSelector

    LogSelector provides a coarse-grained ability to configure logging behavior based on certain traffic metadata (such as traffic direction). LogSelector applies to traffic metadata which is not represented in the attribute set currently supported by Filters. It allows control planes to limit the configuration sent to individual workloads. Finer-grained logging behavior can be further configured via filter.

    FieldTypeDescriptionRequired
    modeWorkloadMode

    This determines whether or not to apply the access logging configuration based on the direction of traffic relative to the proxied workload.

    		No

    AccessLogging.Filter

    Allows specification of an access log filter.

    FieldTypeDescriptionRequired
    expressionstring

    CEL expression for selecting when requests/connections should be logged.

    Examples:

    • response.code >= 400
    • connection.mtls && request.url_path.contains(‘v1beta3’)
    		No

    MetricSelector.IstioMetric

    NameDescription
    ALL_METRICS

    Use of this enum indicates that the override should apply to all Istio default metrics.

    REQUEST_COUNT

    Counter of requests to/from an application, generated for HTTP, HTTP/2, and GRPC traffic.

    The Prometheus provider exports this metric as: istio_requests_total.

    The Stackdriver provider exports this metric as:

    • istio.io/service/server/request_count (SERVER mode)
    • istio.io/service/client/request_count (CLIENT mode)
    REQUEST_DURATION

    Histogram of request durations, generated for HTTP, HTTP/2, and GRPC traffic.

    The Prometheus provider exports this metric as: istio_request_duration_milliseconds.

    The Stackdriver provider exports this metric as:

    • istio.io/service/server/response_latencies (SERVER mode)
    • istio.io/service/client/roundtrip_latencies (CLIENT mode)
    REQUEST_SIZE

    Histogram of request body sizes, generated for HTTP, HTTP/2, and GRPC traffic.

    The Prometheus provider exports this metric as: istio_request_bytes.

    The Stackdriver provider exports this metric as:

    • istio.io/service/server/request_bytes (SERVER mode)
    • istio.io/service/client/request_bytes (CLIENT mode)
    RESPONSE_SIZE

    Histogram of response body sizes, generated for HTTP, HTTP/2, and GRPC traffic.

    The Prometheus provider exports this metric as: istio_response_bytes.

    The Stackdriver provider exports this metric as:

    • istio.io/service/server/response_bytes (SERVER mode)
    • istio.io/service/client/response_bytes (CLIENT mode)
    TCP_OPENED_CONNECTIONS

    Counter of TCP connections opened over lifetime of workload.

    The Prometheus provider exports this metric as: istio_tcp_connections_opened_total.

    The Stackdriver provider exports this metric as:

    • istio.io/service/server/connection_open_count (SERVER mode)
    • istio.io/service/client/connection_open_count (CLIENT mode)
    TCP_CLOSED_CONNECTIONS

    Counter of TCP connections closed over lifetime of workload.

    The Prometheus provider exports this metric as: istio_tcp_connections_closed_total.

    The Stackdriver provider exports this metric as:

    • istio.io/service/server/connection_close_count (SERVER mode)
    • istio.io/service/client/connection_close_count (CLIENT mode)
    TCP_SENT_BYTES

    Counter of bytes sent during a response over a TCP connection.

    The Prometheus provider exports this metric as: istio_tcp_sent_bytes_total.

    The Stackdriver provider exports this metric as:

    • istio.io/service/server/sent_bytes_count (SERVER mode)
    • istio.io/service/client/sent_bytes_count (CLIENT mode)
    TCP_RECEIVED_BYTES

    Counter of bytes received during a request over a TCP connection.

    The Prometheus provider exports this metric as: istio_tcp_received_bytes_total.

    The Stackdriver provider exports this metric as:

    • istio.io/service/server/received_bytes_count (SERVER mode)
    • istio.io/service/client/received_bytes_count (CLIENT mode)
    GRPC_REQUEST_MESSAGES

    Counter incremented for every gRPC messages sent from a client.

    The Prometheus provider exports this metric as: istio_request_messages_total

    GRPC_RESPONSE_MESSAGES

    Counter incremented for every gRPC messages sent from a server.

    The Prometheus provider exports this metric as: istio_response_messages_total

    MetricsOverrides.TagOverride.Operation

    NameDescription
    UPSERT

    Insert or Update the tag with the provided value expression. The value field MUST be specified if UPSERT is used as the operation.

    REMOVE

    Specifies that the tag should not be included in the metric when generated.

    WorkloadMode

    WorkloadMode allows selection of the role of the underlying workload in network traffic. A workload is considered as acting as a SERVER if it is the destination of the traffic (that is, traffic direction, from the perspective of the workload is inbound). If the workload is the source of the network traffic, it is considered to be in CLIENT mode (traffic is outbound from the workload).

    NameDescription
    CLIENT_AND_SERVER

    Selects for scenarios when the workload is either the source or destination of the network traffic.

    CLIENT

    Selects for scenarios when the workload is the destination of the network traffic.