Tracing

    • Request ID generation: Envoy will generate UUIDs when needed and populate the x-request-id HTTP header. Applications can forward the x-request-id header for unified logging as well as tracing. The behavior can be configured on a per basis using an extension.

    • Client trace ID joining: The x-client-trace-id header can be used to join untrusted request IDs to the trusted internal .

    • External trace service integration: Envoy supports pluggable external trace visualization providers, that are divided into two subgroups:

      • External tracers which are part of the Envoy code base, like LightStep, or any Zipkin compatible backends (e.g. Jaeger), , SkyWalking and .

      • External tracers which come as a third party plugin, like Instana.

    The HTTP connection manager that handles the request must have the object set. There are several ways tracing can be initiated:

    • By an internal service via the x-envoy-force-trace header.

    • Randomly sampled via the runtime setting.

    The router filter is also capable of creating a child span for egress calls via the start_child_span option.

    Envoy provides the capability for reporting tracing information regarding communications between services in the mesh. However, to be able to correlate the pieces of tracing information generated by the various proxies within a call flow, the services must propagate certain trace context between the inbound and outbound requests.

    Whichever tracing provider is being used, the service should propagate the to enable logging across the invoked services to be correlated.

    Attention

    Attention

    The sampling policy for Envoy is determined by the value of x-request-id by default. However, such a sampling policy is only valid for a fleet of Envoys. If a service proxy that is not Envoy is present in the fleet, sampling is performed without considering the policy of that proxy. For meshes consisting of multiple service proxies such as this, it is more effective to bypass Envoy’s sampling policy and sample based on the trace provider’s sampling policy. This can be achieved by setting to false.

    The tracing providers also require additional context, to enable the parent/child relationships between the spans (logical units of work) to be understood. This can be achieved by using the LightStep (via OpenTracing API) or Zipkin tracer directly within the service itself, to extract the trace context from the inbound request and inject it into any subsequent outbound requests. This approach would also enable the service to create additional spans, describing work being done internally within the service, that may be useful when examining the end-to-end trace.

    Alternatively the trace context can be manually propagated by the service:

    • When using the LightStep tracer, Envoy relies on the service to propagate the x-ot-span-context HTTP header while sending HTTP requests to other services.

    • When using the Zipkin tracer, Envoy relies on the service to propagate the B3 HTTP headers ( , x-b3-spanid, , x-b3-sampled, and ). The x-b3-sampled header can also be supplied by an external client to either enable or disable tracing for a particular request. In addition, the single header propagation format is supported, which is a more compressed format.

    • When using the Datadog tracer, Envoy relies on the service to propagate the Datadog-specific HTTP headers ( x-datadog-trace-id, , x-datadog-sampling-priority).

    • When using the SkyWalking tracer, Envoy relies on the service to propagate the SkyWalking-specific HTTP headers ( ).

    • When using the AWS X-Ray tracer, Envoy relies on the service to propagate the X-Ray-specific HTTP headers ( x-amzn-trace-id).

    An end-to-end trace is comprised of one or more spans. A span represents a logical unit of work that has a start time and duration and can contain metadata associated with it. Each span generated by Envoy contains the following data:

    • Originating service cluster set via .

    • Start time and duration of the request.

    • Originating host set via --service-node.

    • HTTP request URL, method, protocol and user-agent.

    • Additional custom tags set via .

    • Upstream cluster name, observability name, and address.

    • HTTP response status code.

    • GRPC response status and message (if available).

    • An error tag when HTTP status is 5xx or GRPC status is not “OK”.

    • Tracing system-specific metadata.

    The span also includes a name (or operation) which by default is defined as the host of the invoked service. However this can be customized using a config.route.v3.Decorator on the route. The name can also be overridden using the header.

    Envoy automatically sends spans to tracing collectors. Depending on the tracing collector, multiple spans are stitched together using common information such as the globally unique request ID x-request-id (LightStep) or the trace ID configuration (Zipkin and Datadog). See for more information on how to setup tracing in Envoy.

    Baggage provides a mechanism for data to be available throughout the entirety of a trace. While metadata such as tags are usually communicated to collectors out-of-band, baggage data is injected into the actual request context and available to applications during the duration of the request. This enables metadata to transparently travel from the beginning of the request throughout your entire mesh without relying on application-specific modifications for propagation. See OpenTracing’s documentation for more information about baggage.

    Tracing providers have varying level of support for getting and setting baggage:

    • Lightstep (and any OpenTracing-compliant tracer) can read/write baggage

    • Zipkin support is not yet implemented