Traces For Kubernetes System Components

System component traces record the latency of and relationships between operations in the cluster.

Kubernetes components emit traces using the OpenTelemetry Protocol with the gRPC exporter and can be collected and routed to tracing backends using an .

For a complete guide to collecting traces and using the collector, see . However, there are a few things to note that are specific to Kubernetes components.

The kube-apiserver generates spans for incoming HTTP requests, and for outgoing requests to webhooks, etcd, and re-entrant requests. It propagates the W3C Trace Context with outgoing requests but does not make use of the trace context attached to incoming requests, as the kube-apiserver is often a public endpoint.

Enabling tracing in the kube-apiserver

To enable tracing, enable the APIServerTracing feature gate on the kube-apiserver. Also, provide the kube-apiserver with a tracing configuration file with . This is an example config that records spans for 1 in 10000 requests, and uses the default OpenTelemetry endpoint:

For more information about the TracingConfiguration struct, see .

kubelet traces

The kubelet CRI interface and authenticated http servers are instrumented to generate trace spans. As with the apiserver, the endpoint and sampling rate are configurable. Trace context propagation is also configured. A parent span’s sampling decision is always respected. A provided tracing configuration sampling rate will apply to spans without a parent. Enabled without a configured endpoint, the default OpenTelemetry Collector reciever address of “localhost:4317” is set.

Enabling tracing in the kubelet

To enable tracing, enable the KubeletTracing feature gate on the kubelet. Also, provide the kubelet with a . This is an example snippet of a kubelet config that records spans for 1 in 10000 requests, and uses the default OpenTelemetry endpoint:

Tracing instrumentation is still under active development, and may change in a variety of ways. This includes span names, attached attributes, instrumented endpoints, etc. Until this feature graduates to stable, there are no guarantees of backwards compatibility for tracing instrumentation.