Traffic Metrics

    You can add metrics to a mesh configuration, or to an individual data plane proxy configuration. For example, you might need metrics for individual data plane proxies to override the default metrics port if it’s already in use on the specified machine.

    Kuma provides full integration with Prometheus:

    • Each proxy can expose its metrics in Prometheus format.
    • Because metrics are part of the mesh configuration, Kuma exposes an API called the monitoring assignment service (MADS) which exposes every proxy in the mesh.

    To collect metrics from Kuma, you need to expose metrics from proxies and applications.

    In the rest of this page we assume you have already configured your observability tools to work with Kuma. If you haven’t already read the .

    To expose metrics from every proxy in the mesh, configure the resource:

    which is a shortcut for:

    1. apiVersion: kuma.io/v1alpha1
    2. kind: Mesh
    3. metadata:
    4.   name: default
    5. spec:
    6.   metrics:
    7.     enabledBackend: prometheus-1
    8.     backends:
    9.     - name: prometheus-1
    10.       type: prometheus
    11.       conf:
    12.         skipMTLS: false
    13.         port: 5670
    14.         path: /metrics
    15.         tags: # tags that can be referred in Traffic Permission when metrics are secured by mTLS  
    16.           kuma.io/service: dataplane-metrics
    1. type: Mesh
    2. name: default
    3. metrics:
    4.   enabledBackend: prometheus-1
    5.   backends:
    6.   - name: prometheus-1
    7.     type: prometheus
    8.     conf:
    9.       skipMTLS: true # by default mTLS metrics are also protected by mTLS. Scraping metrics with mTLS without transparent proxy is not supported at the moment.

    which is a shortcut for:

    1. type: Mesh
    2. name: default
    3. metrics:
    4.   enabledBackend: prometheus-1
    5.   backends:
    6.   - name: prometheus-1
    7.     type: prometheus
    8.     conf:
    9.       skipMTLS: true
    10.       port: 5670
    11.       path: /metrics
    12.       tags: # tags that can be referred in Traffic Permission when metrics are secured by mTLS  
    13.         kuma.io/service: dataplane-metrics

    This tells Kuma to configure every proxy in the default mesh to expose an HTTP endpoint with Prometheus metrics on port 5670 and URI path /metrics.

    In addition to exposing metrics from the data plane proxies, you might want to expose metrics from applications running next to the proxies. Kuma allows scraping Prometheus metrics from the applications endpoint running in the same Pod or VM. Later those metrics are aggregated and exposed at the same port/path as data plane proxy metrics. It is possible to configure it at the Mesh level, for all the applications in the Mesh, or just for specific applications.

    Here are reasons where you’d want to use this feature:

    • Application metrics are labelled with your mesh parameters (tags, mesh, data plane name…), this means that in mixed Universal and Kubernetes mode metrics are reported with the same types of labels.
    • Both application and sidecar metrics are scraped at the same time. This makes sure they are coherent (with 2 different scrapers they can end up scraping at different intervals and make metrics harder to correlate).
    • If you disable passthrough and your mesh uses mTLS but Prometheus is outside the mesh (skipMTLS: true) this will be the only way to retrieve these metrics as the application is completely hidden behind the sidecar.

    Any configuration change requires redeployment of the data plane.

    1. apiVersion: kuma.io/v1alpha1
    2. kind: Mesh
    3. metadata:
    4.   name: default
    5. spec:
    6.   metrics:
    7.     enabledBackend: prometheus-1
    8.     backends:
    9.     - name: prometheus-1
    10.       type: prometheus
    11.       conf:
    12.         skipMTLS: false
    13.         port: 5670
    14.         path: /metrics
    15.         tags: # tags that can be referred in Traffic Permission when metrics are secured by mTLS 
    16.           kuma.io/service: dataplane-metrics
    17.         aggregate:
    18.           - name: my-service # name of the metric, required to later disable/override with pod annotations 
    19.             path: "/metrics/prometheus"
    20.             port: 8888
    21.           - name: other-sidecar
    22.             port: 8000

    This configuration will cause every application in the mesh to be scrapped for metrics by the data plane proxy. If you need to expose metrics only for the specific application it is possible through annotation for Kubernetes or Dataplane resource for Universal deployment.

    Kubernetes allows to configure it through annotations. In case to configure you can use , where name is used to match the Mesh configuration and override or disable it.

    1. apiVersion: apps/v1
    2. kind: Deployment
    3. metadata:
    4.  namespace: kuma-example
    5.  name: kuma-tcp-echo
    6. spec:
    7.  ...
    8.  template:
    9.    metadata:
    10.      ...
    11.      annotations:
    12.        prometheus.metrics.kuma.io/aggregate-my-service-enabled: "false"  # causes that configuration from Mesh to be disabled and result in this endpoint's metrics to not be exposed
    13.        prometheus.metrics.kuma.io/aggregate-other-sidecar-port: "1234" # override port from Mesh
    14.        prometheus.metrics.kuma.io/aggregate-application-port: "80"
    15.        prometheus.metrics.kuma.io/aggregate-application-path: "/stats"
    16.    spec:
    17.      containers:
    18.      ...
    1. type: Dataplane
    2. mesh: default
    3. name: example
    4. metrics:
    5.   type: prometheus
    6.   conf:
    7.     path: /metrics/overridden
    8.     aggregate:
    9.     - name: my-service # causes that configuration from Mesh to be disabled and result in this endpoint's metrics to not be exposed
    10.       enabled: false
    11.     - name: other-sidecar
    12.       port: 1234 # override port from Mesh
    13.       address: "127.0.0.1"
    14.     - name: application
    15.       path: "/stats"
    16.       port: 80`

    By default, sidecar scrapes metrics on dataplane.networking.address. If you set dataplane.networking.inbound[].address.serviceAddress, and you want to scrape metrics from the application, you also need to set dataplane.metrics.conf.aggregate[].address.

    To override mesh-wide defaults for a particular Pod, use the following annotations:

    • prometheus.metrics.kuma.io/port - to override mesh-wide default port
    • prometheus.metrics.kuma.io/path - to override mesh-wide default path

    For example:

    1. apiVersion: apps/v1
    2. kind: Deployment
    3. metadata:
    4.   namespace: kuma-example
    5.   name: kuma-tcp-echo
    6. spec:
    7.   ...
    8.   template:
    9.     metadata:
    10.       ...
    11.       annotations:
    12.         prometheus.metrics.kuma.io/port: "1234"               # override Mesh-wide default port
    13.         prometheus.metrics.kuma.io/path: "/non-standard-path" # override Mesh-wide default path
    14.     spec:
    15.       containers:
    16.       ...

    To override mesh-wide defaults on a particular machine, configure the Dataplane resource:

    1. type: Dataplane
    2. mesh: default
    3. name: example
    4. metrics:
    5.   type: prometheus
    6.   conf:
    7.     skipMTLS: false
    8.     port: 1234
    9.     path: /non-standard-path

    This proxy exposes an HTTP endpoint with Prometheus metrics on port 1234 and URI path /non-standard-path.

    In case you don’t want to retrieve all Envoy’s metrics, it’s possible to filter them. Configuration is dynamic and doesn’t require a restart of a sidecar. You are able to specify which causes that metric’s endpoint returns only matching metrics. Also, you can set flag usedOnly that returns only metrics updated by Envoy.

    1. type: Mesh
    2. name: default
    4.   enabledBackend: prometheus-1
    5.   backends:
    6.     type: prometheus
    7.     conf:
    8.       port: 5670
    9.       path: /metrics
    10.       envoy:
    11.         filterRegex: http2_act.*
    12.         usedOnly: true

    Kuma lets you expose proxy metrics in a secure way by leveraging mTLS. Prometheus needs to be a part of the mesh for this feature to work, which is the default deployment mode on Kubernetes when using kumactl install observability.

    Make sure that mTLS is enabled in the mesh.

    1. apiVersion: kuma.io/v1alpha1
    2. kind: Mesh
    3. metadata:
    4.   name: default
    5. spec:
    6.   mtls:
    7.     enabledBackend: ca-1
    8.     backends:
    9.     - name: ca-1
    10.       type: builtin
    11.   metrics:
    12.     enabledBackend: prometheus-1
    13.     backends:
    14.     - name: prometheus-1
    15.       type: prometheus
    16.       conf:
    17.         port: 5670
    18.         path: /metrics
    19.         skipMTLS: false
    20.         tags: # tags that can be referred in a TrafficPermission resource 
    21.           kuma.io/service: dataplane-metrics

    If you have strict you will want to allow the traffic from Grafana to Prometheus and from Prometheus to data plane proxy metrics:

    1. apiVersion: kuma.io/v1alpha1
    2. kind: TrafficPermission
    3. mesh: default
    4. metadata:
    5.   name: metrics-permissions
    6. spec:
    7.   sources:
    8.     - match:
    9.        kuma.io/service: prometheus-server_mesh-observability_svc_80
    10.   destinations:
    11.     - match:
    12.        kuma.io/service: dataplane-metrics
    13. ---
    14. apiVersion: kuma.io/v1alpha1
    15. kind: TrafficPermission
    16. mesh: default
    17. metadata:
    18.   name: grafana-to-prometheus
    19. spec:
    20.    sources:
    21.    - match:
    22.       kuma.io/service: "grafana_mesh-observability_svc_80"
    23.    destinations:
    24.    - match:
    25.       kuma.io/service: "prometheus-server_mesh-observability_svc_80"

    Make sure that mTLS is enabled in the mesh.

    1. type: Mesh
    2. name: default
    3. spec:
    4.   mtls:
    5.     enabledBackend: ca-1
    6.     backends:
    7.     - name: ca-1
    8.       type: builtin
    9.   metrics:
    10.     enabledBackend: prometheus-1
    11.     backends:
    12.     - name: prometheus-1
    13.       type: prometheus
    14.       conf:
    15.         port: 5670
    16.         path: /metrics
    17.         skipMTLS: false

    If you have strict traffic permissions you will want to allow the traffic from Grafana to Prometheus and from Prometheus to data plane proxy metrics: