OpenTelemetry

    • Setup Istio by following the instructions in the .

      The egress gateway and access logging will be enabled if you install the demo configuration profile.

    • Deploy the sample app to use as a test source for sending requests. If you have automatic sidecar injection enabled, run the following command to deploy the sample app:

      Otherwise, manually inject the sidecar before deploying the sleep application with the following command:

      Zip

      1. $ kubectl apply -f <(istioctl kube-inject -f @samples/sleep/sleep.yaml@)

      You can use any pod with curl installed as a test source.

    • Set the SOURCE_POD environment variable to the name of your source pod:

      1. $ export SOURCE_POD=$(kubectl get pod -l app=sleep -o jsonpath={.items..metadata.name})

    • Start the sample.

      If you have enabled automatic sidecar injection, deploy the httpbin service:

      1. $ kubectl apply -f @samples/httpbin/httpbin.yaml@

      Otherwise, you have to manually inject the sidecar before deploying the httpbin application:

      1. $ kubectl apply -f <(istioctl kube-inject -f @samples/httpbin/httpbin.yaml@)

    • Start the otel-collector sample.

    To enable access logging, use the .

    Edit MeshConfig to add an OpenTelemetry provider, named otel. This involves adding an extension provider stanza:

    1. extensionProviders:
    2. - name: otel
    3.   envoyOtelAls:
    4.     service: opentelemetry-collector.istio-system.svc.cluster.local
    5.     port: 4317

    The final configuration should look something like:

    1. apiVersion: v1
    2. kind: ConfigMap
    3. metadata:
    4.   name: istio
    5.   namespace: istio-system
    6. data:
    7.   mesh: |-
    8.     accessLogFile: /dev/stdout
    9.     defaultConfig:
    10.       proxyMetadata: {}
    11.       tracing:
    12.         zipkin:
    13.           address: zipkin.istio-system:9411
    14.     enablePrometheusMerge: true
    15.     extensionProviders:
    16.     - name: otel
    17.       envoyOtelAls:
    18.         service: opentelemetry-collector.istio-system.svc.cluster.local
    19.         port: 4317
    20.     rootNamespace: istio-system
    21.     trustDomain: cluster.local
    22.   meshNetworks: 'networks: {}'

    Next, add a Telemetry resource that tells Istio to send access logs to the OpenTelemetry collector.

    1. $ cat <<EOF | kubectl apply -n default -f -
    2. apiVersion: telemetry.istio.io/v1alpha1
    3. kind: Telemetry
    4. metadata:
    5.   name: sleep-logging
    6. spec:
    7.   selector:
    8.     matchLabels:
    9.       app: sleep
    10.   accessLogging:
    11.     - providers:
    12.       - name: otel
    13. EOF

    The above example uses the otel access log provider, and we do not configure anything other than default settings.

    Similar configuration can also be applied on an individual namespace, or to an individual workload, to control logging at a fine grained level.

    If you used an IstioOperator CR to install Istio, add the following field to your configuration:

    1. spec:
    3.     accessLogFile: /dev/stdout
    4.     extensionProviders:
    5.       envoyOtelAls:
    6.         service: opentelemetry-collector.istio-system.svc.cluster.local
    7.         port: 4317
    8.     defaultProviders:
    9.       accessLogging:
    10.       - envoy
    11.       - otel

    Otherwise, add the equivalent setting to your original istioctl install command, for example:

    Istio will use the following default access log format if accessLogFormat is not specified:

    1. [%START_TIME%] \"%REQ(:METHOD)% %REQ(X-ENVOY-ORIGINAL-PATH?:PATH)% %PROTOCOL%\" %RESPONSE_CODE% %RESPONSE_FLAGS% %RESPONSE_CODE_DETAILS% %CONNECTION_TERMINATION_DETAILS%
    2. \"%UPSTREAM_TRANSPORT_FAILURE_REASON%\" %BYTES_RECEIVED% %BYTES_SENT% %DURATION% %RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)% \"%REQ(X-FORWARDED-FOR)%\" \"%REQ(USER-AGENT)%\" \"%REQ(X-REQUEST-ID)%\"
    3. \"%REQ(:AUTHORITY)%\" \"%UPSTREAM_HOST%\" %UPSTREAM_CLUSTER% %UPSTREAM_LOCAL_ADDRESS% %DOWNSTREAM_LOCAL_ADDRESS% %DOWNSTREAM_REMOTE_ADDRESS% %REQUESTED_SERVER_NAME% %ROUTE_NAME%\n

    The following table shows an example using the default access log format for a request sent from sleep to httpbin:

    1. Send a request from sleep to httpbin:

      1. $ kubectl exec "$SOURCE_POD" -c sleep -- curl -sS -v httpbin:8000/status/418
      2. ...
      3. < HTTP/1.1 418 Unknown
      4. < server: envoy
      5. ...
      6.     -=[ teapot ]=-
      8.        _...._
      9.      .'  _ _ `.
      10.     | ."` ^ `". _,
      11.     \_;`"---"`|//
      12.       |       ;/
      13.       \_     _/
      14.         `"""`

    2. Check otel-collector’s log:

      1. $ kubectl logs -l app=opentelemetry-collector -n istio-system
      2. [2020-11-25T21:26:18.409Z] "GET /status/418 HTTP/1.1" 418 - via_upstream - "-" 0 135 3 1 "-" "curl/7.73.0-DEV" "84961386-6d84-929d-98bd-c5aee93b5c88" "httpbin:8000" "127.0.0.1:80" inbound|8000|| 127.0.0.1:41854 10.44.1.27:80 10.44.1.23:37652 outbound_.8000_._.httpbin.foo.svc.cluster.local default

    Note that the messages corresponding to the request appear in logs of the Istio proxies of both the source and the destination, sleep and httpbin, respectively. You can see in the log the HTTP verb (GET), the HTTP path (/status/418), the response code (418) and other .

    Shutdown the and httpbin services:

    Zip

    1. $ kubectl delete telemetry sleep-logging
    2. $ kubectl delete -f @samples/sleep/sleep.yaml@
    3. $ kubectl delete -f @samples/open-telemetry/otel.yaml@

    Disable Envoy’s access logging

    Remove, or set to "", the meshConfig.extensionProviders and setting in your Istio install configuration.