How-To: Use W3C trace context with Dapr

How to use trace context

Dapr uses W3C trace context for distributed tracing for both service invocation and pub/sub messaging. Dapr does all the heavy lifting of generating and propagating the trace context information and there are very few cases where you need to either propagate or create a trace context. First read scenarios in the article to understand whether you need to propagate or create a trace context.

To view traces, read the how to diagnose with tracing article.

For HTTP calls

OpenCensus Go SDK provides ochttp package that provides methods to retrieve trace context from http response.

To retrieve the trace context from HTTP response, you can use :

For gRPC calls

To retrieve the trace context header when the gRPC call is returned, you can pass the response header reference as gRPC call option which contains response headers:

  1. var responseHeader metadata.MD
  3. // Call the InvokeService with call option
  4. // grpc.Header(&responseHeader)
  6. client.InvokeService(ctx, &pb.InvokeServiceRequest{
  7.         Id: "client",
  8.         Message: &commonv1pb.InvokeRequest{
  9.             Method:      "MyMethod",
  10.             ContentType: "text/plain; charset=UTF-8",
  11.             Data:        &any.Any{Value: []byte("Hello")},
  12.         },
  13.     },
  14.     grpc.Header(&responseHeader))

Retrieve trace context in C

For HTTP calls

To retrieve the trace context from HTTP response, you can use .NET API :

  1. HttpResponseMessage response = await client.SendAsync(req);
  2. IEnumerable<string> values1, values2;
  3. string traceparentValue = "";
  4. string tracestateValue = "";
  5. if (response.Headers.TryGetValues("traceparent", out values1))
  6. {
  7.     traceparentValue = values1.FirstOrDefault();
  8. }
  9. if (response.Headers.TryGetValues("tracestate", out values2))
  10. {
  11.     tracestateValue = values2.FirstOrDefault();
  12. }

For gRPC calls

To retrieve the trace context from gRPC response, you can use Grpc.Net.Client ResponseHeadersAsync method.

  1. // client is Dapr proto client
  2. using var call = client.InvokeServiceAsync(req);
  3. var response = await call.ResponseAsync;
  4. var headers = await call.ResponseHeadersAsync();
  5. var tracecontext = headers.First(e => e.Key == "grpc-trace-bin");

Note: There are no helper methods exposed in Dapr SDKs to propagate and retrieve trace context. You need to use http/gRPC clients to propagate and retrieve trace headers through http headers and gRPC metadata.

Pass trace context in Go

For HTTP calls

OpenCensus Go SDK provides package that provides methods to attach trace context in http request.

  1. f := tracecontext.HTTPFormat{}
  2. req, _ := http.NewRequest("GET", "http://localhost:3500/v1.0/invoke/mathService/method/api/v1/add", nil)
  5. f.SpanContextToRequest(traceContext, req)

For gRPC calls

  1. traceContext := span.SpanContext()
  2. traceContextBinary := propagation.Binary(traceContext)

You can then pass the trace context through through grpc-trace-bin header.

  1. ctx = metadata.AppendToOutgoingContext(ctx, "grpc-trace-bin", string(traceContextBinary))

You can then continuing passing this go context ctx in subsequent Dapr gRPC calls as first parameter. For example InvokeService, context is passed in first parameter.

Pass trace context in C

For HTTP calls

To pass trace context in HTTP request, you can use .NET API :

For gRPC calls

To pass the trace context in gRPC call metadata, you can use Grpc.Net.Client ResponseHeadersAsync method.

  1. // client is Dapr.Client.Autogen.Grpc.v1
  2. var headers = new Metadata();
  3. headers.Add("grpc-trace-bin", tracecontext);
  4. using var call = client.InvokeServiceAsync(req, headers);

Additional general details on calling gRPC services with .NET client .

You can create a trace context using the recommended OpenCensus SDKs. OpenCensus supports several different programming languages.

Create trace context in Go

1. Get the OpenCensus Go SDK

2. Import the package “go.opencensus.io/trace”

$ go get -u go.opencensus.io

3. Create trace context

  1. ctx, span := trace.StartSpan(ctx, "cache.Get")
  2. defer span.End()
  4. // Do work to get from cache.
  1. try (Scope ss = TRACER.spanBuilder("cache.Get").startScopedSpan()) {
  2. }

Create trace context in Python

  1. with tracer.span(name="cache.get") as span:
  2.     pass

Create trace context in NodeJS

  1. tracer.startRootSpan({name: 'cache.Get'}, rootSpan => {
  2. });

Create trace context in C++

  1. opencensus::trace::Span span = opencensus::trace::Span::StartSpan(
  2.                                             "cache.Get", nullptr, {&sampler});

Create trace context in C

First you need to enable tracing configuration in Dapr. This step is mentioned for completeness from enabling tracing to invoking Dapr with trace context. Create a deployment config yaml e.g. appconfig.yaml with following configuration.

  1. apiVersion: dapr.io/v1alpha1
  2. kind: Configuration
  3. metadata:
  4.   name: appconfig
  5. spec:
  6.     samplingRate: "1"

In Kubernetes, you can apply the configuration as below :

  1. kubectl apply -f appconfig.yaml

You then set the following tracing annotation in your deployment YAML. You can add the following annotaion in sample grpc app deployment yaml.

  1. dapr.io/config: "appconfig"

Invoking Dapr with trace context

Dapr covers generating trace context and you do not need to explicitly create trace context.

However if you choose to pass the trace context explicitly, then Dapr will use the passed trace context and propagate all across the HTTP/gRPC call.

Using the grpc app in the example and putting this all together, the following steps show you how to create a Dapr client and call the InvokeService method passing the trace context:

The Rest code snippet and details, refer to the .

1. Import the package

  1. package main
  4.     pb "github.com/dapr/go-sdk/dapr"
  5.     "go.opencensus.io/trace"
  6.       "go.opencensus.io/trace/propagation"
  7.       "google.golang.org/grpc"
  8.       "google.golang.org/grpc/metadata"
  9. )

2. Create the client

  1.   // Get the Dapr port and create a connection
  2.   daprPort := os.Getenv("DAPR_GRPC_PORT")
  3.   daprAddress := fmt.Sprintf("localhost:%s", daprPort)
  4.   conn, err := grpc.Dial(daprAddress, grpc.WithInsecure())
  5.   if err != nil {
  6.     fmt.Println(err)
  7.   }
  8.   defer conn.Close()
  10.   // Create the client
  11.   client := pb.NewDaprClient(conn)

3. Invoke the InvokeService method With Trace Context

  1.   // Create the Trace Context
  2.   ctx , span := trace.StartSpan(context.Background(), "InvokeService")
  4.   // The returned context can be used to keep propagating the newly created span in the current context.
  5.   // In the same process, context.Context is used to propagate trace context.
  7.   // Across the process, use the propagation format of Trace Context to propagate trace context.
  8.   traceContext := propagation.Binary(span.SpanContext())
  9.   ctx = metadata.NewOutgoingContext(ctx, string(traceContext))
  11.   // Pass the trace context
  12.   resp, err := client.InvokeService(ctx, &pb.InvokeServiceRequest{
  13.         Id: "client",
  14.         Message: &commonv1pb.InvokeRequest{
  15.             Method:      "MyMethod",
  16.             ContentType: "text/plain; charset=UTF-8",
  17.             Data:        &any.Any{Value: []byte("Hello")},
  18.         },
  19.     })