Profiling & Debugging

    In any real world scenario, an app might start exhibiting undesirable behavior in terms of resource spikes. CPU/Memory spikes are not uncommon in most cases.

    Dapr allows users to start an on-demand profiling session using through its profiling server endpoint and start an instrumentation session to discover problems and issues such as concurrency, performance, cpu and memory usage.

    Dapr allows you to enable profiling in both Kubernetes and stand-alone modes.

    To enable profiling in Standalone mode, pass the --enable-profiling and the --profile-port flags to the Dapr CLI: Note that profile-port is not required, and if not provided Dapr will pick an available port.

    To enable profiling in Kubernetes, simply add the dapr.io/enable-profiling annotation to your Dapr annotated pod:

    1.     dapr.io/enabled: "true"
    2.     dapr.io/app-id: "rust-app"

    Debug a profiling session

    After profiling is enabled, we can start a profiling session to investigate what’s going on with the Dapr runtime.

    For Standalone mode, locate the Dapr instance that you want to profile:

    1. dapr list
    2. APP ID           DAPR PORT     APP PORT  COMMAND      AGE  CREATED              PID
    3. node-subscriber  3500          3000      node app.js  12s  2019-09-09 15:11.24  896

    More info on pprof can be found .

    First, find the pod containing the Dapr runtime. If you don’t already know the the pod name, type kubectl get pods:

    If profiling has been enabled successfully, the runtime logs should show the following: time="2019-09-09T20:56:21Z" level=info msg="starting profiling server on port 7777"

    In this case, we want to start a session with the Dapr runtime inside of pod divideapp-6dddf7dc74-6sq4l.

    We can do so by connecting to the pod via port forwarding:

    1. Forwarding from 127.0.0.1:7777 -> 7777
    2. Forwarding from [::1]:7777 -> 7777
    3. Handling connection for 7777

    Now that the connection has been established, we can use pprof to profile the Dapr runtime.

    The following example will create a cpu.pprof file containing samples from a profile session that lasts 120 seconds:

    1. curl "http://localhost:7777/debug/pprof/profile?seconds=120" > cpu.pprof

    You can also save the results in a visualized way inside a PDF:

    1. go tool pprof --pdf your-binary-file http://localhost:7777/debug/pprof/profile?seconds=120 > profile.pdf

    For memory related issues, you can profile the heap:

    Profiling allocated objects:

    To analyze, grab the file path above (its a dynamic file path, so pay attention to note paste this one), and execute:

      alloc