If is set, the etcd server exports debugging information on its client port under the /debug path. Take care when setting —debug, since there will be degraded performance and verbose logging.

The /debug/pprof endpoint is the standard go runtime profiling endpoint. This can be used to profile CPU, heap, mutex, and goroutine utilization. For example, here go tool pprof gets the top 10 functions where etcd spends its time:

The /debug/requests endpoint gives gRPC traces and performance statistics through a web browser. For example, here is a Range request for the key abc:

  1. 2017/08/18 17:34:51.999317     0.000244     /etcdserverpb.KV/Range
  3. 17:34:51.999395      .    13     ... recv: key:"abc"
  4. 17:34:51.999499      .   104     ... OK
  5. 17:34:51.999535      .    36     ... sent: header:<cluster_id:14841639068965178418 member_id:10276657743932975437 revision:15 raft_term:17 > kvs:<key:"abc" create_revision:6 mod_revision:14 version:9 value:"asda" > count:1

Each etcd server exports metrics under the /metrics path on its client port and optionally on locations given by —listen-metrics-urls.

The metrics can be fetched with curl:

Running a Prometheus monitoring service is the easiest way to ingest and record etcd’s metrics.

First, install Prometheus:

  1. wget https://github.com/prometheus/prometheus/releases/download/v$PROMETHEUS_VERSION/prometheus-$PROMETHEUS_VERSION.linux-amd64.tar.gz -O /tmp/prometheus-$PROMETHEUS_VERSION.linux-amd64.tar.gz
  2. tar -xvzf /tmp/prometheus-$PROMETHEUS_VERSION.linux-amd64.tar.gz --directory /tmp/ --strip-components=1
  3. /tmp/prometheus -version

Set Prometheus’s scraper to target the etcd cluster endpoints:

Set up the Prometheus handler:

  1. nohup /tmp/prometheus \
  2.     -config.file /tmp/test-etcd.yaml \
  3.     -storage.local.path "test-etcd.data" >> /tmp/test-etcd.log  2>&1 &

Now Prometheus will scrape etcd metrics every 10 seconds.

Grafana

has built-in Prometheus support; just add a Prometheus data source:

Then import the default etcd dashboard template and customize. For instance, if Prometheus data source name is my-etcd, the field values in JSON also need to be my-etcd.

Sample dashboard: