Telemetry

When telemetry is being streamed to an external metrics store, the interval is defined to be that store’s flush interval.

To view this data, you must send a signal to the Consul process: on Unix, this is while on Windows it is BREAK. Once Consul receives the signal, it will dump the current telemetry information to the agent’s stderr.

This telemetry information can be used for debugging or otherwise getting a better view of what Consul is doing. Review the Monitoring and Metrics tutorial to learn how collect and interpret Consul data.

Additionally, if the are provided, the telemetry information will be streamed to a statsite or server where it can be aggregated and flushed to Graphite or any other metrics store. For a configuration example for Telegraf, review the Monitoring with Telegraf tutorial.

This information can also be viewed with the in JSON format or using Prometheus format.

Sample output of telemetry dump

Key Metrics

These are some metrics emitted that can help you understand the health of your cluster at a glance. A Grafana dashboard is also available, which is maintained by the Consul team and displays these metrics for easy visualization. For a full list of metrics emitted by Consul, see

Why they’re important: Taken together, these metrics indicate how long it takes to complete write operations in various parts of the Consul cluster. Generally these should all be fairly consistent and no more than a few milliseconds. Sudden changes in any of the timing values could be due to unexpected load on the Consul servers, or due to problems on the servers themselves.

What to look for: Deviations (in any of these metrics) of more than 50% from baseline over the previous hour.

Leadership changes

Why they’re important: Normally, your Consul cluster should have a stable leader. If there are frequent elections or leadership changes, it would likely indicate network issues between the Consul servers, or that the Consul servers themselves are unable to keep up with the load.

What to look for: For a healthy cluster, you’re looking for a lastContact lower than 200ms, leader > 0 and candidate == 0. Deviations from this might indicate flapping leadership.

Autopilot

Why it’s important: Autopilot can expose the overall health of your cluster with a simple boolean.

What to look for: Alert if healthy is 0. Some other indicators of an unhealthy cluster would be:

• consul.raft.commitTime - This can help reflect the speed of state store changes being performed by the agent. If this number is rising, the server may be experiencing an issue due to degraded resources on the host.
• - Check for deviation from the recommended values. This can indicate failed leadership elections or flapping nodes.

Memory usage

Why they’re important: Consul keeps all of its data in memory. If Consul consumes all available memory, it will crash.

What to look for: If consul.runtime.sys_bytes exceeds 90% of total available system memory.

NOTE: This metric is calculated using Go’s runtime package . This will have a different output than using information gathered from top. For more information, see GH-4734.

Why it’s important: GC pause is a “stop-the-world” event, meaning that all runtime threads are blocked until GC completes. Normally these pauses last only a few nanoseconds. But if memory usage is high, the Go runtime may GC so frequently that it starts to slow down Consul.

What to look for: Warning if total_gc_pause_ns exceeds 2 seconds/minute, critical if it exceeds 5 seconds/minute.

NOTE: total_gc_pause_ns is a cumulative counter, so in order to calculate rates (such as GC/minute), you will need to apply a function such as InfluxDB’s .

Network activity - RPC Count

Why they’re important: These measurements indicate the current load created from a Consul agent, including when the load becomes high enough to be rate limited. A high RPC count, especially from consul.client.rpcexceeded meaning that the requests are being rate-limited, could imply a misconfigured Consul agent.

What to look for: Sudden large changes to the consul.client.rpc metrics (greater than 50% deviation from baseline). consul.client.rpc.exceeded or consul.client.rpc.failed count > 0, as it implies that an agent is being rate-limited or fails to make an RPC request to a Consul server

Raft Replication Capacity Issues

Why they’re important: These metrics allow operators to monitor the health and capacity of raft replication on servers. When Consul is handling large amounts of data and high write throughput it is possible for the cluster to get into the following state:

• Write throughput is high (say 500 commits per second or more) and constant
• The leader is writing out a large snapshot every minute or so
• The snapshot is large enough that it takes considerable time to restore from disk on a restart or from the leader if a follower gets behind
• Disk IO available allows the leader to write a snapshot faster than it can be restored from disk on a follower

In this state, followers must be able to restore a snapshot into memory and resume replication in under 80 seconds otherwise they will never be able to rejoin the cluster until write rates reduce. If they take more than 20 seconds then there will be a chance that they are unlucky with timing when they restart and have to download a snapshot again from the servers one or more times. If they take 50 seconds or more then they will likely fail to catch up more often than they succeed and will remain non-voters for some time until they happen to complete the restore just before the leader truncates its logs.

In the worst case, the follower will be left continually downloading snapshots from the leader which are always too old to use by the time they are restored. This can put additional strain on the leader transferring large snapshots repeatedly as well as reduce the fault tolerance and serving capacity of the cluster.

Since Consul 1.5.3 has been configurable. Increasing it allows the leader to retain more logs and give followers more time to restore and catch up. The tradeoff is potentially slower appends which eventually might affect write throughput and latency negatively so setting it arbitrarily high is not recommended. Before Consul 1.10.0 it required a rolling restart to change this configuration on the leader though and since no followers could restart without loosing health this could mean loosing cluster availability and needing to recover the cluster from a loss of quorum.

Since Consul 1.10.0 raft_trailing_logs is now reloadable with consul reload or SIGHUP allowing operators to increase this without the leader restarting or loosing leadership allowing the cluster to be recovered gracefully.

Monitoring these metrics can help avoid or diagnose this state.

What to look for:

consul.raft.leader.oldestLogAge should look like a saw-tooth wave increasing linearly with time until the leader takes a snapshot and then jumping down as the oldest logs are truncated. The lowest point on that line should remain comfortably higher (i.e. 2x or more) than the time it takes to restore a snapshot.

There are two ways a snapshot can be restored on a follower: from disk on startup or from the leader during an installSnapshot RPC. The leader only sends an installSnapshot RPC if the follower is new and has no state, or if it’s state is too old for it to catch up with the leaders logs.

consul.raft.fsm.lastRestoreDuration shows the time it took to restore from either source the last time it happened. Most of the time this is when the server was started. It’s a gauge that will always show the last restore duration (in Consul 1.10.0 and later) however long ago that was.

consul.raft.rpc.installSnapshot is the timing information from the leader’s perspective when it installs a new snapshot on a follower. It includes the time spent transferring the data as well as the follower restoring it. Since these events are typically infrequent, you may need to graph the last value observed, for example using max_over_time with a large range in Prometheus. While the restore part will also be reflected in lastRestoreDuration, it can be useful to observe this too since the logs need to be able to cover this entire operation including the snapshot delivery to ensure followers can always catch up safely.

Graphing consul.raft.leader.oldestLogAge on the same axes as the other two metrics here can help see at a glance if restore times are creeping dangerously close to the limit of what the leader is retaining at the current write rate.

Note that if servers don’t restart often, then the snapshot could have grown significantly since the last restore happened so last restore times might not reflect what would happen if an agent restarts now.

License Expiration

Enterprise

Why they’re important:

This measurement indicates how many hours are left before the Consul Enterprise license expires. When the license expires some Consul Enterprise features will cease to work. An example of this is that after expiration, it is no longer possible to create or modify resources in non-default namespaces or to manage namespace definitions themselves even though reads of namespaced resources will still work.

What to look for:

This metric should be monitored to ensure that the license doesn’t expire to prevent degradation of functionality.

Requirements:

Why they’re important:

The consul.raft.boltdb.storeLogs metric is a direct indicator of disk write performance of a Consul server. If there are issues with the disk or performance degradations related to Bolt DB, these metrics will show the issue and potentially the cause as well.

What to look for:

The primary thing to look for are increases in the consul.raft.boltdb.storeLogs times. Its value will directly govern an upper limit to the throughput of write operations within Consul.

There can be a number of potential issues that can cause this. Often times it could be performance of the underlying disks that is the issue. Other times it may be caused by Bolt DB behavior. Bolt DB keeps track of free space within the raft.db file. When needing to allocate data it will use existing free space first before further expanding the file. By default, Bolt DB will write a data structure containing metadata about free pages within the DB to disk for every log storage operation. Therefore if the free space within the database grows excessively large, such as after a large spike in writes beyond the normal steady state and a subsequent slow down in the write rate, then Bolt DB could end up writing a large amount of extra data to disk for each log storage operation. This has the potential to drastically increase disk write throughput, potentially beyond what the underlying disks can keep up with. To detect this situation you can look at the consul.raft.boltdb.freelistBytes metric. This metric is a count of the extra bytes that are being written for each log storage operation beyond the log data itself. While not a clear indicator of an actual issue, this metric can be used to diagnose why the consul.raft.boltdb.storeLogs metric is high.

If Bolt DB log storage performance becomes an issue and is caused by free list management then setting to true in the server’s configuration may help to reduce disk IO and log storage operation times. Disabling free list syncing will however increase the startup time for a server as it must scan the raft.db file for free space instead of loading the already populated free list structure.

This is a full list of metrics emitted by Consul.

These metrics are used to monitor the health of the Consul servers.

Requirements:

• Consul 1.12.0+

Label based RPC metrics were added in Consul 1.12.0 as a Beta feature to better understand the workload on a Consul server and, where that workload is coming from. The following metric(s) provide that insight

Note that values of the consul.rpc.server.call may emit as 0 ms. That means that the elapsed time < 1 ms.

Labels

The the server workload metrics above come with the following labels:

Label Explanations

The internal value for the rpc_type in the table above refers to leader and cluster management RPC operations that Consul performs. Historically, internal RPC operation metrics were accounted under the same metric names.

The unreported value for the request_type in the table above refers to RPC requests within Consul where it is difficult to ascertain whether a request is read or write type.

The unreported value for the leader label in the table above refers to RPC requests where Consul cannot determine the leadership status for a server.

Read Request Labels

In addition to the labels above, for read requests, the following may be populated:

Here is a Prometheus style example of an RPC metric and its labels:

Sample output of telemetry dump

Any metric in this section can be turned off with the prefix_filter.

These metrics give insight into the health of the cluster as a whole.

Consul Connect’s built-in proxy is by default configured to log metrics to the same sink as the agent that starts it.

When running in this mode it emits some basic metrics. These will be expanded upon in the future.

All metrics are prefixed with consul.proxy.<proxied-service-id> to distinguish between multiple proxies on a given host. The table below use web as an example service name for brevity.

Labels

Most labels have a dst label and some have a src label. When using metrics sinks and timeseries stores that support labels or tags, these allow aggregating the connections by service name.

Assuming all services are using a managed built-in proxy, you can get a complete overview of both number of open connections and bytes sent and received between all services by aggregating over these metrics.

For example aggregating over all upstream (i.e. outbound) connections which have both src and dst labels, you can get a sum of all the bandwidth in and out of a given service or the total number of connections between two services.

Metrics Reference