Using the Prometheus query log

    The query log can be toggled at runtime. It can therefore be activated when you want to investigate slownesses or high load on your Prometheus instance.

    To enable or disable the query log, two steps are needed:

    1. Adapt the configuration to add or remove the query log configuration.
    2. Reload the Prometheus server configuration.

    This example demonstrates how to log all the queries to a file called . We will assume that /prometheus is the data directory and that Prometheus has write access to it.

    First, adapt the prometheus.yml configuration file:

    Then, the Prometheus configuration:

    1. $ curl -X POST http://127.0.0.1:9090/-/reload

    Or, if Prometheus is not launched with --web.enable-lifecycle, and you’re not running on Windows, you can trigger the reload by sending a SIGHUP to the Prometheus process.

    To disable the query log, repeat the operation but remove query_log_file from the configuration.

    Prometheus conveniently exposes metrics that indicates if the query log is enabled and working:

    The first metric, prometheus_engine_query_log_enabled is set to 1 of the query log is enabled, and 0 otherwise. The second one, prometheus_engine_query_log_failures_total, indicates the number of queries that could not be logged.

    The query log is a JSON-formatted log. Here is an overview of the fields present for a query:

    1. {
    2.     "params": {
    3.         "query": "up == 0",
    5.         "step": 5
    6.     },
    7.     "stats": {
    8.         "timings": {
    9.             "evalTotalTime": 0.000447452,
    10.             "execQueueTime": 7.599e-06,
    11.             "execTotalTime": 0.000461232,
    12.             "innerEvalTime": 0.000427033,
    13.             "queryPreparationTime": 1.4177e-05,
    14.             "resultSortTime": 6.48e-07
    15.     },
    17. }
    • params: The query. The start and end timestamp, the step and the actual query statement.
    • stats: Statistics. Currently, it contains internal engine timers.
    • ts: The timestamp when the query ended.

    Additionally, depending on what triggered the request, you will have additional fields in the JSON lines.

    HTTP requests contain the client IP, the method, and the path:

    The client IP is the network IP address and does not take into consideration the headers like X-Forwarded-For. If you wish to log the original caller behind a proxy, you need to do so in the proxy itself.

    Recording rules and alerts contain a ruleGroup element which contains the path of the file and the name of the group:

    1. {
    2.     "ruleGroup": {
    3.         "file": "rules.yml",
    4.         "name": "partners"
    5.     }
    6. }

    Prometheus will not rotate the query log itself. Instead, you can use external tools to do so.

    One of those tools is logrotate. It is enabled by default on most Linux distributions.

    Here is an example of file you can add as /etc/logrotate.d/prometheus:

    That will rotate your file daily and keep one week of history.