If you deploy the TiKV cluster using TiUP, the monitoring and alert services are automatically deployed, and no manual deployment is needed.

Assume that the TiKV cluster topology is as follows:

  1. # Extracts the package.
  2. tar -xzf prometheus-2.8.1.linux-amd64.tar.gz
  3. tar -xzf node_exporter-0.17.0.linux-amd64.tar.gz
  4. tar -xzf grafana-6.1.6.linux-amd64.tar.gz

Step 2: Start node_exporter on all nodes

  1. cd node_exporter-0.17.0.linux-amd64
  3. # Starts the node_exporter service.
  4. $ ./node_exporter --web.listen-address=":9100" \
  5.     --log.level="info" &

Edit the Prometheus configuration file:

  1. ...
  3. global:
  4.   scrape_interval:     15s  # By default, scrape targets every 15 seconds.
  5.   evaluation_interval: 15s  # By default, scrape targets every 15 seconds.
  6.   # scrape_timeout is set to the global default value (10s).
  7.   external_labels:
  8.     cluster: 'test-cluster'
  9.     monitor: "prometheus"
  11. scrape_configs:
  12.   - job_name: 'overwritten-nodes'
  13.     honor_labels: true  # Do not overwrite job & instance labels.
  14.     static_configs:
  15.     - targets:
  16.       - '192.168.199.113:9100'
  17.       - '192.168.199.114:9100'
  18.       - '192.168.199.115:9100'
  19.       - '192.168.199.116:9100'
  20.       - '192.168.199.117:9100'
  21.       - '192.168.199.118:9100'
  24.     static_configs:
  25.     - targets:
  26.       - '192.168.199.113:2379'
  27.       - '192.168.199.114:2379'
  28.       - '192.168.199.115:2379'
  30.   - job_name: 'tikv'
  31.     honor_labels: true  # Do not overwrite job & instance labels.
  32.     static_configs:
  33.     - targets:
  34.       - '192.168.199.116:20180'
  35.       - '192.168.199.117:20180'
  36.       - '192.168.199.118:20180'
  38. ...

Start the Prometheus service:

  1. $ ./prometheus \
  2.     --config.file="./prometheus.yml" \
  3.     --web.listen-address=":9090" \
  4.     --web.external-url="http://192.168.199.113:9090/" \
  5.     --web.enable-admin-api \
  6.     --log.level="info" \
  7.     --storage.tsdb.path="./data.metrics" \
  8.     --storage.tsdb.retention="15d" &

Step 4: Start Grafana on Node1

Edit the Grafana configuration file:

  1. ...
  3. [paths]
  4. data = ./data
  5. logs = ./data/log
  6. plugins = ./data/plugins
  7. [server]
  8. http_port = 3000
  10. [database]
  11. [session]
  12. [analytics]
  13. check_for_updates = true
  14. [security]
  15. admin_user = admin
  16. admin_password = admin
  17. [snapshots]
  18. [auth.anonymous]
  19. [auth.basic]
  20. [auth.ldap]
  21. [smtp]
  22. [emails]
  23. [log]
  24. mode = file
  25. [log.console]
  26. [log.file]
  27. level = info
  28. format = text
  29. [log.syslog]
  30. [event_publisher]
  31. [dashboards.json]
  32. enabled = false
  33. path = ./data/dashboards
  34. [metrics]
  35. [grafana_net]
  36. url = https://grafana.net
  38. ...

Start the Grafana service:

  1. $ ./bin/grafana-server \
  2.     --config="./conf/grafana.ini" &

  1. Log in to the Grafana Web interface.

    For the Change Password step, you can choose Skip.

  2. In the Grafana sidebar menu, click Data Source within the Configuration.

  3. Click Add data source.

  4. Specify the data source information.

    • Specify a Name for the data source.
    • For Type, select Prometheus.
    • For URL, specify the Prometheus address.
    • Specify other fields as needed.
  5. Click Add to save the new data source.

Step 2: Import a Grafana dashboard

To import a Grafana dashboard for the PD server and the TiKV server, take the following steps respectively:

  2. In the sidebar menu, click Dashboards -> Import to open the Import Dashboard window.

  3. Click Upload .json File to upload a JSON file (Download TiKV Grafana configuration file and ).

  4. Click Load.

  5. Select a Prometheus data source.

Click New dashboard in the top menu and choose the dashboard you want to view.