Watches

    Watches are implemented using blocking queries in the HTTP API. Agents automatically make the proper API calls to watch for changes and inform a handler when the data view has updated.

    Watches can be configured as part of the , causing them to run once the agent is initialized. Reloading the agent configuration allows for adding or removing watches dynamically.

    Alternatively, the watch command enables a watch to be started outside of the agent. This can be used by an operator to inspect data in Consul or to easily pipe data into processes without being tied to the agent lifecycle.

    In either case, the of the watch must be specified. Each type of watch supports different parameters, some required and some optional. These options are specified in a JSON body when using agent configuration or as CLI flags for the watch command.

    The watch configuration specifies the view of data to be monitored. Once that view is updated, the specified handler is invoked. Handlers can be either an executable or an HTTP endpoint. A handler receives JSON formatted data with invocation info, following a format that depends on the type of the watch. Each watch type documents the format type. Because they map directly to an HTTP API, handlers should expect the input to match the format of the API. A Consul index is also given, corresponding to the responses from the .

    An executable handler reads the JSON invocation info from stdin. Additionally, the CONSUL_INDEX environment variable will be set to the Consul index. Anything written to stdout is logged.

    Here is an example configuration, where handler_type is optionally set to script:

    Consul watch with script handler defined in agent configuration

    Consul watch with script handler defined in agent configuration

    HCL

    • HCL
    • JSON

    Watches - 图2

    agent-config.hcl

    1. watches = [
    2.   {
    3.     type = "key"
    4.     key = "foo/bar/baz"
    5.     handler_type = "script"
    6.     args = ["/usr/bin/my-service-handler.sh", "-redis"]
    7.   }
    8. ]
    1. {
    2.   "watches": [
    3.     {
    4.       "type": "key",
    5.       "key": "foo/bar/baz",
    6.       "handler_type": "script",
    7.       "args": ["/usr/bin/my-service-handler.sh", "-redis"]
    8.     }
    9.   ]
    10. }

    agent-config.json

    1. {
    2.   "watches": [
    3.     {
    4.       "type": "key",
    5.       "key": "foo/bar/baz",
    6.       "handler_type": "script",
    7.       "args": ["/usr/bin/my-service-handler.sh", "-redis"]
    8.     }
    9.   ]
    10. }

    Prior to Consul 1.0, watches used a single handler field to define the command to run, and would always run in a shell. In Consul 1.0, the args array was added so that handlers can be run without a shell. The handler field is deprecated, and you should include the shell in the args to run under a shell, eg. "args": ["sh", "-c", "..."].

    HTTP endpoint

    An HTTP handler sends an HTTP request when a watch is invoked. The JSON invocation info is sent as a payload along the request. The response also contains the Consul index as a header named X-Consul-Index.

    The HTTP handler can be configured by setting handler_type to http. Additional handler options are set using http_handler_config. The only required parameter is the path field which specifies the URL to the HTTP endpoint. Consul uses POST as the default HTTP method, but this is also configurable. Other optional fields are header, timeout and tls_skip_verify. The watch invocation data is always sent as a JSON payload.

    Here is an example configuration:

    Consul watch with HTTP handler defined in agent configuration

    Consul watch with HTTP handler defined in agent configuration

    HCL

    Watches - 图4

    • HCL
    • JSON
    1. watches = [
    2.   {
    3.     type = "key"
    4.     key = "foo/bar/baz"
    5.     handler_type = "http"
    6.     http_handler_config {
    7.       path = "https://localhost:8000/watch"
    8.       method = "POST"
    9.       header = {
    10.         x-foo = ["bar", "baz"]
    11.       }
    12.       timeout = "10s"
    13.       tls_skip_verify = false
    14.     }
    15.   }
    16. ]

    agent-config.hcl

    1. watches = [
    2.   {
    3.     type = "key"
    4.     key = "foo/bar/baz"
    5.     handler_type = "http"
    6.     http_handler_config {
    7.       path = "https://localhost:8000/watch"
    8.       method = "POST"
    9.       header = {
    10.         x-foo = ["bar", "baz"]
    11.       }
    12.       timeout = "10s"
    13.       tls_skip_verify = false
    14.     }
    15.   }
    16. ]
    1. {
    2.   "watches": [
    3.     {
    4.       "type": "key",
    5.       "key": "foo/bar/baz",
    6.       "handler_type": "http",
    7.       "http_handler_config": {
    8.         "path": "https://localhost:8000/watch",
    9.         "method": "POST",
    10.         "header": { "x-foo": ["bar", "baz"] },
    11.         "timeout": "10s",
    12.         "tls_skip_verify": false
    13.       }
    14.     }
    15.   ]
    16. }

    Watches - 图6

    agent-config.json

    1. {
    2.   "watches": [
    3.     {
    4.       "type": "key",
    5.       "key": "foo/bar/baz",
    6.       "handler_type": "http",
    7.       "http_handler_config": {
    8.         "path": "https://localhost:8000/watch",
    9.         "method": "POST",
    10.         "header": { "x-foo": ["bar", "baz"] },
    11.         "timeout": "10s",
    12.         "tls_skip_verify": false
    13.       }
    14.     }
    15.   ]
    16. }

    In addition to the parameters supported by each option type, there are a few global parameters that all watches support:

    • - Can be provided to override the agent’s default datacenter.
    • token - Can be provided to override the agent’s default ACL token.
    • - The handler subprocess and arguments to invoke when the data view updates.
    • handler - The handler shell command to invoke when the data view updates.

    The following types are supported. Detailed documentation on each is below:

    • - Watch a specific KV pair
    • keyprefix - Watch a prefix in the KV store
    • - Watch the list of available services
    • nodes - Watch the list of nodes
    • - Watch the instances of a service
    • checks - Watch the value of health checks
    • - Watch for custom user events

    Type: key

    The “key” watch type is used to watch a specific key in the KV store. It requires that the key parameter be specified.

    This maps to the /v1/kv/ API internally.

    Here is an example configuration:

    Example key watch type

    Example key watch type

    HCL

    • HCL
    • JSON
    1. {
    2.   type = "key"
    3.   key = "foo/bar/baz"
    4.   args = ["/usr/bin/my-service-handler.sh", "-redis"]
    5. }
    1. {
    2.   type = "key"
    3.   key = "foo/bar/baz"
    4.   args = ["/usr/bin/my-service-handler.sh", "-redis"]
    5. }
    1. {
    2.   "type": "key",
    3.   "key": "foo/bar/baz",
    4.   "args": ["/usr/bin/my-service-handler.sh", "-redis"]
    5. }
    1. {
    2.   "type": "key",
    3.   "key": "foo/bar/baz",
    4.   "args": ["/usr/bin/my-service-handler.sh", "-redis"]
    5. }

    Or, using the watch command:

    1. $ consul watch -type=key -key=foo/bar/baz /usr/bin/my-key-handler.sh
    1. $ consul watch -type=key -key=foo/bar/baz /usr/bin/my-key-handler.sh

    An example of the output of this command:

    1. {
    2.   "Key": "foo/bar/baz",
    3.   "CreateIndex": 1793,
    4.   "ModifyIndex": 1793,
    5.   "LockIndex": 0,
    6.   "Flags": 0,
    7.   "Value": "aGV5",
    8.   "Session": ""
    9. }
    1. {
    2.   "Key": "foo/bar/baz",
    3.   "CreateIndex": 1793,
    4.   "ModifyIndex": 1793,
    5.   "LockIndex": 0,
    6.   "Flags": 0,
    7.   "Value": "aGV5",
    8.   "Session": ""
    9. }

    The keyprefix watch type is used to watch a prefix of keys in the KV store. It requires that the prefix parameter be specified. This watch returns all keys matching the prefix whenever any key matching the prefix changes.

    This maps to the /v1/kv/ API internally.

    Here is an example configuration:

    Example keyprefix watch type

    Example keyprefix watch type

    HCL

    • HCL
    • JSON
    1. {
    2.   type = "keyprefix"
    3.   prefix = "foo/"
    4.   args = ["/usr/bin/my-prefix-handler.sh", "-redis"]
    5. }
    1. {
    2.   type = "keyprefix"
    3.   prefix = "foo/"
    4.   args = ["/usr/bin/my-prefix-handler.sh", "-redis"]
    5. }
    1. {
    2.   "type": "keyprefix",
    3.   "prefix": "foo/",
    4.   "args": ["/usr/bin/my-prefix-handler.sh", "-redis"]
    5. }
    1. {
    2.   "type": "keyprefix",
    3.   "prefix": "foo/",
    4.   "args": ["/usr/bin/my-prefix-handler.sh", "-redis"]
    5. }

    Or, using the watch command:

    1. $ consul watch -type=keyprefix -prefix=foo/ /usr/bin/my-prefix-handler.sh
    1. $ consul watch -type=keyprefix -prefix=foo/ /usr/bin/my-prefix-handler.sh

    An example of the output of this command:

    1. [
    2.   {
    3.     "Key": "foo/bar",
    4.     "CreateIndex": 1796,
    5.     "ModifyIndex": 1796,
    6.     "LockIndex": 0,
    7.     "Flags": 0,
    8.     "Value": "TU9BUg==",
    9.     "Session": ""
    10.   },
    11.   {
    12.     "Key": "foo/baz",
    13.     "ModifyIndex": 1795,
    14.     "LockIndex": 0,
    15.     "Flags": 0,
    16.     "Value": "YXNkZg==",
    17.     "Session": ""
    18.   },
    20.     "Key": "foo/test",
    21.     "CreateIndex": 1793,
    22.     "ModifyIndex": 1793,
    23.     "LockIndex": 0,
    24.     "Flags": 0,
    25.     "Value": "aGV5",
    26.     "Session": ""
    27.   }
    28. ]
    1. [
    2.   {
    3.     "Key": "foo/bar",
    4.     "CreateIndex": 1796,
    5.     "ModifyIndex": 1796,
    6.     "LockIndex": 0,
    7.     "Flags": 0,
    8.     "Value": "TU9BUg==",
    9.     "Session": ""
    10.   },
    11.   {
    12.     "Key": "foo/baz",
    13.     "CreateIndex": 1795,
    14.     "ModifyIndex": 1795,
    15.     "LockIndex": 0,
    16.     "Flags": 0,
    17.     "Value": "YXNkZg==",
    18.     "Session": ""
    19.   },
    20.   {
    21.     "Key": "foo/test",
    22.     "CreateIndex": 1793,
    23.     "ModifyIndex": 1793,
    24.     "LockIndex": 0,
    25.     "Flags": 0,
    26.     "Value": "aGV5",
    27.     "Session": ""
    28.   }
    29. ]

    Type: services

    The “services” watch type is used to watch the list of available services. It has no parameters.

    This maps to the /v1/catalog/services API internally.

    Below is an example configuration:

    Example services watch type

    Example services watch type

    HCL

    Watches - 图9

    • HCL
    • JSON
    1. {
    2.   type = "services"
    3.   args = ["/usr/bin/my-services-handler.sh"]
    4. }
    1. {
    2.   type = "services"
    3.   args = ["/usr/bin/my-services-handler.sh"]
    4. }
    1. {
    2.   "type": "services",
    3.   "args": ["/usr/bin/my-services-handler.sh"]
    4. }

    Or, using the watch command:

    1. $ consul watch -type=services /usr/bin/my-services-handler.sh
    1. $ consul watch -type=services /usr/bin/my-services-handler.sh

    An example of the output of this command:

    1. {
    2.   "consul": [],
    3.   "redis": [],
    4.   "web": []
    5. }
    1. {
    2.   "consul": [],
    3.   "redis": [],
    4.   "web": []
    5. }

    Type: nodes

    The “nodes” watch type is used to watch the list of available nodes. It has no parameters.

    This maps to the /v1/catalog/nodes API internally.

    Below is an example configuration:

    Example nodes watch type

    Example nodes watch type

    HCL

    • HCL
    • JSON
    1. {
    2.   type = "nodes"
    3.   args = ["/usr/bin/my-nodes-handler.sh"]
    4. }
    1. {
    2.   type = "nodes"
    3.   args = ["/usr/bin/my-nodes-handler.sh"]
    4. }
    1. {
    2.   "type": "nodes",
    3.   "args": ["/usr/bin/my-nodes-handler.sh"]
    4. }
    1. {
    2.   "type": "nodes",
    3.   "args": ["/usr/bin/my-nodes-handler.sh"]
    4. }

    Or, using the watch command:

    1. $ consul watch -type=nodes /usr/bin/my-nodes-handler.sh
    1. $ consul watch -type=nodes /usr/bin/my-nodes-handler.sh

    An example of the output of this command:

    1. [
    2.   {
    3.     "ID": "8d3088b5-ce7d-0b94-f185-ae70c3445642",
    4.     "Node": "nyc1-consul-1",
    5.     "Address": "192.0.2.10",
    6.     "Datacenter": "dc1",
    7.     "TaggedAddresses": null,
    8.     "Meta": null,
    9.     "CreateIndex": 23792324,
    10.     "ModifyIndex": 23792324
    11.   },
    12.   {
    13.     "ID": "1edb564e-65ee-9e60-5e8a-83eae4637357",
    14.     "Node": "nyc1-worker-1",
    15.     "Address": "192.0.2.20",
    16.     "Datacenter": "dc1",
    17.     "TaggedAddresses": {
    18.       "lan": "192.0.2.20",
    19.       "lan_ipv4": "192.0.2.20",
    20.       "wan": "192.0.2.20",
    21.       "wan_ipv4": "192.0.2.20"
    22.     },
    23.     "Meta": {
    24.       "consul-network-segment": "",
    25.       "host-ip": "192.0.2.20",
    26.       "pod-name": "hashicorp-consul-q7nth"
    27.     },
    28.     "CreateIndex": 23792336,
    29.     "ModifyIndex": 23792338
    30.   }
    31. ]
    1. [
    2.   {
    3.     "ID": "8d3088b5-ce7d-0b94-f185-ae70c3445642",
    4.     "Node": "nyc1-consul-1",
    5.     "Address": "192.0.2.10",
    6.     "Datacenter": "dc1",
    7.     "TaggedAddresses": null,
    8.     "Meta": null,
    9.     "CreateIndex": 23792324,
    10.     "ModifyIndex": 23792324
    11.   },
    12.   {
    13.     "ID": "1edb564e-65ee-9e60-5e8a-83eae4637357",
    14.     "Node": "nyc1-worker-1",
    15.     "Address": "192.0.2.20",
    16.     "Datacenter": "dc1",
    17.     "TaggedAddresses": {
    18.       "lan": "192.0.2.20",
    19.       "lan_ipv4": "192.0.2.20",
    20.       "wan": "192.0.2.20",
    21.       "wan_ipv4": "192.0.2.20"
    22.     },
    23.     "Meta": {
    24.       "consul-network-segment": "",
    25.       "host-ip": "192.0.2.20",
    26.       "pod-name": "hashicorp-consul-q7nth"
    27.     },
    28.     "CreateIndex": 23792336,
    29.     "ModifyIndex": 23792338
    30.   }
    31. ]

    The “service” watch type is used to monitor the providers of a single service. It requires the service parameter and optionally takes the parameters tag and passingonly. The tag parameter will filter by one or more tags. It may be either a single string value or a slice of strings. The passingonly parameter is a boolean that will filter to only the instances passing all health checks.

    This maps to the /v1/health/service API internally.

    Here is an example configuration with a single tag:

    Example service watch type

    Example service watch type

    HCL

    • HCL
    • JSON
    1. {
    2.   type = "service"
    3.   service = "redis"
    4.   args = ["/usr/bin/my-service-handler.sh", "-redis"]
    5.   tag = "bar"
    6. }
    1. {
    2.   type = "service"
    3.   service = "redis"
    4.   args = ["/usr/bin/my-service-handler.sh", "-redis"]
    5.   tag = "bar"
    6. }
    1. {
    2.   "type": "service",
    3.   "service": "redis",
    4.   "args": ["/usr/bin/my-service-handler.sh", "-redis"],
    5.   "tag": "bar"
    6. }
    1. {
    2.   "type": "service",
    3.   "service": "redis",
    4.   "args": ["/usr/bin/my-service-handler.sh", "-redis"],
    5.   "tag": "bar"
    6. }

    Here is an example configuration with multiple tags:

    Example service watch type with multiple tags

    Example service watch type with multiple tags

    HCL

    Watches - 图12

    • HCL
    • JSON
    1. {
    2.   type = "service"
    3.   service = "redis"
    4.   args = ["/usr/bin/my-service-handler.sh", "-redis"]
    5.   tag = ["bar", "foo"]
    6. }
    1. {
    2.   type = "service"
    3.   service = "redis"
    4.   args = ["/usr/bin/my-service-handler.sh", "-redis"]
    5.   tag = ["bar", "foo"]
    6. }
    1. {
    2.   "type": "service",
    3.   "service": "redis",
    4.   "args": ["/usr/bin/my-service-handler.sh", "-redis"],
    5. }
    1. {
    2.   "type": "service",
    3.   "service": "redis",
    5.   "tag": ["bar", "foo"]
    6. }

    Or, using the watch command:

    Single tag:

    1. $ consul watch -type=service -service=redis -tag=bar /usr/bin/my-service-handler.sh
    1. $ consul watch -type=service -service=redis -tag=bar /usr/bin/my-service-handler.sh

    Multiple tags:

    1. $ consul watch -type=service -service=redis -tag=bar -tag=foo /usr/bin/my-service-handler.sh
    1. $ consul watch -type=service -service=redis -tag=bar -tag=foo /usr/bin/my-service-handler.sh

    An example of the output of this command:

    1. [
    2.   {
    3.     "Node": {
    4.       "ID": "f013522f-aaa2-8fc6-c8ac-c84cb8a56405",
    5.       "Node": "hashicorp-consul-server-1",
    6.       "Address": "192.0.2.50",
    7.       "Datacenter": "dc1",
    8.       "TaggedAddresses": null,
    9.       "Meta": null,
    10.       "CreateIndex": 23785783,
    11.       "ModifyIndex": 23785783
    12.     },
    13.     "Service": {
    14.       "ID": "redis",
    15.       "Service": "redis",
    16.       "Tags": [],
    17.       "Meta": null,
    18.       "Port": 6379,
    19.       "Address": "",
    20.       "Weights": {
    21.         "Passing": 1,
    22.         "Warning": 1
    23.       },
    24.       "EnableTagOverride": false,
    25.       "CreateIndex": 23785794,
    26.       "ModifyIndex": 23785794,
    27.       "Proxy": {
    28.         "MeshGateway": {},
    29.         "Expose": {}
    30.       },
    31.       "Connect": {}
    32.     },
    33.     "Checks": [
    34.       {
    35.         "Node": "hashicorp-consul-server-1",
    36.         "CheckID": "serfHealth",
    37.         "Name": "Serf Health Status",
    38.         "Status": "passing",
    39.         "Notes": "",
    40.         "Output": "Agent alive and reachable",
    41.         "ServiceID": "",
    42.         "ServiceName": "",
    43.         "ServiceTags": [],
    44.         "Type": "",
    45.         "Definition": {
    46.           "Interval": "0s",
    47.           "Timeout": "0s",
    48.           "DeregisterCriticalServiceAfter": "0s",
    49.           "HTTP": "",
    50.           "Header": null,
    51.           "Method": "",
    52.           "Body": "",
    53.           "TLSServerName": "",
    54.           "TLSSkipVerify": false,
    55.           "TCP": "",
    56.           "GRPC": "",
    57.           "GRPCUseTLS": false
    58.         },
    59.         "CreateIndex": 23785783,
    60.         "ModifyIndex": 23791503
    61.       }
    62.     ]
    63.   }
    64. ]

    Type: checks

    The “checks” watch type is used to monitor the checks of a given service or those in a specific state. It optionally takes the service parameter to filter to a specific service or the state parameter to filter to a specific state. By default, it will watch all checks.

    This maps to the /v1/health/state/ API if monitoring by state or /v1/health/checks/ if monitoring by service.

    Here is an example configuration for monitoring by state:

    Example checks watch type for all services in the passing state

    Example checks watch type for all services in the passing state

    HCL

    • HCL
    • JSON
    1. {
    2.   type = "checks"
    3.   state = "passing"
    4.   args = ["/usr/bin/my-check-handler.sh", "-passing"]
    5. }
    1. {
    2.   type = "checks"
    3.   state = "passing"
    4.   args = ["/usr/bin/my-check-handler.sh", "-passing"]
    5. }
    1. {
    2.   "type": "checks",
    3.   "state": "passing",
    4.   "args": ["/usr/bin/my-check-handler.sh", "-passing"]
    5. }
    1. {
    2.   "type": "checks",
    3.   "state": "passing",
    4.   "args": ["/usr/bin/my-check-handler.sh", "-passing"]
    5. }

    Here is an example configuration for monitoring by service:

    Example checks watch type for a specific service

    Example checks watch type for a specific service

    HCL

    Watches - 图14

    • HCL
    • JSON
    1. {
    2.   type = "checks"
    3.   service = "redis"
    4.   args = ["/usr/bin/my-check-handler.sh", "-redis"]
    5. }
    1. {
    2.   type = "checks"
    3.   service = "redis"
    4.   args = ["/usr/bin/my-check-handler.sh", "-redis"]
    5. }
    1. {
    2.   "type": "checks",
    3.   "service": "redis",
    4.   "args": ["/usr/bin/my-check-handler.sh", "-redis"]
    5. }
    1. {
    2.   "type": "checks",
    3.   "service": "redis",
    4.   "args": ["/usr/bin/my-check-handler.sh", "-redis"]
    5. }

    Or, using the watch command:

    State:

    1. $ consul watch -type=checks -state=passing /usr/bin/my-check-handler.sh -passing
    1. $ consul watch -type=checks -state=passing /usr/bin/my-check-handler.sh -passing

    Service:

    1. $ consul watch -type=checks -service=redis /usr/bin/my-check-handler.sh -redis
    1. $ consul watch -type=checks -service=redis /usr/bin/my-check-handler.sh -redis

    An example of the output of this command:

    1. [
    2.   {
    3.     "Node": "foobar",
    4.     "CheckID": "service:redis",
    5.     "Name": "Service 'redis' check",
    6.     "Status": "passing",
    7.     "Notes": "",
    8.     "Output": "",
    9.     "ServiceID": "redis",
    10.     "ServiceName": "redis"
    11.   }
    12. ]
    1. [
    2.   {
    3.     "Node": "foobar",
    4.     "CheckID": "service:redis",
    5.     "Name": "Service 'redis' check",
    6.     "Status": "passing",
    7.     "Notes": "",
    8.     "Output": "",
    9.     "ServiceID": "redis",
    10.     "ServiceName": "redis"
    11.   }
    12. ]

    Type: event

    The “event” watch type is used to monitor for custom user events. These are fired using the command. It takes only a single optional name parameter which restricts the watch to only events with the given name.

    This maps to the /v1/event/list API internally.

    Here is an example configuration:

    Example event watch type

    Example event watch type

    HCL

    • HCL
    • JSON
    1. {
    2.   type = "event"
    3.   name = "web-deploy"
    4.   args = ["/usr/bin/my-event-handler.sh", "-web-deploy"]
    5. }
    1. {
    2.   type = "event"
    3.   name = "web-deploy"
    4.   args = ["/usr/bin/my-event-handler.sh", "-web-deploy"]
    5. }
    1. {
    2.   "type": "event",
    3.   "name": "web-deploy",
    4.   "args": ["/usr/bin/my-event-handler.sh", "-web-deploy"]
    5. }
    1. {
    2.   "type": "event",
    3.   "name": "web-deploy",
    4.   "args": ["/usr/bin/my-event-handler.sh", "-web-deploy"]
    5. }

    Or, using the watch command:

    1. $ consul watch -type=event -name=web-deploy /usr/bin/my-event-handler.sh -web-deploy
    1. $ consul watch -type=event -name=web-deploy /usr/bin/my-event-handler.sh -web-deploy

    An example of the output of this command:

    1. [
    2.   {
    3.     "ID": "f07f3fcc-4b7d-3a7c-6d1e-cf414039fcee",
    4.     "Name": "web-deploy",
    5.     "Payload": "MTYwOTAzMA==",
    6.     "NodeFilter": "",
    7.     "ServiceFilter": "",
    8.     "TagFilter": "",
    9.     "Version": 1,
    10.     "LTime": 18
    11.   },
    12.   ...
    13. ]
    1. [
    2.   {
    3.     "ID": "f07f3fcc-4b7d-3a7c-6d1e-cf414039fcee",
    4.     "Name": "web-deploy",
    5.     "Payload": "MTYwOTAzMA==",
    6.     "NodeFilter": "",
    7.     "ServiceFilter": "",
    8.     "TagFilter": "",
    9.     "Version": 1,
    10.     "LTime": 18
    11.   },
    12.   ...
    13. ]

    To fire a new web-deploy event the following could be used:

      1. $ consul event -name=web-deploy 1609030