Manual Installation

    If you intend to peer the service mesh to multiple Consul datacenters or partitions, you must use the Consul ECS Terraform module to install your service mesh on ECS. Manually configuring mesh gateways without using the gateway-task Terraform module is not supported.

    This topic does not include instructions for creating all AWS resources necessary to install Consul, such as a VPC or the ECS cluster. Refer to the linked guides in the section for complete, runnable examples.

    You should have some familiarity with AWS ECS. See What is Amazon Elastic Container Service for details.

    Task Definition

    Configure a task definition that creates the containers:

    • Your application container
    • An Envoy sidecar-proxy container
    • A Consul client container
    • A consul-ecs-mesh-init container for service mesh setup
    • (Optional) A consul-ecs-health-sync container to sync ECS health checks into Consul

    Your task definition must include the following top-level fields.

    The volumes list contains two bind mounts, named consul_data and consul_binary. Bind mounts are directories on the host which can be mounted into one or more containers in order to share files among containers. For Consul on ECS, certain binaries and configuration are shared among containers during task startup.

    1. {
    2.   "family": "my-example-client-app",
    3.   "networkMode": "awsvpc",
    4.   "volumes": [
    5.     {
    6.       "name": "consul_data"
    7.     },
    8.     {
    9.       "name": "consul_binary"
    10.     }
    11.   ],
    12.   "containerDefinitions": [...]
    13.   "tags": [
    14.     {
    15.       "key": "consul.hashicorp.com/mesh",
    16.       "value": "true"
    17.     },
    18.     {
    19.       "key": "consul.hashicorp.com/service-name",
    20.       "value":  "example-client-app"
    21.     }
    22.   ]
    23. }

    The tags list must include the following if you are using the ACL controller in a . Without these tags, the ACL controller will be unable to provision a service token for the task.

    Tag KeyTag ValueDescription
    consul.hashicorp.com/meshtrue (string)The ACL controller ignores tasks without this tag set to true.
    consul.hashicorp.com/service-nameConsul service nameSpecifies the Consul service associated with this task. Required if the service name is different than the task family.
    consul.hashicorp.com/partitionConsul admin partition
    Enterprise
    Specifies the Consul admin partition associated with this task. Defaults to the default admin partition if omitted.
    consul.hashicorp.com/namespaceConsul namespace
    Enterprise
    Specifies the Consul namespace associated with this task. Defaults to the default namespace if omitted.

    Application container

    First, include your application container in the containerDefinitions list in the task definition.

    1. {
    2.   "containerDefinitions": [
    3.     {
    4.       "name": "example-client-app",
    5.       "image": "docker.io/org/my_task:v0.0.1",
    6.       "essential": true,
    7.       "dependsOn": [
    8.         {
    9.           "containerName": "consul-ecs-mesh-init",
    10.           "condition": "SUCCESS"
    11.         },
    12.         {
    13.           "containerName": "sidecar-proxy",
    14.           "condition": "HEALTHY"
    15.         }
    16.       ],
    17.       ...
    18.     }
    19.   ]
    20. }
    1. {
    2.   "containerDefinitions": [
    3.     {
    4.       "name": "example-client-app",
    5.       "image": "docker.io/org/my_task:v0.0.1",
    6.       "essential": true,
    7.       "dependsOn": [
    8.         {
    9.           "containerName": "consul-ecs-mesh-init",
    10.           "condition": "SUCCESS"
    11.         },
    12.         {
    13.           "containerName": "sidecar-proxy",
    14.           "condition": "HEALTHY"
    15.         }
    16.       ],
    17.       ...
    18.     }
    19.   ]
    20. }
    Field nameTypeDescription
    namestringThe name of your application container.
    imagestringThe container image used to run your application.
    essentialbooleanMust be true to ensure the health of your application container affects the health status of the task.
    dependsOnlistMust be set as shown above. Container dependencies ensure your application container starts after service mesh setup is complete.

    See the documentation for a complete reference.

    The sidecar-proxy container runs Envoy proxy for Consul Connect. In most cases, the container should contain the following parameters and values.

    The mountPoints list must be set as shown in the following example. This will mount the shared consul_data volume into the sidecar-proxy container at the path /consul. This volume is where the consul-ecs-mesh-init container copies the envoy-bootstrap.json file and the consul-ecs binary, which are required to start Envoy. The dependsOn list must also be defined as follows to ensure the sidecar-proxy container starts after consul-ecs-mesh-init has successfully written these files to the shared volume.

    1. {
    2.   "containerDefinitions": [
    3.     {
    4.       "name": "example-client-app",
    5.       "image": "docker.io/org/my_task:v0.0.1",
    6.       ...
    7.     },
    8.     {
    9.       "name": "sidecar-proxy",
    10.       "image": "envoyproxy/envoy-alpine:<VERSION>",
    11.       "essential": false,
    12.       "dependsOn": [
    13.         {
    14.           "containerName": "consul-ecs-mesh-init",
    15.           "condition": "SUCCESS"
    16.         }
    17.       ],
    18.       "healthCheck": {
    19.         "retries": 3,
    20.         "command": ["nc", "-z", "127.0.0.1", "20000"],
    21.         "timeout": 5,
    22.         "interval": 30
    23.       },
    24.       "mountPoints": [
    25.         {
    26.           "readOnly": true,
    27.           "containerPath": "/consul",
    28.           "sourceVolume": "consul_data"
    29.         }
    30.       ],
    31.       "ulimits": [
    32.         {
    33.           "name": "nofile",
    34.           "softLimit": 1048576,
    35.           "hardLimit": 1048576
    36.         }
    37.       ],
    38.       "command": ["envoy", "--config-path", "/consul/envoy-bootstrap.json"],
    39.       "entryPoint": ["/consul/consul-ecs", "envoy-entrypoint"],
    40.     }
    41.   ]
    42. }
    1. {
    2.   "containerDefinitions": [
    3.     {
    4.       "name": "example-client-app",
    5.       "image": "docker.io/org/my_task:v0.0.1",
    6.       ...
    7.     },
    8.     {
    9.       "name": "sidecar-proxy",
    10.       "image": "envoyproxy/envoy-alpine:<VERSION>",
    11.       "essential": false,
    12.       "dependsOn": [
    13.         {
    14.           "containerName": "consul-ecs-mesh-init",
    15.           "condition": "SUCCESS"
    16.         }
    17.       ],
    18.       "healthCheck": {
    19.         "retries": 3,
    20.         "command": ["nc", "-z", "127.0.0.1", "20000"],
    21.         "timeout": 5,
    22.         "interval": 30
    23.       },
    24.       "mountPoints": [
    25.         {
    26.           "readOnly": true,
    27.           "containerPath": "/consul",
    28.           "sourceVolume": "consul_data"
    29.         }
    30.       ],
    31.       "ulimits": [
    32.         {
    33.           "name": "nofile",
    34.           "softLimit": 1048576,
    35.           "hardLimit": 1048576
    36.         }
    37.       ],
    38.       "command": ["envoy", "--config-path", "/consul/envoy-bootstrap.json"],
    39.       "entryPoint": ["/consul/consul-ecs", "envoy-entrypoint"],
    40.     }
    41. }

    The following table describes the necessary configuration settings.

    NOTE: Envoy and Consul must be compatible versions. See the in the Consul documentation.

    consul-client container

    Each task must include a Consul client container in order for the task to join your Consul cluster.

    1. {
    2.   "containerDefinitions": [
    3.     {
    4.       "name": "example-client-app",
    5.       "image": "docker.io/org/my_task:v0.0.1",
    6.       ...
    7.     },
    8.     {
    9.       "name": "sidecar-proxy",
    10.       "image": "envoyproxy/envoy-alpine:<ENVOY_VERSION>",
    11.       ...
    12.     }
    13.     {
    14.       "name": "consul-client",
    15.       "image": "public.ecr.aws/hashicorp/consul:<CONSUL_VERSION>",
    16.       "mountPoints": [
    17.         {
    18.           "readOnly": false,
    19.           "containerPath": "/consul",
    20.           "sourceVolume": "consul_data"
    21.         },
    22.         {
    23.           "containerPath": "/bin/consul-inject",
    24.           "sourceVolume": "consul_binary"
    25.         }
    26.       ],
    27.       "entryPoint": ["/bin/sh", "-ec"],
    28.       "command": [
    29.         "cp /bin/consul /bin/consul-inject/consul\n\nECS_IPV4=$(curl -s $ECS_CONTAINER_METADATA_URI_V4 | jq -r '.Networks[0].IPv4Addresses[0]')\n\n\ncat << EOF > /consul/agent-defaults.hcl\naddresses = {\n  dns = \"127.0.0.1\"\n  grpc = \"127.0.0.1\"\n  http = \"127.0.0.1\"\n}\nadvertise_addr = \"$ECS_IPV4\"\nadvertise_reconnect_timeout = \"15m\"\nclient_addr = \"0.0.0.0\"\ndatacenter = \"dc1\"\nenable_central_service_config = true\nleave_on_terminate = true\nports {\n  grpc = 8502\n}\nretry_join = [\n  \"<Consul server location>\",\n]\ntelemetry {\n  disable_compat_1.9 = true\n}\n\nEOF\n\ncat << EOF > /consul/agent-extra.hcl\naddresses = {\n  dns = \"0.0.0.0\"\n}\nlog_level = \"debug\"\n\nEOF\n\nexec consul agent \\\n    -data-dir /consul/data \\\n    -config-file /consul/agent-defaults.hcl \\\n    -config-file /consul/agent-extra.hcl\n"
    30.       ]
    31.     }
    32.   ]
    33. }
    Field nameTypeDescription
    namestringThe container name, which should always be consul-client.
    imagestringThe Consul image. Use our public AWS registry, public.ecr.aws/hashicorp/consul, to avoid rate limits.
    mountPointslistMust be set as shown above. Volumes are mounted to share information with other containers for task setup.
    entrypointlistMust be set to a plain shell so that the startup command works properly.
    commandlistSpecifies the contents of the . Copy the script and format it into a JSON string.

    The following script is used to start the Consul client for Consul on ECS.

    1. # Copy the consul binary to a shared volume for `consul-ecs-mesh-init` to use to generate Envoy configuration.
    2. cp /bin/consul /bin/consul-inject/consul
    4. # At runtime, determine the IP address assigned to this ECS Task.
    5. ECS_IPV4=$(curl -s $ECS_CONTAINER_METADATA_URI_V4 | jq -r '.Networks[0].IPv4Addresses[0]')
    7. # Write the Consul agent configuration file.
    8. cat << EOF > /consul/agent-defaults.hcl
    9. addresses = {
    10.   dns = "127.0.0.1"
    11.   grpc = "127.0.0.1"
    12.   http = "127.0.0.1"
    13. }
    14. advertise_addr = "$ECS_IPV4"
    15. advertise_reconnect_timeout = "15m"
    16. client_addr = "0.0.0.0"
    17. datacenter = "dc1"
    18. enable_central_service_config = true
    19. leave_on_terminate = true
    20. ports {
    21.   grpc = 8502
    22. }
    23. retry_join = ["<consul server location>"]
    24. telemetry {
    25.   disable_compat_1.9 = true
    26. }
    28. EOF
    30. # Start the consul agent.
    31. exec consul agent \
    32.     -data-dir /consul/data \
    33.     -config-file /consul/agent-defaults.hcl
    1. # Copy the consul binary to a shared volume for `consul-ecs-mesh-init` to use to generate Envoy configuration.
    2. cp /bin/consul /bin/consul-inject/consul
    4. # At runtime, determine the IP address assigned to this ECS Task.
    5. ECS_IPV4=$(curl -s $ECS_CONTAINER_METADATA_URI_V4 | jq -r '.Networks[0].IPv4Addresses[0]')
    7. # Write the Consul agent configuration file.
    8. cat << EOF > /consul/agent-defaults.hcl
    9. addresses = {
    10.   dns = "127.0.0.1"
    11.   grpc = "127.0.0.1"
    12.   http = "127.0.0.1"
    13. }
    14. advertise_addr = "$ECS_IPV4"
    15. advertise_reconnect_timeout = "15m"
    16. client_addr = "0.0.0.0"
    17. datacenter = "dc1"
    18. enable_central_service_config = true
    19. leave_on_terminate = true
    20. ports {
    21.   grpc = 8502
    22. }
    23. retry_join = ["<consul server location>"]
    24. telemetry {
    25.   disable_compat_1.9 = true
    26. }
    28. EOF
    30. # Start the consul agent.
    31. exec consul agent \
    32.     -data-dir /consul/data \
    33.     -config-file /consul/agent-defaults.hcl

    The following table describes the values that you should use to configure the command script:

    Field nameTypeDescription
    addresses.*stringsSet the DNS, GRPC, and HTTP addresses to 127.0.0.1 to ensure these are not accessible outside of the task.
    advertise_addrstringMust be set to the task IP address so that other Consul agents know how to reach this agent.
    client_addrstringMust be set to an interface reachable by other Consul agents.
    datacenterstringMust be set to the Consul datacenter this task will join.
    leave_on_terminatebooleanMust be set to true so that the Consul agent leaves the cluster gracefully before exiting.
    retry_joinstringMust be set to your Consul server location(s) so this agent can join the Consul cluster.

    Refer to the Consul Agent documentation for a complete reference of Consul agent configuration options.

    The consul-ecs-mesh-init container runs at task startup to setup this instance for Consul service mesh. It registers the service and proxy for this task with Consul and writes Envoy bootstrap configuration to a shared volume.

    1. {
    2.   "containerDefinitions": [
    3.     {
    4.       "name": "example-client-app",
    5.       "image": "docker.io/org/my_task:v0.0.1",
    6.       ...
    7.     },
    8.     {
    9.       "name": "sidecar-proxy",
    10.       "image": "envoyproxy/envoy-alpine:<ENVOY_VERSION>",
    11.       ...
    12.     },
    13.     {
    14.       "name": "consul-client"
    15.       "image": "public.ecr.aws/hashicorp/consul:<CONSUL_VERSION>",
    16.       ...
    17.     },
    18.     {
    19.       "name": "consul-ecs-mesh-init",
    20.       "image": "public.ecr.aws/hashicorp/consul-ecs:<CONSUL_ECS_VERSION>",
    21.       "command": ["mesh-init"],
    22.       "essential": false,
    23.       "environment": [
    24.         {
    25.           "name": "CONSUL_ECS_CONFIG_JSON",
    26.           "value": "{\"bootstrapDir\":\"/consul\",\"healthSyncContainers\":[],\"proxy\":{\"upstreams\":[{\"destinationName\":\"example-server-app\",\"localBindPort\":1234}]},\"service\":{\"checks\":[],\"meta\":{},\"name\":\"example-client-app\",\"port\":9090,\"tags\":[]}}"
    27.         }
    28.       ],
    29.       "mountPoints": [
    30.         {
    31.           "readOnly": false,
    32.           "containerPath": "/consul",
    33.           "sourceVolume": "consul_data"
    34.         },
    35.         {
    36.           "readOnly": true,
    37.           "containerPath": "/bin/consul-inject",
    38.           "sourceVolume": "consul_binary"
    39.         }
    40.       ]
    41.     }
    42.   ]
    43. }
    1. {
    2.   "containerDefinitions": [
    3.     {
    4.       "name": "example-client-app",
    5.       "image": "docker.io/org/my_task:v0.0.1",
    6.       ...
    7.     },
    8.     {
    9.       "name": "sidecar-proxy",
    10.       "image": "envoyproxy/envoy-alpine:<ENVOY_VERSION>",
    11.       ...
    12.     },
    13.     {
    14.       "name": "consul-client"
    15.       "image": "public.ecr.aws/hashicorp/consul:<CONSUL_VERSION>",
    16.       ...
    17.     },
    18.     {
    19.       "name": "consul-ecs-mesh-init",
    20.       "command": ["mesh-init"],
    21.       "essential": false,
    22.       "environment": [
    23.         {
    24.           "name": "CONSUL_ECS_CONFIG_JSON",
    25.           "value": "{\"bootstrapDir\":\"/consul\",\"healthSyncContainers\":[],\"proxy\":{\"upstreams\":[{\"destinationName\":\"example-server-app\",\"localBindPort\":1234}]},\"service\":{\"checks\":[],\"meta\":{},\"name\":\"example-client-app\",\"port\":9090,\"tags\":[]}}"
    26.         }
    27.       ],
    28.       "mountPoints": [
    29.         {
    30.           "readOnly": false,
    31.           "containerPath": "/consul",
    32.           "sourceVolume": "consul_data"
    33.         },
    35.           "readOnly": true,
    36.           "containerPath": "/bin/consul-inject",
    37.           "sourceVolume": "consul_binary"
    38.         }
    39.       ]
    40.     }
    41.   ]
    42. }

    Configuration is passed to the consul-ecs binary in JSON format using the CONSUL_ECS_CONFIG_JSON environment variable.

    The following is an example of the configuration that might be used for a service named example-client-app with one upstream service name example-server-app. The proxy and service blocks include information used by consul-ecs-mesh-init to perform with Consul during task startup. The same configuration format is used for the consul-ecs-health-sync container.

    1. {
    2.   "bootstrapDir": "/consul",
    3.   "healthSyncContainers": [],
    4.   "proxy": {
    5.     "upstreams": [
    6.       {
    7.         "destinationName": "example-server-app",
    8.         "localBindPort": 1234
    9.       }
    10.     ]
    11.   },
    12.   "service": {
    13.     "checks": [],
    14.     "meta": {},
    15.     "name": "example-client-app",
    16.     "port": 9090,
    17.     "tags": []
    18.   }
    19. }
    1. {
    2.   "bootstrapDir": "/consul",
    3.   "healthSyncContainers": [],
    4.   "proxy": {
    5.     "upstreams": [
    6.       {
    7.         "destinationName": "example-server-app",
    8.         "localBindPort": 1234
    9.       }
    10.     ]
    11.   },
    12.   "service": {
    13.     "checks": [],
    14.     "meta": {},
    15.     "name": "example-client-app",
    16.     "port": 9090,
    17.     "tags": []
    18.   }
    19. }
    Field nameTypeDescription
    bootstrapDirstringThis is the path of a shared volume that is mounted to other containers, where consul-ecs-mesh-init will write out Envoy configuration.
    healthSyncContainerslistUsed for health status syncing from ECS to Consul. See below for details.
    proxy.upstreamslistThe upstream services that your application calls over the service mesh, if any. The destinationName and localBindPort fields are required.
    service.namestringThe name used to register this service into the Consul service catalog.
    service.portintegerThe port your application listens on. Set to 0 if your application does not listen on any port.
    service.checkslistConsul to include so that Consul can run health checks against your application.

    See the Configuration Reference for a complete reference of fields.

    consul-ecs-health-sync container

    Optionally, Consul ECS can sync health checks for this task into Consul checks. This allows you to configure a health check for your application in one place and see a consistent health status in both ECS and Consul.

    For example, the following defines an ECS health check command that runs curl localhost:9090/health:

    1. {
    2.   "containerDefinitions": [
    3.     {
    4.       "name": "example-client-app",
    5.       "image": "docker.io/org/my_task:v0.0.1",
    6.       "healthCheck": {
    7.         "retries": 3,
    8.         "command": ["CMD-SHELL", "curl localhost:9090/health"],
    9.         "timeout": 5,
    10.         "interval": 30
    11.       },
    12.       ...
    13.     },
    14.     ...
    15.   ]
    16. }

    First, define which containers need their health status synced into Consul. To do this, add the container name(s) to the healthSyncContainers list of the CONSUL_ECS_CONFIG_JSON variable, as shown in the following example. This configuration must be passed to both the consul-ecs-mesh-init and consul-ecs-health-sync containers.

    1. {
    2.   "bootstrapDir": "/consul",
    3.   "healthSyncContainers": ["example-client-app"],
    4.   ...
    5. }
    1. {
    2.   "bootstrapDir": "/consul",
    3.   "healthSyncContainers": ["example-client-app"],
    4.   ...
    5. }
    1. {
    2.   "containerDefinitions": [
    3.     {
    4.       "name": "consul-ecs-mesh-init",
    5.       "image": "public.ecr.aws/hashicorp/consul-ecs:<VERSION>",
    6.       "environment": [
    7.         {
    8.           "name": "CONSUL_ECS_CONFIG_JSON",
    9.           "value": "{\"bootstrapDir\":\"/consul\",\"healthSyncContainers\":[\"example-client-app\"],\"proxy\":{\"upstreams\":[{\"destinationName\":\"example-server-app\",\"localBindPort\":1234}]},\"service\":{\"checks\":[],\"meta\":{},\"name\":\"example-client-app\",\"port\":9090,\"tags\":[]}}"
    10.         }
    11.       ],
    12.       ...
    13.     },
    14.     ...
    15.   ]
    16. }
    1. {
    2.   "containerDefinitions": [
    3.     {
    4.       "name": "consul-ecs-mesh-init",
    5.       "image": "public.ecr.aws/hashicorp/consul-ecs:<VERSION>",
    6.       "environment": [
    7.         {
    8.           "name": "CONSUL_ECS_CONFIG_JSON",
    9.           "value": "{\"bootstrapDir\":\"/consul\",\"healthSyncContainers\":[\"example-client-app\"],\"proxy\":{\"upstreams\":[{\"destinationName\":\"example-server-app\",\"localBindPort\":1234}]},\"service\":{\"checks\":[],\"meta\":{},\"name\":\"example-client-app\",\"port\":9090,\"tags\":[]}}"
    10.         }
    11.       ],
    12.       ...
    13.     },
    14.     ...
    15.   ]
    16. }

    Finally, include the consul-ecs-health-sync container in the containerDefinitions list. Pass the same value for CONSUL_ECS_CONFIG_JSON for both the consul-ecs-health-sync and consul-ecs-mesh-init containers.

    1. {
    2.   "containerDefinitions": [
    3.     {
    4.       "name": "example-client-app",
    5.       "image": "docker.io/org/my_task:v0.0.1",
    6.       ...
    7.     },
    8.     {
    9.       "name": "sidecar-proxy",
    10.       "image": "envoyproxy/envoy-alpine:<ENVOY_VERSION>",
    11.       ...
    12.     },
    13.     {
    14.       "name": "consul-client"
    15.       "image": "public.ecr.aws/hashicorp/consul:<CONSUL_VERSION>",
    16.       ...
    17.     },
    18.     {
    19.       "name": "consul-ecs-mesh-init",
    20.       "image": "public.ecr.aws/hashicorp/consul-ecs:<CONSUL_ECS_VERSION>",
    21.       ...
    22.     },
    23.     {
    24.       "name": "consul-ecs-health-sync",
    25.       "image": "public.ecr.aws/hashicorp/consul-ecs:<CONSUL_ECS_VERSION>",
    26.       "command": ["health-sync"],
    27.       "essential": false,
    28.       "dependsOn": [
    29.         {
    30.           "containerName": "consul-ecs-mesh-init",
    31.           "condition": "SUCCESS"
    32.         }
    33.       ],
    34.       "environment": [
    35.         {
    36.           "name": "CONSUL_ECS_CONFIG_JSON",
    37.           "value": "{\"bootstrapDir\":\"/consul\",\"healthSyncContainers\":[\"example-client-app\"],\"proxy\":{\"upstreams\":[{\"destinationName\":\"example-server-app\",\"localBindPort\":1234}]},\"service\":{\"checks\":[],\"meta\":{},\"name\":\"example-client-app\",\"port\":9090,\"tags\":[]}}"
    38.         }
    39.       ]
    40.     }
    41.   ]
    42. }
    Field nameTypeDescription
    namestringThe container name, which must be consul-ecs-health-sync.
    imagestringThe consul-ecs image. Use our public AWS registry, public.ecr.aws/hashicorp/consul-ecs, to avoid rate limits.
    commandlistMust be set to [“health-sync”] to run the consul-ecs health-sync command.
    dependsOnlistMust be set as shown above to ensure the health-sync container starts after service registration has completed.
    environmentlistMust include the CONSUL_ECS_CONFIG_JSON variable to pass configuration to the command.

    Next Steps

    • Create the task definition using the or the AWS CLI, or another method of your choice.
    • Follow the to get production-ready.