Terminating Gateways on Kubernetes

    This topic requires familiarity with Terminating Gateways.

    Adding a terminating gateway is a multi-step process:

    • Update the Helm chart with terminating gateway config options
    • Deploy the Helm chart
    • Access the Consul agent
    • Register external services with Consul

    Minimum required Helm options:

    config.yaml

    2.   name: consul
    3. connectInject:
    4.   enabled: true
    5. controller:
    6.   enabled: true
    7. terminatingGateways:
    8.   enabled: true

    Ensure you have the latest consul-helm chart and install Consul via helm using the following while being sure to provide the yaml configuration as previously discussed.

    You can access the Consul server directly from your host via kubectl port-forward. This is helpful for interacting with your Consul UI locally as well as to validate connectivity of the application.

    1. $ kubectl port-forward consul-server-0 8500 &
    1. $ kubectl port-forward consul-server-0 8500 &

    If TLS is enabled use port 8501:

    1. $ kubectl port-forward consul-server-0 8501 &
    1. $ kubectl port-forward consul-server-0 8501 &

    Be sure the latest consul binary is installed locally on your host. https://releases.hashicorp.com/consul/

    1. $ export CONSUL_HTTP_ADDR=http://localhost:8500
    1. $ export CONSUL_HTTP_ADDR=http://localhost:8500

    If TLS is enabled set:

    1. $ export CONSUL_HTTP_ADDR=https://localhost:8501
    2. $ export CONSUL_HTTP_SSL_VERIFY=false
    1. $ export CONSUL_HTTP_ADDR=https://localhost:8501
    2. $ export CONSUL_HTTP_SSL_VERIFY=false

    If ACLs are enabled also set:

    1. $ export CONSUL_HTTP_TOKEN=$(kubectl get secret consul-bootstrap-acl-token --template='{{.data.token | base64decode }}')
    1. $ export CONSUL_HTTP_TOKEN=$(kubectl get secret consul-bootstrap-acl-token --template='{{.data.token | base64decode }}')

    Registering the external services with Consul is a multi-step process:

    • Register external services with Consul
    • Update the terminating gateway ACL token if ACLs are enabled
    • Create a resource to configure the terminating gateway
    • Create a ServiceIntentions resource to allow access from services in the mesh to external service
    • Define upstream annotations for any services that need to talk to the external services

    Create a sample external service and register it with Consul.

    1. {
    2.   "Node": "example_com",
    3.   "Address": "example.com",
    4.   "NodeMeta": {
    5.     "external-node": "true",
    6.     "external-probe": "true"
    7.   },
    8.   "Service": {
    9.     "Address": "example.com",
    10.     "ID": "example-https",
    11.     "Service": "example-https",
    12.     "Port": 443
    13.   }
    14. }

    Terminating Gateways - 图2

    external.json

    1. {
    2.   "Node": "example_com",
    3.   "Address": "example.com",
    4.   "NodeMeta": {
    5.     "external-node": "true",
    6.     "external-probe": "true"
    7.   },
    8.   "Service": {
    9.     "Address": "example.com",
    10.     "ID": "example-https",
    11.     "Service": "example-https",
    12.     "Port": 443
    13.   }
    14. }
    • is our made up node name.
    • “Address”: “example.com” is the address of our node. Services registered to that node will use this address if their own address isn’t specified. If you’re registering multiple external services, ensure you use different node names with different addresses or set the Service.Address key.
    • is the address of our service. In this example this doesn’t need to be set since the address of the node is the same, but if there were two services registered to that same node then this should be set.

    Register the external service with Consul:

    1. $ curl --request PUT --data @external.json --insecure $CONSUL_HTTP_ADDR/v1/catalog/register
    2. true

    If ACLs and TLS are enabled :

    1. $ curl --request PUT --header "X-Consul-Token: $CONSUL_HTTP_TOKEN" --data @external.json --insecure $CONSUL_HTTP_ADDR/v1/catalog/register
    2. true
    1. $ curl --request PUT --header "X-Consul-Token: $CONSUL_HTTP_TOKEN" --data @external.json --insecure $CONSUL_HTTP_ADDR/v1/catalog/register
    2. true

    If ACLs are enabled, update the terminating gateway acl role to have service: write permissions on all of the services being represented by the gateway:

    • Create a new policy that includes these permissions
    • Update the existing role to include the new policy
    1. service "example-https" {
    2.   policy = "write"
    3. }

    write-policy.hcl

    1. service "example-https" {
    2.   policy = "write"
    3. }
    1. $ consul acl policy create -name "example-https-write-policy" -rules @write-policy.hcl
    2. Name:         example-https-write-policy
    3. Description:
    4. Datacenters:
    5. Rules:
    6. service "example-https" {
    7.   policy = "write"
    8. }
    1. $ consul acl policy create -name "example-https-write-policy" -rules @write-policy.hcl
    2. ID:           xxxxxxxxxxxxxxx
    3. Name:         example-https-write-policy
    4. Description:
    5. Datacenters:
    6. Rules:
    7. service "example-https" {
    9. }

    Now fetch the ID of the terminating gateway token

    1. consul acl role list | grep -B 6 -- "- RELEASE_NAME-terminating-gateway-policy" | grep ID
    3. ID:       <role id>
    1. consul acl role list | grep -B 6 -- "- RELEASE_NAME-terminating-gateway-policy" | grep ID
    3. ID:       <role id>

    Update the terminating gateway acl token with the new policy

    1. $ consul acl role update -id <role id> -policy-name example-https-write-policy
    2. AccessorID:       <role id>
    3. SecretID:         <secret id>
    4. Description:      RELEASE_NAME-terminating-gateway-acl-role
    5. Local:            true
    6. Create Time:      2021-01-08 21:18:47.957450486 +0000 UTC
    7. Policies:
    8.    63bf1d9b-a87d-8672-ddcb-d25e2d88adb8 - RELEASE_NAME-terminating-gateway-policy
    9.    f63d1ae6-ffe7-44bd-bf7a-704a86939a63 - example-https-write-policy
    1. $ consul acl role update -id <role id> -policy-name example-https-write-policy
    2. AccessorID:       <role id>
    3. SecretID:         <secret id>
    4. Description:      RELEASE_NAME-terminating-gateway-acl-role
    5. Local:            true
    6. Create Time:      2021-01-08 21:18:47.957450486 +0000 UTC
    7. Policies:
    8.    63bf1d9b-a87d-8672-ddcb-d25e2d88adb8 - RELEASE_NAME-terminating-gateway-policy
    9.    f63d1ae6-ffe7-44bd-bf7a-704a86939a63 - example-https-write-policy

    Once the roles have been updated, create the TerminatingGateway resource to configure the terminating gateway:

    1. apiVersion: consul.hashicorp.com/v1alpha1
    2. kind: TerminatingGateway
    3. metadata:
    4.   name: terminating-gateway
    5. spec:
    6.   services:
    7.     - name: example-https
    8.       caFile: /etc/ssl/certs/ca-certificates.crt

    Terminating Gateways - 图4

    1. apiVersion: consul.hashicorp.com/v1alpha1
    2. kind: TerminatingGateway
    3. metadata:
    4.   name: terminating-gateway
    5. spec:
    6.   services:
    7.     - name: example-https
    8.       caFile: /etc/ssl/certs/ca-certificates.crt

    If TLS is enabled, you must include the caFile parameter that points to the system trust store of the terminating gateway container. By default, the trust store is located in the /etc/ssl/certs/ca-certificates.crt directory.

    Configure the caFile parameter to point to the /etc/ssl/cert.pem directory if TLS is enabled and you are using one of the following components:

    • Consul Helm chart 0.43 or older
    • Or an Envoy image with an alpine base image

    Apply the TerminatingGateway resource with kubectl apply:

    1. $ kubectl apply --filename terminating-gateway.yaml

    If using ACLs and TLS, create a resource to allow access from services in the mesh to the external service

    1. apiVersion: consul.hashicorp.com/v1alpha1
    2. kind: ServiceIntentions
    3. metadata:
    4.   name: example-https
    5. spec:
    6.   destination:
    7.     name: example-https
    8.   sources:
    9.     - name: static-client
    10.       action: allow

    service-intentions.yaml

    1. apiVersion: consul.hashicorp.com/v1alpha1
    2. kind: ServiceIntentions
    3. metadata:
    4.   name: example-https
    5. spec:
    6.   destination:
    7.     name: example-https
    8.   sources:
    9.     - name: static-client
    10.       action: allow

    Apply the ServiceIntentions resource with kubectl apply:

    1. $ kubectl apply --filename service-intentions.yaml
    1. $ kubectl apply --filename service-intentions.yaml

    Finally define and deploy the external services as upstreams for the internal mesh services that wish to talk to them. An example deployment is provided which will serve as a static client for the terminating gateway service.

    1. apiVersion: v1
    2. kind: Service
    3.   name: static-client
    4. spec:
    5.   selector:
    7.   ports:
    8.     - port: 80
    9. ---
    10. apiVersion: v1
    11. kind: ServiceAccount
    12. metadata:
    13.   name: static-client
    14. ---
    15. apiVersion: apps/v1
    16. kind: Deployment
    17. metadata:
    18.   name: static-client
    19. spec:
    20.   replicas: 1
    21.   selector:
    22.     matchLabels:
    23.       app: static-client
    24.   template:
    25.     metadata:
    26.       name: static-client
    27.       labels:
    28.         app: static-client
    29.       annotations:
    30.         'consul.hashicorp.com/connect-inject': 'true'
    31.         'consul.hashicorp.com/connect-service-upstreams': 'example-https:1234'
    32.     spec:
    33.       containers:
    34.         - name: static-client
    35.           image: curlimages/curl:latest
    36.           command: ['/bin/sh', '-c', '--']
    37.           args: ['while true; do sleep 30; done;']
    38.       serviceAccountName: static-client

    Terminating Gateways - 图6

    static-client.yaml

    1. apiVersion: v1
    2. kind: Service
    3. metadata:
    4.   name: static-client
    5. spec:
    6.   selector:
    7.     app: static-client
    8.   ports:
    9.     - port: 80
    10. ---
    11. apiVersion: v1
    12. kind: ServiceAccount
    13. metadata:
    14.   name: static-client
    15. ---
    16. apiVersion: apps/v1
    17. kind: Deployment
    18. metadata:
    19.   name: static-client
    20. spec:
    21.   replicas: 1
    22.   selector:
    23.     matchLabels:
    24.       app: static-client
    25.   template:
    26.     metadata:
    27.       name: static-client
    28.       labels:
    29.         app: static-client
    30.       annotations:
    31.         'consul.hashicorp.com/connect-inject': 'true'
    32.         'consul.hashicorp.com/connect-service-upstreams': 'example-https:1234'
    33.     spec:
    34.       containers:
    35.         - name: static-client
    36.           image: curlimages/curl:latest
    37.           command: ['/bin/sh', '-c', '--']
    38.           args: ['while true; do sleep 30; done;']
    39.       serviceAccountName: static-client

    Run the service via kubectl apply:

    1. $ kubectl apply --filename static-client.yaml
    1. $ kubectl apply --filename static-client.yaml

    Wait for the service to be ready:

    1. $ kubectl rollout status deploy static-client --watch
    2. deployment "static-client" successfully rolled out
    1. $ kubectl rollout status deploy static-client --watch
    2. deployment "static-client" successfully rolled out
    1. $ kubectl exec deploy/static-client -- curl -vvvs --header "Host: example-https.com" http://localhost:1234/