Mirroring

    Traffic mirroring, also called shadowing, is a powerful concept that allows feature teams to bring changes to production with as little risk as possible. Mirroring sends a copy of live traffic to a mirrored service. The mirrored traffic happens out of band of the critical request path for the primary service.

    In this task, you will first force all traffic to of a test service. Then, you will apply a rule to mirror a portion of traffic to v2.

    • Set up Istio by following the instructions in the .

    • Start by deploying two versions of the httpbin service that have access logging enabled:

      httpbin-v1:

      httpbin-v2:

      1. $ cat <<EOF | istioctl kube-inject -f - | kubectl create -f -
      2. apiVersion: apps/v1
      3. kind: Deployment
      4. metadata:
      5.   name: httpbin-v2
      6. spec:
      7.   replicas: 1
      8.   selector:
      9.     matchLabels:
      10.       app: httpbin
      11.       version: v2
      12.   template:
      13.     metadata:
      14.       labels:
      15.         app: httpbin
      16.         version: v2
      17.     spec:
      18.       containers:
      19.       - image: docker.io/kennethreitz/httpbin
      20.         imagePullPolicy: IfNotPresent
      21.         name: httpbin
      22.         command: ["gunicorn", "--access-logfile", "-", "-b", "0.0.0.0:80", "httpbin:app"]
      23.         ports:
      24.         - containerPort: 80
      25. EOF
      1. $ kubectl create -f - <<EOF
      2. apiVersion: v1
      3. kind: Service
      4.   name: httpbin
      5.   labels:
      6.     app: httpbin
      7. spec:
      8.   ports:
      9.   - name: http
      10.     port: 8000
      12.   selector:
      13.     app: httpbin
      14. EOF

    • Start the sleep service so you can use curl to provide load:

      sleep service:

      1. $ cat <<EOF | istioctl kube-inject -f - | kubectl create -f -
      2. apiVersion: apps/v1
      3. kind: Deployment
      4. metadata:
      5.   name: sleep
      6. spec:
      7.   replicas: 1
      8.   selector:
      9.     matchLabels:
      10.       app: sleep
      11.   template:
      12.     metadata:
      13.       labels:
      14.         app: sleep
      15.     spec:
      16.       containers:
      17.       - name: sleep
      18.         image: curlimages/curl
      19.         command: ["/bin/sleep","3650d"]
      20.         imagePullPolicy: IfNotPresent
      21. EOF

    By default Kubernetes load balances across both versions of the httpbin service. In this step, you will change that behavior so that all traffic goes to v1.

    1. Create a default route rule to route all traffic to v1 of the service:

      Now all traffic goes to the httpbin:v1 service.

      1. $ export V1_POD=$(kubectl get pod -l app=httpbin,version=v1 -o jsonpath={.items..metadata.name})
      2. $ kubectl logs "$V1_POD" -c httpbin
      3. 127.0.0.1 - - [07/Mar/2018:19:02:43 +0000] "GET /headers HTTP/1.1" 200 321 "-" "curl/7.35.0"
      1. $ export V2_POD=$(kubectl get pod -l app=httpbin,version=v2 -o jsonpath={.items..metadata.name})
      2. $ kubectl logs "$V2_POD" -c httpbin
      3. <none>

    1. Change the route rule to mirror traffic to v2:

      This route rule sends 100% of the traffic to v1. The last stanza specifies that you want to mirror (i.e., also send) 100% of the same traffic to the httpbin:v2 service. When traffic gets mirrored, the requests are sent to the mirrored service with their Host/Authority headers appended with -shadow. For example, cluster-1 becomes cluster-1-shadow.

      Also, it is important to note that these requests are mirrored as “fire and forget”, which means that the responses are discarded.

      You can use the value field under the mirrorPercentage field to mirror a fraction of the traffic, instead of mirroring all requests. If this field is absent, all traffic will be mirrored.

    2. Send in traffic:

      1. $ kubectl exec "${SLEEP_POD}" -c sleep -- curl -sS http://httpbin:8000/headers

      Now, you should see access logging for both v1 and v2. The access logs created in v2 are the mirrored requests that are actually going to v1.

      1. $ kubectl logs "$V1_POD" -c httpbin
      2. 127.0.0.1 - - [07/Mar/2018:19:02:43 +0000] "GET /headers HTTP/1.1" 200 321 "-" "curl/7.35.0"
      3. 127.0.0.1 - - [07/Mar/2018:19:26:44 +0000] "GET /headers HTTP/1.1" 200 321 "-" "curl/7.35.0"
      1. $ kubectl logs "$V2_POD" -c httpbin
      2. 127.0.0.1 - - [07/Mar/2018:19:26:44 +0000] "GET /headers HTTP/1.1" 200 361 "-" "curl/7.35.0"

    2. Shutdown the httpbin service and client:

      1. $ kubectl delete deploy httpbin-v1 httpbin-v2 sleep