Networking and security observability with Hubble

    If you haven’t read the Introduction to Cilium & Hubble yet, we’d encourage you to do that first.

    The best way to get help if you get stuck is to ask a question on the . With Cilium contributors across the globe, there is almost always someone available to help.

    To run a Kubernetes cluster on your local machine, you have the choice to either set up a single-node cluster with minikube, or a local multi-node cluster on Docker using :

    • runs a single-node Kubernetes cluster inside a Virtual Machine (VM) and is the easiest way to run a Kubernetes cluster locally.
    • kind runs a multi-node Kubernetes using Docker container to emulate cluster nodes. It allows you to experiment with the cluster-wide observability features of Hubble Relay.

    When unsure about the option to pick, follow the instructions for minikube as it is less likely to cause friction.

    Single-node cluster with minikube

    Multi-node cluster with kind

    Install kubectl & minikube

    1. Install kubectl version >= v1.10.0 as described in the Kubernetes Docs
    2. Install minikube >= v1.3.1 as per minikube documentation: .

    Note

    It is important to validate that you have minikube v1.3.1 installed. Older versions of minikube are shipping a kernel configuration that is not compatible with the TPROXY requirements of Cilium >= 1.6.0.

    1. Create a minikube cluster:
    1. minikube start --network-plugin=cni --memory=4096
    1. # Only available for minikube >= v1.12.1
    2. minikube start --cni=cilium --memory=4096

    Note

    From minikube v1.12.1+, cilium networking plugin can be enabled directly with --network-plugin=cilium parameter in minikube start command. With this flag enabled, minikube will not only mount eBPF file system but also deploy quick-install.yaml automatically.

    1. Mount the eBPF filesystem
    1. minikube ssh -- sudo mount bpffs -t bpf /sys/fs/bpf

    Note

    In case of installing Cilium for a specific Kubernetes version, the --kubernetes-version vx.y.z parameter can be appended to the minikube start command for bootstrapping the local cluster. By default, minikube will install the most recent version of Kubernetes.

    Install dependencies

    1. Install docker stable as described in Install Docker Engine
    2. Install kubectl version >= v1.14.0 as described in the
    3. Install helm >= v3.0.3 per Helm documentation: Installing Helm
    4. Install kind >= v0.7.0 per kind documentation:

    Configure kind

    Configuring kind cluster creation is done using a YAML configuration file. This step is necessary in order to disable the default CNI and replace it with Cilium.

    Create a kind-config.yaml file based on the following template. It will create a cluster with 3 worker nodes and 1 control-plane node.

    1. kind: Cluster
    2. apiVersion: kind.x-k8s.io/v1alpha4
    3. nodes:
    4. - role: control-plane
    5. - role: worker
    6. - role: worker
    7. - role: worker
    8. networking:
    9.   disableDefaultCNI: true

    By default, the latest version of Kubernetes from when the kind release was created is used.

    To change the version of Kubernetes being run, image has to be defined for each node. See the documentation for more information.

    Tip

    By default, kind uses the following pod and service subnets:

    1. Networking.PodSubnet     = "10.244.0.0/16"
    2. Networking.ServiceSubnet = "10.96.0.0/12"

    If any of these subnets conflicts with your local network address range, update the networking section of the kind configuration file to specify different subnets that do not conflict or you risk having connectivity issues when deploying Cilium. For example:

    1. networking:
    2.   disableDefaultCNI: true
    3.   podSubnet: "10.10.0.0/16"
    4.   serviceSubnet: "10.11.0.0/16"

    Create a cluster

    To create a cluster with the configuration defined above, pass the kind-config.yaml you created with the --config flag of kind.

    1. kind create cluster --config=kind-config.yaml

    After a couple of seconds or minutes, a 4 nodes cluster should be created.

    1. kubectl cluster-info --context kind-kind

    Note

    The cluster nodes will remain in state NotReady until Cilium is deployed. This behavior is expected.

    Preload images

    Preload the cilium image into each worker node in the kind cluster:

    1. docker pull cilium/cilium:v1.8.10
    2. kind load docker-image cilium/cilium:v1.8.10

    Deploy Cilium and Hubble

    This section shows how to install Cilium, enable Hubble and deploy Hubble Relay and Hubble’s graphical UI.

    Single-node cluster with minikube

    Multi-node cluster with kind

    Deploy Hubble and Cilium with the provided pre-rendered YAML manifest:

    Note

    First, make sure you have Helm 3 .

    If you have (or planning to have) Helm 2 charts (and Tiller) in the same cluster, there should be no issue as both version are mutually compatible in order to support gradual migration. Cilium chart is targeting Helm 3 (v3.0.3 and above).

    Setup Helm repository:

    Deploy Hubble and Cilium with the following Helm command:

    1. helm install cilium cilium/cilium --version 1.8.10 \
    2.    --namespace kube-system \
    3.    --set global.nodeinit.enabled=true \
    4.    --set global.kubeProxyReplacement=partial \
    5.    --set global.hostServices.enabled=false \
    6.    --set global.externalIPs.enabled=true \
    8.    --set global.hostPort.enabled=true \
    9.    --set config.ipam=kubernetes \
    10.    --set global.hubble.enabled=true \
    11.    --set global.hubble.listenAddress=":4244" \
    12.    --set global.hubble.relay.enabled=true \
    13.    --set global.hubble.ui.enabled=true

    Note

    Please note that Hubble Relay and Hubble UI are currently in beta status and are not yet recommended for production use.

    You can monitor as Cilium and all required components are being installed:

    1. kubectl -n kube-system get pods --watch
    2. NAME                                         READY   STATUS              RESTARTS   AGE
    3. cilium-2rlwx                                 0/1     Init:0/2            0          2s
    4. cilium-ncqtb                                 0/1     Init:0/2            0          2s
    5. cilium-node-init-9h9dd                       0/1     ContainerCreating   0          2s
    6. cilium-node-init-cmks4                       0/1     ContainerCreating   0          2s
    7. cilium-node-init-vnx5n                       0/1     ContainerCreating   0          2s
    8. cilium-node-init-zhs66                       0/1     ContainerCreating   0          2s
    9. cilium-nrzsp                                 0/1     Init:0/2            0          2s
    10. cilium-operator-599dbcf854-7w4rr             0/1     Pending             0          2s
    11. cilium-pghbg                                 0/1     Init:0/2            0          2s
    12. coredns-66bff467f8-gnzk7                     0/1     Pending             0          6m6s
    13. coredns-66bff467f8-wzh49                     0/1     Pending             0          6m6s
    14. etcd-kind-control-plane                      1/1     Running             0          6m15s
    15. hubble-relay-5684848cc8-6ldhj                0/1     ContainerCreating   0          2s
    16. hubble-ui-54c6bc4cdc-h5drq                   0/1     Pending             0          2s
    17. kube-apiserver-kind-control-plane            1/1     Running             0          6m15s
    18. kube-controller-manager-kind-control-plane   1/1     Running             0          6m15s
    19. kube-proxy-dchqv                             1/1     Running             0          5m51s
    20. kube-proxy-jkvhr                             1/1     Running             0          5m53s
    21. kube-proxy-nb9b2                             1/1     Running             0          6m5s
    22. kube-proxy-ttf7z                             1/1     Running             0          5m50s
    23. kube-scheduler-kind-control-plane            1/1     Running             0          6m15s
    24. cilium-node-init-zhs66                       1/1     Running             0          4s

    It may take a couple of minutes for all components to come up:

    1. kubectl -n kube-system get pods
    2. NAME                                         READY   STATUS    RESTARTS   AGE
    3. cilium-2rlwx                                 1/1     Running   0          16m
    4. cilium-ncqtb                                 1/1     Running   0          16m
    5. cilium-node-init-9h9dd                       1/1     Running   1          16m
    6. cilium-node-init-cmks4                       1/1     Running   1          16m
    7. cilium-node-init-vnx5n                       1/1     Running   1          16m
    8. cilium-node-init-zhs66                       1/1     Running   1          16m
    9. cilium-nrzsp                                 1/1     Running   0          16m
    10. cilium-operator-599dbcf854-7w4rr             1/1     Running   0          16m
    11. cilium-pghbg                                 1/1     Running   0          16m
    12. coredns-66bff467f8-gnzk7                     1/1     Running   0          22m
    13. coredns-66bff467f8-wzh49                     1/1     Running   0          22m
    14. etcd-kind-control-plane                      1/1     Running   0          22m
    15. hubble-relay-5684848cc8-2z6qk                1/1     Running   0          21s
    16. hubble-ui-54c6bc4cdc-g5mgd                   1/1     Running   0          17s
    17. kube-apiserver-kind-control-plane            1/1     Running   0          22m
    18. kube-controller-manager-kind-control-plane   1/1     Running   0          22m
    19. kube-proxy-dchqv                             1/1     Running   0          21m
    20. kube-proxy-nb9b2                             1/1     Running   0          22m
    21. kube-proxy-ttf7z                             1/1     Running   0          21m
    22. kube-scheduler-kind-control-plane            1/1     Running   0          22m

    Accessing the Graphical User Interface

    Hubble provides a graphical user interface which displays a service map of your service dependencies. To access Hubble UI, you can use the following command to forward the port of the web frontend to your local machine:

    1. kubectl port-forward -n kube-system svc/hubble-ui 12000:80

    Open http://localhost:12000 in your browser. You should see a screen with an invitation to select a namespace, use the namespace selector dropdown on the left top corner to select a namespace:

    In this example, we are deploying the Star Wars demo from the guide. However you can apply the same techniques to observe application connectivity dependencies in your own namespace, and clusters for application of any type.

    Once the the deployment is ready, issue a request from both spaceships to emulate some traffic.

    1. $ kubectl exec xwing -- curl -s -XPOST deathstar.default.svc.cluster.local/v1/request-landing
    2. Ship landed
    3. $ kubectl exec tiefighter -- curl -s -XPOST deathstar.default.svc.cluster.local/v1/request-landing
    4. Ship landed

    These requests will then be displayed in the UI as service dependencies between the different pods:

    ../../_images/hubble_sw_service_map.png

    In the bottom of the interface, you may also inspect each recent Hubble flow event in your current namespace individually.

    If you enable Layer 7 Protocol Visibility on your pods, the Hubble UI service map will display the HTTP endpoints which are being accessed by the requests.

    Now let’s install the Hubble CLI on your PC/laptop. This will allow you to inspect the traffic using Hubble Relay.

    Linux

    MacOS

    Windows

    Download the latest hubble release:

    2. curl -LO "https://github.com/cilium/hubble/releases/download/$HUBBLE_VERSION/hubble-linux-amd64.tar.gz"
    3. curl -LO "https://github.com/cilium/hubble/releases/download/$HUBBLE_VERSION/hubble-linux-amd64.tar.gz.sha256sum"
    4. sha256sum --check hubble-linux-amd64.tar.gz.sha256sum
    5. tar zxf hubble-linux-amd64.tar.gz

    and move the hubble CLI to a directory listed in the $PATH environment variable. For example:

    1. sudo mv hubble /usr/local/bin

    Download the latest hubble release:

    1. export HUBBLE_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/hubble/master/stable.txt)
    2. curl -LO "https://github.com/cilium/hubble/releases/download/$HUBBLE_VERSION/hubble-darwin-amd64.tar.gz"
    3. curl -LO "https://github.com/cilium/hubble/releases/download/$HUBBLE_VERSION/hubble-darwin-amd64.tar.gz.sha256sum"
    4. shasum -a 256 -c hubble-darwin-amd64.tar.gz.sha256sum
    5. tar zxf hubble-darwin-amd64.tar.gz

    and move the hubble CLI to a directory listed in the $PATH environment variable. For example:

    Download the latest hubble release:

    1. curl -LO "https://raw.githubusercontent.com/cilium/hubble/master/stable.txt"
    2. set /p HUBBLE_VERSION=<stable.txt
    3. curl -LO "https://github.com/cilium/hubble/releases/download/%HUBBLE_VERSION%/hubble-windows-amd64.tar.gz"
    4. curl -LO "https://github.com/cilium/hubble/releases/download/%HUBBLE_VERSION%/hubble-windows-amd64.tar.gz.sha256sum"
    5. certutil -hashfile hubble-windows-amd64.tar.gz SHA256
    6. type hubble-windows-amd64.tar.gz.sha256sum
    7. :: verify that the checksum from the two commands above match
    8. tar zxf hubble-windows-amd64.tar.gz

    and move the hubble.exe CLI to a directory listed in the %PATH% environment variable after extracting it from the tarball.

    In order to access Hubble Relay with the hubble CLI, let’s make sure to port-forward the Hubble Relay service locally:

    1. $ kubectl port-forward -n kube-system svc/hubble-relay 4245:80

    Note

    This terminal window needs to be remain open to keep port-forwarding in place. Open a separate terminal window to use the hubble CLI.

    Confirm that the Hubble Relay service is healthy via hubble status:

    1. $ hubble status --server localhost:4245
    2. Healthcheck (via localhost:4245): Ok
    3. Max Flows: 16384

    In order to avoid passing --server localhost:4245 to every command, you may export the following environment variable:

    1. $ export HUBBLE_DEFAULT_SOCKET_PATH=localhost:4245

    Let’s now issue some requests to emulate some traffic again. This first request is allowed by the policy.

    1. $ kubectl exec tiefighter -- curl -s -XPOST deathstar.default.svc.cluster.local/v1/request-landing
    2. Ship landed

    This next request is accessing an HTTP endpoint which is denied by policy.

    1. $ kubectl exec tiefighter -- curl -s -XPUT deathstar.default.svc.cluster.local/v1/exhaust-port
    2. Access denied

    Finally, this last request will hang because the xwing pod does not have the org=empire label required by policy. Press Control-C to kill the curl request, or wait for it to time out.

    1. $ kubectl exec xwing -- curl -s -XPOST deathstar.default.svc.cluster.local/v1/request-landing
    2. command terminated with exit code 28

    Let’s now inspect this traffic using the CLI. The command below filters all traffic on the application layer (L7, HTTP) to the deathstar pod:

    1. $ hubble observe --pod deathstar --protocol http
    2. TIMESTAMP             SOURCE                                  DESTINATION                             TYPE            VERDICT     SUMMARY
    3. Jun 18 13:52:23.843   default/tiefighter:52568                default/deathstar-5b7489bc84-8wvng:80   http-request    FORWARDED   HTTP/1.1 POST http://deathstar.default.svc.cluster.local/v1/request-landing
    4. Jun 18 13:52:23.844   default/deathstar-5b7489bc84-8wvng:80   default/tiefighter:52568                http-response   FORWARDED   HTTP/1.1 200 0ms (POST http://deathstar.default.svc.cluster.local/v1/request-landing)
    5. Jun 18 13:52:31.019   default/tiefighter:52628                default/deathstar-5b7489bc84-8wvng:80   http-request    DROPPED     HTTP/1.1 PUT http://deathstar.default.svc.cluster.local/v1/exhaust-port

    The following command shows all traffic to the deathstar pod that has been dropped:

    1. $ hubble observe --pod deathstar --verdict DROPPED
    2. TIMESTAMP             SOURCE                     DESTINATION                             TYPE            VERDICT   SUMMARY
    3. Jun 18 13:52:31.019   default/tiefighter:52628   default/deathstar-5b7489bc84-8wvng:80   http-request    DROPPED   HTTP/1.1 PUT http://deathstar.default.svc.cluster.local/v1/exhaust-port
    4. Jun 18 13:52:38.321   default/xwing:34138        default/deathstar-5b7489bc84-v4s7d:80   Policy denied   DROPPED   TCP Flags: SYN
    5. Jun 18 13:52:38.321   default/xwing:34138        default/deathstar-5b7489bc84-v4s7d:80   Policy denied   DROPPED   TCP Flags: SYN

    Feel free to further inspect the traffic. To get help for the observe command, use hubble help observe.

    Cleanup

    Once you are done experimenting with Hubble, you can remove all traces of the cluster by running the following command:

    Single-node cluster with minikube

    Multi-node cluster with kind

    1. kind delete cluster