Debugging Virtual Machines

    Troubleshooting an Istio Virtual Machine installation is similar to troubleshooting issues with proxies running inside Kubernetes, but there are some key differences to be aware of.

    While much of the same information is available on both platforms, accessing this information differs.

    The Istio sidecar is typically run as a unit. To ensure its running properly, you can check that status:

    Additionally, the sidecar health can be programmatically check at its health endpoint:

    1. $ curl localhost:15021/healthz/ready -I

    Logs

    To access the systemd logs, which has details about the initialization of the proxy:

      The proxy will redirect stderr and stdout to /var/log/istio/istio.err.log and /var/log/istio/istio.log, respectively. To view these in a format similar to kubectl:

      Log levels can be modified by changing the cluster.env configuration file. Make sure to restart istio if it is already running:

      2. $ systemctl restart istio

      To ensure iptables rules have been successfully applied:

      1. $ sudo iptables-save
      2. ...
      3. -A ISTIO_OUTPUT -d 127.0.0.1/32 -j RETURN
      4. -A ISTIO_OUTPUT -j ISTIO_REDIRECT

      Istioctl

      However, istioctl proxy-config relies on functionality in Kubernetes to connect to a proxy, which will not work for virtual machines. Instead, a file containing the configuration dump from Envoy can be passed. For example:

      1. $ curl -s localhost:15000/config_dump | istioctl proxy-config clusters --file -
      2. istiod.istio-system.svc.cluster.local   443       -       outbound      EDS
      3. istiod.istio-system.svc.cluster.local   15010     -       outbound      EDS
      4. istiod.istio-system.svc.cluster.local   15012     -       outbound      EDS
      5. istiod.istio-system.svc.cluster.local   15014     -       outbound      EDS

      When a virtual machine connects to Istiod, a WorkloadEntry will automatically be created. This enables the virtual machine to become a part of a Service, similar to an Endpoint in Kubernetes.

      To check these are created correctly:

      1. $ kubectl get workloadentries
      2. NAME             AGE   ADDRESS
      3. vm-10.128.0.50   14m   10.128.0.50

      Certificates

      Virtual machines handle certificates differently than Kubernetes Pods, which use a Kubernetes-provided service account token to authenticate and renew mTLS certificates. Instead, existing mTLS credentials are used to authenticate with the certificate authority and renew certificates.

      Additionally, these are persisted to disk to ensure downtime or restarts do not lose state.

      1. cert-chain.pem  key.pem  root-cert.pem