Debugging DNS Resolution

    You need to have a Kubernetes cluster, and the kubectl command-line tool must be configured to communicate with your cluster. It is recommended to run this tutorial on a cluster with at least two nodes that are not acting as control plane hosts. If you do not already have a cluster, you can create one by using minikube or you can use one of these Kubernetes playgrounds:

    Your cluster must be configured to use the CoreDNS or its precursor, kube-dns.

    Your Kubernetes server must be at or later than version v1.6. To check the version, enter .

    admin/dns/dnsutils.yaml

    Note: This example creates a pod in the default namespace. DNS name resolution for services depends on the namespace of the pod. For more information, review .

    Use that manifest to create a Pod:

    1. kubectl apply -f https://k8s.io/examples/admin/dns/dnsutils.yaml
    1. pod/dnsutils created

    …and verify its status:

    1. kubectl get pods dnsutils
    1. NAME      READY     STATUS    RESTARTS   AGE
    2. dnsutils   1/1       Running   0          <some-time>

    Once that Pod is running, you can exec nslookup in that environment. If you see something like the following, DNS is working correctly.

    1. kubectl exec -i -t dnsutils -- nslookup kubernetes.default
    1. Server:    10.0.0.10
    2. Address 1: 10.0.0.10
    4. Name:      kubernetes.default
    5. Address 1: 10.0.0.1

    If the nslookup command fails, check the following:

    Check the local DNS configuration first

    Take a look inside the resolv.conf file. (See and Known issues below for more information)

    1. kubectl exec -ti dnsutils -- cat /etc/resolv.conf

    Verify that the search path and name server are set up like the following (note that search path may vary for different cloud providers):

    Errors such as the following indicate a problem with the CoreDNS (or kube-dns) add-on or with associated Services:

    1. kubectl exec -i -t dnsutils -- nslookup kubernetes.default
    1. Server:    10.0.0.10
    3. nslookup: can't resolve 'kubernetes.default'

    or

    1. kubectl exec -i -t dnsutils -- nslookup kubernetes.default
    1. Server:    10.0.0.10
    2. Address 1: 10.0.0.10 kube-dns.kube-system.svc.cluster.local
    4. nslookup: can't resolve 'kubernetes.default'
    1. kubectl get pods --namespace=kube-system -l k8s-app=kube-dns
    1. NAME                       READY     STATUS    RESTARTS   AGE
    2. ...
    3. coredns-7b96bf9f76-5hsxb   1/1       Running   0           1h
    4. coredns-7b96bf9f76-mvmmt   1/1       Running   0           1h
    5. ...

    Note: The value for label k8s-app is for both CoreDNS and kube-dns deployments.

    If you see that no CoreDNS Pod is running or that the Pod has failed/completed, the DNS add-on may not be deployed by default in your current environment and you will have to deploy it manually.

    Check for errors in the DNS pod

    Use the kubectl logs command to see logs for the DNS containers.

    For CoreDNS:

    1. kubectl logs --namespace=kube-system -l k8s-app=kube-dns

    Here is an example of a healthy CoreDNS log:

    See if there are any suspicious or unexpected messages in the logs.

    Verify that the DNS service is up by using the kubectl get service command.

    1. kubectl get svc --namespace=kube-system
    1. NAME         TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)             AGE
    2. ...
    3. kube-dns     ClusterIP   10.0.0.10      <none>        53/UDP,53/TCP        1h
    4. ...

    Note: The service name is kube-dns for both CoreDNS and kube-dns deployments.

    If you have created the Service or in the case it should be created by default but it does not appear, see debugging Services for more information.

    Are DNS endpoints exposed?

    You can verify that DNS endpoints are exposed by using the kubectl get endpoints command.

    1. kubectl get endpoints kube-dns --namespace=kube-system
    1. NAME       ENDPOINTS                       AGE
    2. kube-dns   10.180.3.17:53,10.180.3.17:53    1h

    If you do not see the endpoints, see the endpoints section in the debugging Services documentation.

    For additional Kubernetes DNS examples, see the in the Kubernetes GitHub repository.

    You can verify if queries are being received by CoreDNS by adding the log plugin to the CoreDNS configuration (aka Corefile). The CoreDNS Corefile is held in a ConfigMap named coredns. To edit it, use the command:

    1. kubectl -n kube-system edit configmap coredns
    1. apiVersion: v1
    2. kind: ConfigMap
    3. metadata:
    4.   name: coredns
    5. data:
    6.   Corefile: |
    7.     .:53 {
    8.         log
    9.         errors
    10.         health
    11.         kubernetes cluster.local in-addr.arpa ip6.arpa {
    12.           pods insecure
    13.           upstream
    14.           fallthrough in-addr.arpa ip6.arpa
    15.         }
    17.         forward . /etc/resolv.conf
    18.         cache 30
    19.         loop
    20.         reload
    21.         loadbalance
    22.     }

    After saving the changes, it may take up to minute or two for Kubernetes to propagate these changes to the CoreDNS pods.

    Next, make some queries and view the logs per the sections above in this document. If CoreDNS pods are receiving the queries, you should see them in the logs.

    Here is an example of a query in the log:

    1. .:53
    2. 2018/08/15 14:37:15 [INFO] CoreDNS-1.2.0
    3. 2018/08/15 14:37:15 [INFO] linux/amd64, go1.10.3, 2e322f6
    4. CoreDNS-1.2.0
    5. linux/amd64, go1.10.3, 2e322f6
    6. 2018/09/07 15:29:04 [INFO] plugin/reload: Running configuration MD5 = 162475cdf272d8aa601e6fe67a6ad42f
    7. 2018/09/07 15:29:04 [INFO] Reloading complete
    8. 172.17.0.18:41675 - [07/Sep/2018:15:29:11 +0000] 59925 "A IN kubernetes.default.svc.cluster.local. udp 54 false 512" NOERROR qr,aa,rd,ra 106 0.000066649s

    Are you in the right namespace for the service?

    DNS queries that don’t specify a namespace are limited to the pod’s namespace.

    If the namespace of the pod and service differ, the DNS query must include the namespace of the service.

    This query is limited to the pod’s namespace:

    This query specifies the namespace:

    1. kubectl exec -i -t dnsutils -- nslookup <service-name>.<namespace>

    To learn more about name resolution, see DNS for Services and Pods.

    Some Linux distributions (e.g. Ubuntu) use a local DNS resolver by default (systemd-resolved). Systemd-resolved moves and replaces /etc/resolv.conf with a stub file that can cause a fatal forwarding loop when resolving names in upstream servers. This can be fixed manually by using kubelet’s --resolv-conf flag to point to the correct resolv.conf (With systemd-resolved, this is /run/systemd/resolve/resolv.conf). kubeadm automatically detects systemd-resolved, and adjusts the kubelet flags accordingly.

    Kubernetes installs do not configure the nodes’ resolv.conf files to use the cluster DNS by default, because that process is inherently distribution-specific. This should probably be implemented eventually.

    Linux’s libc (a.k.a. glibc) has a limit for the DNS nameserver records to 3 by default. What’s more, for the glibc versions which are older than glibc-2.17-222 (), the allowed number of DNS search records has been limited to 6 (see this bug from 2005). Kubernetes needs to consume 1 nameserver record and 3 search records. This means that if a local installation already uses 3 nameservers or uses more than 3 searches while your glibc version is in the affected list, some of those settings will be lost. To work around the DNS nameserver records limit, the node can run dnsmasq, which will provide more nameserver entries. You can also use kubelet’s --resolv-conf flag. To fix the DNS search records limit, consider upgrading your linux distribution or upgrading to an unaffected version of glibc.

    Note: With , Kubernetes allows more DNS records.

    If you are using Alpine version 3.3 or earlier as your base image, DNS may not work properly due to a known issue with Alpine. Kubernetes issue 30215 details more information on this.

    • Read