Debugging DNS Resolution

    You need to have a Kubernetes cluster, and the kubectl command-line tool must be configured to communicate with your cluster. It is recommended to run this tutorial on a cluster with at least two nodes that are not acting as control plane hosts. If you do not already have a cluster, you can create one by using or you can use one of these Kubernetes playgrounds:

    Your cluster must be configured to use the CoreDNS addon or its precursor, kube-dns.

    Your Kubernetes server must be at or later than version v1.6. To check the version, enter .

    admin/dns/dnsutils.yaml

    Note: This example creates a pod in the default namespace. DNS name resolution for services depends on the namespace of the pod. For more information, review .

    Use that manifest to create a Pod:

    1. kubectl apply -f https://k8s.io/examples/admin/dns/dnsutils.yaml
    1. pod/dnsutils created

    …and verify its status:

    1. kubectl get pods dnsutils
    1. NAME      READY     STATUS    RESTARTS   AGE
    2. dnsutils   1/1       Running   0          <some-time>

    Once that Pod is running, you can exec nslookup in that environment. If you see something like the following, DNS is working correctly.

    1. kubectl exec -i -t dnsutils -- nslookup kubernetes.default
    1. Server:    10.0.0.10
    2. Address 1: 10.0.0.10
    4. Name:      kubernetes.default
    5. Address 1: 10.0.0.1

    If the nslookup command fails, check the following:

    Check the local DNS configuration first

    Take a look inside the resolv.conf file. (See and Known issues below for more information)

    1. kubectl exec -ti dnsutils -- cat /etc/resolv.conf

    Verify that the search path and name server are set up like the following (note that search path may vary for different cloud providers):

    1. search default.svc.cluster.local svc.cluster.local cluster.local google.internal c.gce_project_id.internal
    2. nameserver 10.0.0.10
    3. options ndots:5

    Errors such as the following indicate a problem with the CoreDNS (or kube-dns) add-on or with associated Services:

    1. kubectl exec -i -t dnsutils -- nslookup kubernetes.default

    or

    1. kubectl exec -i -t dnsutils -- nslookup kubernetes.default
    1. Server:    10.0.0.10
    2. Address 1: 10.0.0.10 kube-dns.kube-system.svc.cluster.local
    4. nslookup: can't resolve 'kubernetes.default'

    Check if the DNS pod is running

    Use the kubectl get pods command to verify that the DNS pod is running.

    1. kubectl get pods --namespace=kube-system -l k8s-app=kube-dns
    1. ...
    2. coredns-7b96bf9f76-5hsxb   1/1       Running   0           1h
    3. coredns-7b96bf9f76-mvmmt   1/1       Running   0           1h
    4. ...

    If you see that no CoreDNS Pod is running or that the Pod has failed/completed, the DNS add-on may not be deployed by default in your current environment and you will have to deploy it manually.

    Use the kubectl logs command to see logs for the DNS containers.

    For CoreDNS:

    1. kubectl logs --namespace=kube-system -l k8s-app=kube-dns

    Here is an example of a healthy CoreDNS log:

    1. .:53
    2. 2018/08/15 14:37:17 [INFO] CoreDNS-1.2.2
    3. 2018/08/15 14:37:17 [INFO] linux/amd64, go1.10.3, 2e322f6
    4. CoreDNS-1.2.2
    6. 2018/08/15 14:37:17 [INFO] plugin/reload: Running configuration MD5 = 24e6c59e83ce706f07bcc82c31b1ea1c

    See if there are any suspicious or unexpected messages in the logs.

    Is DNS service up?

    Verify that the DNS service is up by using the kubectl get service command.

    1. kubectl get svc --namespace=kube-system
    1. NAME         TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)             AGE
    2. ...
    3. kube-dns     ClusterIP   10.0.0.10      <none>        53/UDP,53/TCP        1h
    4. ...

    Note: The service name is kube-dns for both CoreDNS and kube-dns deployments.

    If you have created the Service or in the case it should be created by default but it does not appear, see debugging Services for more information.

    Are DNS endpoints exposed?

    You can verify that DNS endpoints are exposed by using the kubectl get endpoints command.

    1. kubectl get endpoints kube-dns --namespace=kube-system

    If you do not see the endpoints, see the endpoints section in the debugging Services documentation.

    For additional Kubernetes DNS examples, see the in the Kubernetes GitHub repository.

    You can verify if queries are being received by CoreDNS by adding the log plugin to the CoreDNS configuration (aka Corefile). The CoreDNS Corefile is held in a named coredns. To edit it, use the command:

    1. kubectl -n kube-system edit configmap coredns

    Then add log in the Corefile section per the example below:

    1. apiVersion: v1
    2. kind: ConfigMap
    3. metadata:
    4.   name: coredns
    5.   namespace: kube-system
    6. data:
    7.   Corefile: |
    8.     .:53 {
    9.         log
    10.         errors
    11.         health
    12.         kubernetes cluster.local in-addr.arpa ip6.arpa {
    13.           pods insecure
    14.           upstream
    15.           fallthrough in-addr.arpa ip6.arpa
    16.         }
    17.         prometheus :9153
    18.         forward . /etc/resolv.conf
    19.         loop
    20.         reload
    21.         loadbalance
    22.     }

    After saving the changes, it may take up to minute or two for Kubernetes to propagate these changes to the CoreDNS pods.

    Here is an example of a query in the log:

    1. .:53
    2. 2018/08/15 14:37:15 [INFO] CoreDNS-1.2.0
    4. CoreDNS-1.2.0
    5. linux/amd64, go1.10.3, 2e322f6
    6. 2018/09/07 15:29:04 [INFO] plugin/reload: Running configuration MD5 = 162475cdf272d8aa601e6fe67a6ad42f
    7. 2018/09/07 15:29:04 [INFO] Reloading complete
    8. 172.17.0.18:41675 - [07/Sep/2018:15:29:11 +0000] 59925 "A IN kubernetes.default.svc.cluster.local. udp 54 false 512" NOERROR qr,aa,rd,ra 106 0.000066649s

    Does CoreDNS have sufficient permissions?

    CoreDNS must be able to list and endpoint related resources to properly resolve service names.

    Sample error message:

    1. 2022-03-18T07:12:15.699431183Z [INFO] 10.96.144.227:52299 - 3686 "A IN serverproxy.contoso.net.cluster.local. udp 52 false 512" SERVFAIL qr,aa,rd 145 0.000091221s

    First, get the current ClusterRole of system:coredns:

    1. kubectl describe clusterrole system:coredns -n kube-system

    Expected output:

    1. PolicyRule:
    2.   Resources                        Non-Resource URLs  Resource Names  Verbs
    3.   ---------                        -----------------  --------------  -----
    4.   nodes                            []                 []              [get]
    5.   endpoints                        []                 []              [list watch]
    6.   namespaces                       []                 []              [list watch]
    7.   pods                             []                 []              [list watch]
    8.   services                         []                 []              [list watch]
    9.   endpointslices.discovery.k8s.io  []                 []              [list watch]

    If any permissions are missing, edit the ClusterRole to add them:

    1. kubectl edit clusterrole system:coredns -n kube-system

    Example insertion of EndpointSlices permissions:

    1. ...
    2. - apiGroups:
    3.   - discovery.k8s.io
    4.   resources:
    5.   - endpointslices
    6.   verbs:
    7.   - list
    8.   - watch
    9. ...

    Are you in the right namespace for the service?

    DNS queries that don’t specify a namespace are limited to the pod’s namespace.

    If the namespace of the pod and service differ, the DNS query must include the namespace of the service.

    This query is limited to the pod’s namespace:

    1. kubectl exec -i -t dnsutils -- nslookup <service-name>

    This query specifies the namespace:

    To learn more about name resolution, see DNS for Services and Pods.

    Some Linux distributions (e.g. Ubuntu) use a local DNS resolver by default (systemd-resolved). Systemd-resolved moves and replaces /etc/resolv.conf with a stub file that can cause a fatal forwarding loop when resolving names in upstream servers. This can be fixed manually by using kubelet’s --resolv-conf flag to point to the correct resolv.conf (With systemd-resolved, this is /run/systemd/resolve/resolv.conf). kubeadm automatically detects systemd-resolved, and adjusts the kubelet flags accordingly.

    Kubernetes installs do not configure the nodes’ resolv.conf files to use the cluster DNS by default, because that process is inherently distribution-specific. This should probably be implemented eventually.

    If you are using Alpine version 3.3 or earlier as your base image, DNS may not work properly due to a known issue with Alpine. Kubernetes issue 30215 details more information on this.