Troubleshoot Applications

    The first step in troubleshooting is triage. What is the problem? Is it your Pods, your Replication Controller or your Service?

    The first step in debugging a Pod is taking a look at it. Check the current state of the Pod and recent events with the following command:

    Look at the state of the containers in the pod. Are they all ? Have there been recent restarts?

    Continue debugging depending on the state of the pods.

    My pod stays pending

    If a Pod is stuck in Pending it means that it can not be scheduled onto a node. Generally this is because there are insufficient resources of one type or another that prevent scheduling. Look at the output of the kubectl describe ... command above. There should be messages from the scheduler about why it can not schedule your pod. Reasons include:

    • You don’t have enough resources: You may have exhausted the supply of CPU or Memory in your cluster, in this case you need to delete Pods, adjust resource requests, or add new nodes to your cluster. See for more information.

    • You are using hostPort: When you bind a Pod to a hostPort there are a limited number of places that pod can be scheduled. In most cases, hostPort is unnecessary, try using a Service object to expose your Pod. If you do require hostPort then you can only schedule as many Pods as there are nodes in your Kubernetes cluster.

    My pod stays waiting

    • Make sure that you have the name of the image correct.
    • Run a manual docker pull <image> on your machine to see if the image can be pulled.

    My pod is crashing or otherwise unhealthy

    Once your pod has been scheduled, the methods described in Debug Running Pods are available for debugging.

    My pod is running but not doing what I told it to do

    If your pod is not behaving as you expected, it may be that there was an error in your pod description (e.g. mypod.yaml file on your local machine), and that the error was silently ignored when you created the pod. Often a section of the pod description is nested incorrectly, or a key name is typed incorrectly, and so the key is ignored. For example, if you misspelled command as commnd then the pod will be created but will not use the command line you intended it to use.

    The first thing to do is to delete your pod and try creating it again with the --validate option. For example, run kubectl apply --validate -f mypod.yaml. If you misspelled command as commnd then will give an error like this:

    The next thing to check is whether the pod on the apiserver matches the pod you meant to create (e.g. in a yaml file on your local machine). For example, run and then manually compare the original pod description, mypod.yaml with the one you got back from apiserver, mypod-on-apiserver.yaml. There will typically be some lines on the “apiserver” version that are not on the original version. This is expected. However, if there are lines on the original that are not on the apiserver version, then this may indicate a problem with your pod spec.

    Replication controllers are fairly straightforward. They can either create Pods or they can’t. If they can’t create pods, then please refer to the instructions above to debug your pods.

    You can also use kubectl describe rc ${CONTROLLER_NAME} to introspect events related to the replication controller.

    Services provide load balancing across a set of pods. There are several common problems that can make Services not work properly. The following instructions should help debug Service problems.

    You can view this resource with:

    Make sure that the endpoints match up with the number of pods that you expect to be members of your service. For example, if your Service is for an nginx container with 3 replicas, you would expect to see three different IP addresses in the Service’s endpoints.

    My service is missing endpoints

    If you are missing endpoints, try listing pods using the labels that Service uses. Imagine that you have a Service where the labels are:

    You can use:

    to list pods that match this selector. Verify that the list matches the Pods that you expect to provide your Service. Verify that the pod’s containerPort matches up with the Service’s targetPort

    Network traffic is not forwarded

    Please see for more information.

    What’s next

    If none of the above solves your problem, follow the instructions in to make sure that your Service is running, has Endpoints, and your are actually serving; you have DNS working, iptables rules installed, and kube-proxy does not seem to be misbehaving.