System Logs

    klog is the Kubernetes logging library. generates log messages for the Kubernetes system components.

    For more information about klog configuration, see the Command line tool reference.

    Kubernetes is in the process of simplifying logging in its components. The following klog command line flags starting with Kubernetes 1.23 and will be removed in a future release:

    • --alsologtostderr
    • --log-backtrace-at
    • --log-dir
    • --log-file
    • --log-file-max-size
    • --logtostderr
    • --one-output
    • --skip-headers
    • --skip-log-headers

    Output will always be written to stderr, regardless of the output format. Output redirection is expected to be handled by the component which invokes a Kubernetes component. This can be a POSIX shell or a tool like systemd.

    In some cases, for example a distroless container or a Windows system service, those options are not available. Then the kube-log-runner binary can be used as wrapper around a Kubernetes component to redirect output. A prebuilt binary is included in several Kubernetes base images under its traditional name as /go-runner and as kube-log-runner in server and node release archives.

    This table shows how kube-log-runner invocations correspond to shell redirection:

    An example of the traditional klog native format:

    The message string may contain line breaks:

    1. I1025 00:15:15.525108       1 example.go:79] This is a message

    Structured Logging

    FEATURE STATE: Kubernetes v1.23 [beta]

    Warning:

    Log formatting and value serialization are subject to change.

    Structured logging introduces a uniform structure in log messages allowing for programmatic extraction of information. You can store and process structured logs with less effort and cost. The code which generates a log message determines whether it uses the traditional unstructured klog output or structured logging.

    The default formatting of structured log messages is as text, with a format that is backward compatible with traditional klog:

    Example:

    1. I1025 00:15:15.525108       1 controller_utils.go:116] "Pod status updated" pod="kube-system/kubedns" status="ready"

    Strings are quoted. Other values are formatted with , which may cause log messages to continue on the next line depending on the data.

    FEATURE STATE: Kubernetes v1.24 [alpha]

    Contextual logging builds on top of structured logging. It is primarily about how developers use logging calls: code based on that concept is more flexible and supports additional use cases as described in the Contextual Logging KEP.

    If developers use additional functions like WithValues or WithName in their components, then log entries contain additional information that gets passed into functions by their caller.

    Currently this is gated behind the StructuredLogging feature gate and disabled by default. The infrastructure for this was added in 1.24 without modifying components. The command demonstrates how to use the new logging calls and how a component behaves that supports contextual logging.

    1. $ cd $GOPATH/src/k8s.io/kubernetes/staging/src/k8s.io/component-base/logs/example/cmd/
    2. $ go run . --help
    3. ...
    4.       --feature-gates mapStringBool  A set of key=value pairs that describe feature gates for alpha/experimental features. Options are:
    5.                                      AllAlpha=true|false (ALPHA - default=false)
    6.                                      AllBeta=true|false (BETA - default=false)
    7.                                      ContextualLogging=true|false (ALPHA - default=false)
    8. $ go run . --feature-gates ContextualLogging=true
    9. ...
    10. I0404 18:00:02.916429  451895 logger.go:94] "example/myname: runtime" foo="bar" duration="1m0s"
    11. I0404 18:00:02.916447  451895 logger.go:95] "example: another runtime" foo="bar" duration="1m0s"

    The example prefix and foo="bar" were added by the caller of the function which logs the runtime message and duration="1m0s" value, without having to modify that function.

    JSON log format

    FEATURE STATE:

    Warning:

    JSON output does not support many standard klog flags. For list of unsupported klog flags, see the .

    Not all logs are guaranteed to be written in JSON format (for example, during process start). If you intend to parse logs, make sure you can handle log lines that are not JSON as well.

    Field names and JSON serialization are subject to change.

    The --logging-format=json flag changes the format of logs from klog native format to JSON format. Example of JSON log format (pretty printed):

    1. {
    2.    "ts": 1580306777.04728,
    3.    "v": 4,
    4.    "msg": "Pod status updated",
    5.       "name": "nginx-1",
    6.       "namespace": "default"
    7.    },
    8.    "status": "ready"
    9. }

    Keys with special meaning:

    • ts - timestamp as Unix time (required, float)
    • v - verbosity (only for info and not for error messages, int)
    • err - error string (optional, string)
    • msg - message (required, string)

    List of components currently supporting JSON format:

    The -v flag controls log verbosity. Increasing the value increases the number of logged events. Decreasing the value decreases the number of logged events. Increasing verbosity settings logs increasingly less severe events. A verbosity setting of 0 logs only critical events.

    Log location

    There are two types of system components: those that run in a container and those that do not run in a container. For example:

    • The Kubernetes scheduler and kube-proxy run in a container.
    • The kubelet and container runtime do not run in containers.

    What’s next