How to use logging

    The log files can be accessed via the Job-/TaskManager pages of the WebUI. The used Resource Provider (e.g., YARN) may provide additional means of accessing them.

    The logging in Flink uses the logging interface. This allows you to use any logging framework that supports SLF4J, without having to modify the Flink source code.

    By default, Log4j 2 is used as the underlying logging framework.

    Log4j 2 is controlled using property files.

    The Flink distribution ships with the following log4j properties files in the directory, which are used automatically if Log4j 2 is enabled:

    • log4j-cli.properties: used by the command line interface (e.g., flink run)
    • log4j-console.properties: used for Job-/TaskManagers if they are run in the foreground (e.g., Kubernetes)
    • log4j.properties: used for Job-/TaskManagers by default

    Log4j periodically scans this file for changes and adjusts the logging behavior if necessary. By default this check happens every 30 seconds and is controlled by the monitorInterval setting in the Log4j properties files.

    If you have custom Log4j 1 properties files or code that relies on Log4j 1, please check out the official Log4j and migration guides.

    To use Flink with you must ensure that:

    • org.apache.logging.log4j:log4j-core, org.apache.logging.log4j:log4j-slf4j-impl and org.apache.logging.log4j:log4j-1.2-api are not on the classpath,
    • log4j:log4j, org.slf4j:slf4j-log4j12, and org.apache.logging.log4j:log4j-api are on the classpath.

    In the IDE this means you have to replace such dependencies defined in your pom, and possibly add exclusions on dependencies that transitively depend on them.

    For Flink distributions this means you have to

    • remove the log4j-core, log4j-slf4j-impl and log4j-1.2-api jars from the lib directory,
    • add the log4j, slf4j-log4j12 and log4j-to-slf4j jars to the lib directory,

    To use Flink with logback you must ensure that:

    • org.apache.logging.log4j:log4j-slf4j-impl is not on the classpath,
    • ch.qos.logback:logback-core and ch.qos.logback:logback-classic are on the classpath.

    In the IDE this means you have to replace such dependencies defined in your pom, and possibly add exclusions on dependencies that transitively depend on them.

    • remove the jar from the lib directory,
    • add the logback-core, and logback-classic jars to the lib directory.

    The Flink distribution ships with the following logback configuration files in the conf directory, which are used automatically if logback is enabled:

    • logback-session.properties: used by the command line interface when starting a Kubernetes/Yarn session cluster (i.e., kubernetes-session.sh/yarn-session.sh)
    • logback-console.properties: used for Job-/TaskManagers if they are run in the foreground (e.g., Kubernetes)

    You can create an SLF4J logger by calling org.slf4j.LoggerFactory#LoggerFactory.getLogger with the Class of your class as an argument.

    We highly recommend storing this logger in a private static final field.

    In order to benefit most from SLF4J, it is recommended to use its placeholder mechanism. Using placeholders allows avoiding unnecessary string constructions in case that the logging level is set so high that the message would not be logged.

    The syntax of placeholders is the following: