Debugging

You can log contextual and debug information via or standard Python logging modules in PyFlink jobs in places outside Python UDFs. The logging messages will be printed in the log files of the client during job submission.

Note: The default logging level at client side is WARNING and so only messages with logging level WARNING or above will appear in the log files of the client.

You can log contextual and debug information via print or standard Python logging modules in Python UDFs. The logging messages will be printed in the log files of the TaskManagers during job execution.

@udf(result_type=DataTypes.BIGINT())
def add(i, j):
    import logging
    logging.info("debug")
    # use print function
    print('debug')

Note: The default logging level at server side is INFO and so only messages with logging level or above will appear in the log files of the TaskManagers.

You can debug your python functions directly in IDEs such as PyCharm.

You can make use of the pydevd_pycharm tool of PyCharm to debug Python UDFs.

1. Create a Python Remote Debug in PyCharm

   run -> Python Remote Debug -> + -> choose a port (e.g. 6789)

2. $ pip install pydevd-pycharm

3. Add the following command in your Python UDF

4. Start the previously created Python Remote Debug Server

5. Run your Python Code

You can enable the profile to analyze performance bottlenecks.

t_env.get_config().get_configuration().set_boolean("python.profile.enabled", True)