Azure Blob Storage

Flink supports accessing Azure Blob Storage using both or abfs://.

See below for how to use Azure Blob Storage in a Flink job:

env.readTextFile("wasb://<your-container>@$<your-azure-account>.blob.core.windows.net/<object-path>");
// Write to Azure Blob storage
env.getCheckpointConfig().setCheckpointStorage("wasb://<your-container>@$<your-azure-account>.blob.core.windows.net/<object-path>");

To use flink-azure-fs-hadoop, copy the respective JAR file from the opt directory to the plugins directory of your Flink distribution before starting Flink, e.g.

flink-azure-fs-hadoop registers default FileSystem wrappers for URIs with the wasb:// and wasbs:// (SSL encrypted access) scheme.

fs.azure.account.key.<account_name>.blob.core.windows.net: <azure_storage_key>

Alternatively, the filesystem can be configured to read the Azure Blob Storage key from an environment variable AZURE_STORAGE_KEY by setting the following configuration keys in flink-conf.yaml.

Hadoop’s ABFS Azure Filesystem supports several ways of configuring authentication. Please visit the documentation on how to configure.

Azure recommends using Azure managed identity to access the ADLS Gen2 storage accounts using abfs. Please refer to Azure managed identities documentation for more details.

Accessing ABFS using storage Keys (Discouraged)

Azure blob storage key can be configured in flink-conf.yaml via: