Under File Storage Extension API

This page is intended for developers of under storage extensions. Please look at managing extensions for a guide to using existing extensions.

Under storage extensions provide a framework to enable additional storage systems to work with Alluxio and makes it convenient to develop modules not already supported by Alluxio. Extensions are built as JARs and included at a specific extensions location to be picked up by core Alluxio. This page describes the mechanics of how extensions in Alluxio work, and provides detailed instructions for developing an under storage extension.

If the modules included in core Alluxio do not use the interface supported by your desired storage system, you may choose to implement an under storage extension.

Building a new under storage connector involves:

• Implementing the required under storage interface
• Declaring the service implementation
• Bundling up the implementation and transitive dependencies in an uber JAR

A reference implementation can be found in the repository. In the rest of this section, we describe the steps involved in writing a new under storage extension. The sample project, called , uses maven as the build and dependency management tool, and forwards all operations to a local filesystem.

The HDFS Submodule and are good examples of how to enable a storage system to serve as Alluxio’s underlying storage.

The UnderFileSystem interface is defined in the module org.alluxio:alluxio-core-common. Choose to extend either ConsistentUnderFileSystem or ObjectUnderFileSystem to implement the UnderFileSystem interface.

• ObjectUnderFileSystem: suitable for connecting to object storage and abstracts away mapping file system operations to an object store.

or,

public class DummyUnderFileSystem extends ObjectUnderFileSystem {
    // Implement object store operations
    ...
}

Step 2: Implement the interface UnderFileSystemFactory

The under storage factory determines defines which paths the UnderFileSystem implementation supports and how to create the UnderFileSystem implementation.

    ...
        @Override
        public UnderFileSystem create(String path, UnderFileSystemConfiguration conf) {
                // Create the under storage instance
        }
        @Override
        // Choose which schemes to support, e.g., dummy://
    }
}

Step 3: Define any properties required to configure the UnderFileSystem.

Declare the Service

Create a file at src/main/resources/META-INF/services/alluxio.underfs.UnderFileSystemFactory advertising the implemented UnderFileSystemFactory to the ServiceLoader.

In addition, to avoid collisions, specify scope for the dependency alluxio-core-common as provided. The maven definition would look like:

<dependencies>
    <!-- Core Alluxio dependencies -->
    <dependency>
      <groupId>org.alluxio</groupId>
      <artifactId>alluxio-core-common</artifactId>
      <scope>provided</scope>
    </dependency>
    ...
</dependencies>

Build the tarball:

Install to Alluxio

Install the tarball to Alluxio:

$ ./bin/alluxio extensions install <path>/<to>/<probject>/target/alluxio-underfs-<ufsName>-<version>.jar

To ensure the new under storage module fulfills the minimum requirements to work with Alluxio, one can run contract tests to test different workflows with various combinations of operations against the under storage.

In addition, one can also mount the under storage and run other kinds of tests on it. Please refer to managing extensions.

Service Discovery

Extension JARs are loaded dynamically at runtime by Alluxio servers, which enables Alluxio to talk to new under storage systems without requiring a restart. Alluxio servers use Java ServiceLoader to discover implementations of the under storage API. Providers include implementations of the interface. The implementation is advertised by including a text file in META_INF/services with a single line pointing to the class implementing the said interface.

Congratulations! You have developed a new under storage extension to Alluxio. Let the community know by submitting a pull request to the Alluxio to edit the list of extensions section on the documentation page.