COSN

    This guide describes how to configure Alluxio with Tencent COS (Cloud Object Storage) as the under storage system. Tencent Cloud Object Storage (COS) is a distributed storage service offered by Tencent Cloud for unstructured data and accessible via HTTP/HTTPS protocols. It can store massive amounts of data and features imperceptible bandwidth and capacity expansion, making it a perfect data pool for big data computation and analytics.

    Alluxio runs on multiple machines in cluster mode so its binary package needs to be deployed on the machines. You can either or download the binaries locally.

    In preparation for using COS with Alluxio, create a new bucket or use an existing bucket. You should also note the directory you want to use in that bucket, either by creating a new directory in the bucket or using an existing one. For the purposes of this guide, the COS Bucket name is called , the directory in that bucket is called COSN_DATA, and COS Bucket region is called COSN_REGION which specifies the region of your bucket.

    Alluxio unifies access to different storage systems through the feature. COSN UFS is used to access Tencent Cloud object storage and a COS location can be either mounted at the root of the Alluxio namespace or as a nested directory.

    Configure Alluxio to use COSN as its under storage system by modifying conf/alluxio-site.properties and . Specify an existing COS bucket and directory as the under storage system by modifying conf/alluxio-site.properties to include:

    1. alluxio.master.mount.table.root.ufs=cosn://COSN_ALLUXIO_BUCKET/COSN_DATA/

    Specify COS configuration information in order to access COS by modifying conf/core-site.xml to include:

    The above is the most basic configuration. For more configuration please refer to here. After these changes, Alluxio should be configured to work with COSN as its under storage system and you can try .

    Nested Mount

    A COS location can be mounted at a nested directory in the Alluxio namespace to have unified access to multiple under storage systems. The can be used for this purpose.

    1. $ ./bin/alluxio fs mount --option fs.cosn.userinfo.secretId=<COSN_SECRET_ID> \
    2.     --option fs.cosn.userinfo.secretKey=<COSN_SECRET_KEY> \
    4.     --option fs.AbstractFileSystem.cosn.impl=org.apache.hadoop.fs.CosN \
    5.     /cosn cosn://COSN_ALLUXIO_BUCKET/COSN_DATA/

    This should start an Alluxio master and an Alluxio worker. You can see the master UI at http://localhost:19999.

    Run a simple example program:

    1. $ ./bin/alluxio runTests

    Visit your COS directory at COSN_ALLUXIO_BUCKET/COSN_DATA to verify the files and directories created by Alluxio exist. For this test, you should see files named like:

    To stop Alluxio, you can run: