S3 API

    Alluxio supports a RESTful API that is compatible with the basic operations of the Amazon .

    The Alluxio S3 API should be used by applications designed to communicate with an S3-like storage and would benefit from the other features provided by Alluxio, such as data caching, data sharing with file system based applications, and storage system abstraction (e.g., using Ceph instead of S3 as the backing store). For example, a simple application that downloads reports generated by analytic tasks can use the S3 API instead of the more complex file system API.

    There are performance implications of using the S3 API. The S3 API leverages the Alluxio proxy, introducing an extra hop. For optimal performance, it is recommended to run the proxy server and an Alluxio worker on each compute node. It is also recommended to put all the proxy servers behind a load balancer.

    The following table describes the support status for current Amazon S3 functional features:

    Alluxio S3 client supports various programming languages, such as C++, Java, Python, Golang, and Ruby. In this documentation, we use curl REST calls and python S3 client as usage examples.

    Create a bucket

    Get the bucket (listing objects)

    3. HTTP/1.1 200 OK
    4. Date: Tue, 18 Jun 2019 21:23:56 GMT
    5. Content-Type: application/xml
    6. Content-Length: 191
    7. Server: Jetty(9.2.z-SNAPSHOT)
    9. <ListBucketResult><Name>/testbucket</Name><Prefix/><ContinuationToken/><NextContinuationToken/><KeyCount>0</KeyCount><MaxKeys>1000</MaxKeys><IsTruncated>false</IsTruncated></ListBucketResult>

    Put an object

    Assuming there is an existing file on local file system called LICENSE:

    1. $ curl -i -X PUT -T "LICENSE" http://localhost:39999/api/v1/s3/testbucket/testobject
    3. HTTP/1.1 100 Continue
    5. HTTP/1.1 200 OK
    6. Date: Tue, 18 Jun 2019 21:24:32 GMT
    7. ETag: "911df44b7ff57801ca8d74568e4ebfbe"
    8. Content-Length: 0
    9. Server: Jetty(9.2.z-SNAPSHOT)

    Get the object:

    1. $ curl -i -X GET http://localhost:39999/api/v1/s3/testbucket/testobject
    3. HTTP/1.1 200 OK
    4. Date: Tue, 18 Jun 2019 21:24:57 GMT
    5. Last-Modified: Tue, 18 Jun 2019 21:24:33 GMT
    6. Content-Type: application/xml
    7. Content-Length: 27040
    8. Server: Jetty(9.2.z-SNAPSHOT)
    10. .................. Content of the test file ...................

    Listing a bucket with one object

    1. $ curl -i -X GET http://localhost:39999/api/v1/s3/testbucket
    3. HTTP/1.1 200 OK
    4. Date: Tue, 18 Jun 2019 21:25:27 GMT
    5. Content-Type: application/xml
    6. Content-Length: 354
    7. Server: Jetty(9.2.z-SNAPSHOT)
    9. <ListBucketResult><Name>/testbucket</Name><Prefix/><ContinuationToken/><NextContinuationToken/><KeyCount>1</KeyCount><MaxKeys>1000</MaxKeys><IsTruncated>false</IsTruncated><Contents><Key>testobject</Key><LastModified>2019-06-18T14:24:33.029Z</LastModified><ETag></ETag><Size>27040</Size><StorageClass>STANDARD</StorageClass></Contents></ListBucketResult>

    Listing a bucket with multiple objects

    You can upload more files and use the max-keys and continuation-token as the GET bucket request parameter. For example:

    1. $ curl -i -X PUT -T "LICENSE" http://localhost:39999/api/v1/s3/testbucket/key1
    3. HTTP/1.1 100 Continue
    5. HTTP/1.1 200 OK
    6. Date: Tue, 18 Jun 2019 21:26:05 GMT
    7. ETag: "911df44b7ff57801ca8d74568e4ebfbe"
    8. Server: Jetty(9.2.z-SNAPSHOT)
    10. # curl -i -X PUT -T "LICENSE" http://localhost:39999/api/v1/s3/testbucket/key2
    12. HTTP/1.1 100 Continue
    14. HTTP/1.1 200 OK
    15. Date: Tue, 18 Jun 2019 21:26:28 GMT
    16. ETag: "911df44b7ff57801ca8d74568e4ebfbe"
    17. Content-Length: 0
    18. Server: Jetty(9.2.z-SNAPSHOT)
    20. # curl -i -X PUT -T "LICENSE" http://localhost:39999/api/v1/s3/testbucket/key3
    22. HTTP/1.1 100 Continue
    24. HTTP/1.1 200 OK
    25. Date: Tue, 18 Jun 2019 21:26:43 GMT
    26. ETag: "911df44b7ff57801ca8d74568e4ebfbe"
    28. Server: Jetty(9.2.z-SNAPSHOT)
    30. # curl -i -X GET http://localhost:39999/api/v1/s3/testbucket\?max-keys\=2
    32. HTTP/1.1 200 OK
    33. Date: Tue, 18 Jun 2019 21:26:57 GMT
    34. Content-Type: application/xml
    35. Content-Length: 528
    36. Server: Jetty(9.2.z-SNAPSHOT)
    38. <ListBucketResult><Name>/testbucket</Name><Prefix/><ContinuationToken/><NextContinuationToken>key3</NextContinuationToken><KeyCount>2</KeyCount><MaxKeys>2</MaxKeys><IsTruncated>true</IsTruncated><Contents><Key>key1</Key><LastModified>2019-06-18T14:26:05.694Z</LastModified><ETag></ETag><Size>27040</Size><StorageClass>STANDARD</StorageClass></Contents><Contents><Key>key2</Key><LastModified>2019-06-18T14:26:28.153Z</LastModified><ETag></ETag><Size>27040</Size><StorageClass>STANDARD</StorageClass></Contents></ListBucketResult>
    40. # curl -i -X GET http://localhost:39999/api/v1/s3/testbucket\?max-keys\=2\&continuation-token\=key3
    42. HTTP/1.1 200 OK
    43. Date: Tue, 18 Jun 2019 21:28:14 GMT
    44. Content-Type: application/xml
    45. Content-Length: 531
    46. Server: Jetty(9.2.z-SNAPSHOT)
    48. <ListBucketResult><Name>/testbucket</Name><Prefix/><ContinuationToken>key3</ContinuationToken><NextContinuationToken/><KeyCount>2</KeyCount><MaxKeys>2</MaxKeys><IsTruncated>false</IsTruncated><Contents><Key>key3</Key><LastModified>2019-06-18T14:26:43.081Z</LastModified><ETag></ETag><Size>27040</Size><StorageClass>STANDARD</StorageClass></Contents><Contents><Key>testobject</Key><LastModified>2019-06-18T14:24:33.029Z</LastModified><ETag></ETag><Size>27040</Size><StorageClass>STANDARD</StorageClass></Contents></ListBucketResult>

    You can also verify those objects are represented as Alluxio files, under /testbucket directory.

    1. $ ./bin/alluxio fs ls -R /testbucket
    3. -rw-r--r--  alluxio        staff                    27040       PERSISTED 06-18-2019 14:26:05:694 100% /testbucket/key1
    4. -rw-r--r--  alluxio        staff                    27040       PERSISTED 06-18-2019 14:26:28:153 100% /testbucket/key2
    5. -rw-r--r--  alluxio        staff                    27040       PERSISTED 06-18-2019 14:26:43:081 100% /testbucket/key3
    6. -rw-r--r--  alluxio        staff                    27040       PERSISTED 06-18-2019 14:24:33:029 100% /testbucket/testobject

    Delete objects

    1. $ curl -i -X DELETE http://localhost:39999/api/v1/s3/testbucket/key1
    3. HTTP/1.1 204 No Content
    4. Date: Tue, 18 Jun 2019 21:31:27 GMT
    5. Server: Jetty(9.2.z-SNAPSHOT)
    1. $ curl -i -X DELETE http://localhost:39999/api/v1/s3/testbucket/key2
    3. HTTP/1.1 204 No Content
    4. Date: Tue, 18 Jun 2019 21:31:44 GMT
    5. Server: Jetty(9.2.z-SNAPSHOT)
    7. # curl -i -X DELETE http://localhost:39999/api/v1/s3/testbucket/key3
    9. HTTP/1.1 204 No Content
    10. Date: Tue, 18 Jun 2019 21:31:58 GMT
    11. Server: Jetty(9.2.z-SNAPSHOT)
    13. # curl -i -X DELETE http://localhost:39999/api/v1/s3/testbucket/testobject
    15. HTTP/1.1 204 No Content
    16. Date: Tue, 18 Jun 2019 21:32:08 GMT
    17. Server: Jetty(9.2.z-SNAPSHOT)

    Initiate a multipart upload

    Since we deleted the testobject in the previous command, you have to create another testobject before initiating a multipart upload.

    Note that the commands below related to multipart upload need the upload ID shown above, it’s not necessarily 3.

    Upload part

    1. $ curl -i -X PUT 'http://localhost:39999/api/v1/s3/testbucket/testobject?partNumber=1&uploadId=3'
    3. HTTP/1.1 200 OK
    4. Date: Tue, 18 Jun 2019 21:33:36 GMT
    5. ETag: "d41d8cd98f00b204e9800998ecf8427e"
    6. Server: Jetty(9.2.z-SNAPSHOT)

    List parts

    1. $ curl -i -X GET http://localhost:39999/api/v1/s3/testbucket/testobject?uploadId=3
    3. HTTP/1.1 200 OK
    4. Date: Tue, 18 Jun 2019 21:35:10 GMT
    5. Content-Type: application/xml
    6. Content-Length: 296
    7. Server: Jetty(9.2.z-SNAPSHOT)
    9. <ListPartsResult><Bucket>/testbucket</Bucket><Key>testobject</Key><UploadId>3</UploadId><StorageClass>STANDARD</StorageClass><IsTruncated>false</IsTruncated><Part><PartNumber>1</PartNumber><LastModified>2019-06-18T14:33:36.373Z</LastModified><ETag>""</ETag><Size>0</Size></Part></ListPartsResult>

    Complete a multipart upload

    1. $ curl -i -X POST http://localhost:39999/api/v1/s3/testbucket/testobject?uploadId=3
    3. HTTP/1.1 200 OK
    4. Date: Tue, 18 Jun 2019 21:35:47 GMT
    5. Content-Type: application/xml
    6. Content-Length: 201
    7. Server: Jetty(9.2.z-SNAPSHOT)
    9. <CompleteMultipartUploadResult><Location>/testbucket/testobject</Location><Bucket>testbucket</Bucket><Key>testobject</Key><ETag>"d41d8cd98f00b204e9800998ecf8427e"</ETag></CompleteMultipartUploadResult>

    Abort a multipart upload

    1. $ curl -i -X DELETE http://localhost:39999/api/v1/s3/testbucket/testobject?uploadId=3
    3. HTTP/1.1 204 No Content
    4. Date: Tue, 18 Jun 2019 21:37:27 GMT
    5. Server: Jetty(9.2.z-SNAPSHOT)

    Delete an empty bucket

    1. $ curl -i -X DELETE http://localhost:39999/api/v1/s3/testbucket
    3. HTTP/1.1 204 No Content
    4. Date: Tue, 18 Jun 2019 21:38:38 GMT
    5. Server: Jetty(9.2.z-SNAPSHOT)

    Python S3 Client

    Tested for Python 2.7.

    Create a connection:

    Please note you have to install boto package first.

    1. pip install boto
    1. import boto
    2. import boto.s3.connection
    4. conn = boto.connect_s3(
    5.     aws_access_key_id = '',
    6.     aws_secret_access_key = '',
    7.     host = 'localhost',
    8.     port = 39999,
    9.     path = '/api/v1/s3',
    10.     is_secure=False,
    11.     calling_format = boto.s3.connection.OrdinaryCallingFormat(),
    12. )

    Create a bucket

    1. bucketName = 'bucket-for-testing'
    2. bucket = conn.create_bucket(bucketName)

    PUT a small object

    Get the small object

    1. assert smallObjectContent == key.get_contents_as_string()

    Upload a large object

    Create a 8MB file on local file system.

    1. $ dd if=/dev/zero of=8mb.data bs=1048576 count=8

    Then use python S3 client to upload this as an object

    1. largeObjectKey = 'large.txt'
    2. largeObjectFile = '8mb.data'
    4. key = bucket.new_key(largeObjectKey)
    5. with open(largeObjectFile, 'rb') as f:
    6.     key.set_contents_from_file(f)
    7. with open(largeObjectFile, 'rb') as f:
    8.     largeObject = f.read()

    Get the large object

    1. assert largeObject == key.get_contents_as_string()

    Delete the objects

    1. bucket.delete_key(smallObjectKey)
    2. bucket.delete_key(largeObjectKey)

    Initiate a multipart upload

    1. mp = bucket.initiate_multipart_upload(largeObjectKey)

    Upload parts

    1. import math, os
    3. from filechunkio import FileChunkIO
    5. # Use a chunk size of 1MB (feel free to change this)
    6. sourceSize = os.stat(largeObjectFile).st_size
    7. chunkSize = 1048576
    8. chunkCount = int(math.ceil(sourceSize / float(chunkSize)))
    10. for i in range(chunkCount):
    11.     offset = chunkSize * i
    12.     bytes = min(chunkSize, sourceSize - offset)
    13.     with FileChunkIO(largeObjectFile, 'r', offset=offset, bytes=bytes) as fp:
    14.         mp.upload_part_from_file(fp, part_num=i + 1)

    Complete the multipart upload

    1. mp.complete_upload()

    Abort the multipart upload

    Non-completed uploads can be aborted.

    Delete the bucket

    1. bucket.delete_key(largeObjectKey)