Access Ozone object store with Amazon Boto3 client

    • Create bucket
    • List bucket
    • Head bucket
    • Delete bucket
    • Upload file
    • Download file
    • Delete objects(keys)
    • Head object
    • Multipart upload

    You will need a higher version of Python3 for your Boto3 client as Boto3 installation requirement indicates at here: https://boto3.amazonaws.com/v1/documentation/api/latest/index.html

    You may reference Amazon Boto3 documentation regarding the creation of ‘s3’ resources at here:

    You may reference Amazon Boto3 documentation regarding session at here: https://boto3.amazonaws.com/v1/documentation/api/latest/reference/core/session.html

    2.     session = boto3.session.Session()
    4. Obtain s3 client to Ozone via session:
    6.     s3_client = session.client(
    7.         service_name='s3',
    8.         aws_access_key_id='testuser/scm@EXAMPLE.COM',
    9.         endpoint_url='http://localhost:9878',
    10.     )
    11.     'endpoint_url' is pointing to Ozone s3 endpoint.
    13. In our code sample below, we're demonstrating the usage of both s3 and s3_client.

    There are multiple ways to configure Boto3 client credentials if you’re connecting to a secured cluster. In these cases, the above lines of passing ‘aws_access_key_id’ and ‘aws_secret_access_key’ when creating Ozone s3 client shall be skipped.

    1. response = s3_client.create_bucket(Bucket='bucket1')
    2. print(response)

    This will create a bucket ‘bucket1’ in Ozone volume ‘s3v’.

    List buckets

    This will list all buckets in Ozone volume ‘s3v’.

    Head a bucket

    1. response = s3_client.head_bucket(Bucket='bucket1')
    2. print(response)

    This will head bucket ‘bucket1’ in Ozone volume ‘s3v’.

    1. response = s3_client.delete_bucket(Bucket='bucket1')
    2. print(response)

    This will delete the bucket ‘bucket1’ from Ozone volume ‘s3v’.

    Upload a file

    Download a file

    1. response = s3.Bucket('bucket1').download_file('README.md', 'download.md')
    2. print(response)

    This will download ‘README.md’ from Ozone volume ‘s3v’ to local and create a file with name ‘download.md’.

    1. response = s3_client.head_object(Bucket='bucket1', Key='README.md')

    This will head object ‘README.md’ from Ozone volume ‘s3v’ in the bucket ‘bucket1’.

    Delete Objects

    This will delete objects ‘README3.md’ and ‘README4.md’ from Ozone volume ‘s3v’ in bucket ‘bucket1’.

    Multipart upload

    1. response = s3_client.create_multipart_upload(Bucket='bucket1', Key='key1')
    2. print(response)
    3. uid=response['UploadId']
    4. print(uid)
    6. response = s3_client.upload_part_copy(
    7.     Bucket='bucket1',
    8.     CopySource='/bucket1/maven.gz',
    9.     Key='key1',
    10.     PartNumber=1,
    11.     UploadId=str(uid)
    12. )
    13. print(response)
    14. etag1=response.get('CopyPartResult').get('ETag')
    16. response = s3_client.upload_part_copy(
    17.     Bucket='bucket1',
    18.     CopySource='/bucket1/maven1.gz',
    19.     Key='key1',
    20.     PartNumber=2,
    21.     UploadId=str(uid)
    23. print(response)
    24. etag2=response.get('CopyPartResult').get('ETag')
    25. print(etag2)
    27. response = s3_client.complete_multipart_upload(
    28.     Bucket='bucket1',
    29.     Key='key1',
    30.     MultipartUpload={
    31.         'Parts': [
    32.             {
    33.                 'ETag': str(etag1),
    34.                 'PartNumber': 1,
    35.             },
    36.             {
    37.                 'ETag': str(etag2),
    38.                 'PartNumber': 2,
    39.             },
    40.         ],
    41.     },
    42.     UploadId=str(uid),
    43. )

    This will use ‘maven.gz’ and ‘maven1.gz’ as copy source from Ozone volume ‘s3v’ to create a new object ‘key1’ in Ozone volume ‘s3v’. Please note ‘ETag’s is required and important for the call.