Aliyun OSS

Use the tool shipped with Druid to install the extension, as described here on middle manager and historical nodes.

After installation, add this aliyun-oss-extensions extension to druid.extensions.loadList in common.runtime.properties and then restart middle manager and historical nodes.

First add the following OSS configurations to common.runtime.properties

To use OSS as deep storage, add the following configurations:

Property	Description	Required
`druid.storage.type`	Global deep storage provider. Must be set to `oss` to make use of this extension.	yes
`druid.storage.oss.bucket`	Storage bucket name.	yes
`druid.storage.oss.prefix`	Folder where segments will be published to. `druid/segments` is recommended.	No

Currently, Web Console does not support ingestion from OSS, but it could be done by submitting an ingestion task with OSS’s input source configuration.

Below shows the configurations of OSS’s input source.

property	description	Required
type	This should be `oss`.	yes
uris	JSON array of URIs where OSS objects to be ingested are located. For example, `oss://{your_bucket}/{source_file_path}`	`uris` or `prefixes` or `objects` must be set
prefixes	JSON array of URI prefixes for the locations of OSS objects to be ingested. Empty objects starting with one of the given prefixes will be skipped.	`uris` or `prefixes` or `objects` must be set
objects	JSON array of to be ingested.	`uris` or `prefixes` or `objects` must be set
properties	Properties Object for overriding the default OSS configuration. See below for more information.	no (defaults will be used if not given)

OSS Object

Properties Object

Property	Description	Default	Required
accessKey	The or plain text string of this OSS InputSource’s access key	None	yes
secretKey	The Password Provider or plain text string of this OSS InputSource’s secret key	None	yes
endpoint	The endpoint of this OSS InputSource	None	no

Say that the file , which can be found under Druid’s quickstart/tutorial directory, has been uploaded to a folder druid in your OSS bucket, the bucket for which your Druid is configured. In this case, the uris property of the OSS’s input source can be used for reading, as shown:

By posting the above ingestion task spec to http://{YOUR_ROUTER_IP}:8888/druid/indexer/v1/task, an ingestion task will be created by the indexing service to ingest.

The spec above tells the ingestion task to read all files under 2020 and 2021 folders.

If you want to read from files in buckets which are different from the bucket Druid is configured, use objects property of OSS’s InputSource for task submission as below:

If the default druid.oss.accessKey is not able to access a bucket, properties could be used to customize these secret information as below:

This properties could be applied to any of uris, objects, prefixes property above.