我的书签
添加书签
移除书签Use the pull-deps tool shipped with Druid to install the extension, as described on middle manager and historical nodes.
After installation, add this aliyun-oss-extensions extension to druid.extensions.loadList in common.runtime.properties and then restart middle manager and historical nodes.
First add the following OSS configurations to common.runtime.properties
To use OSS as deep storage, add the following configurations:
| Property | Description | Required |
|---|---|---|
druid.storage.type | Global deep storage provider. Must be set to oss to make use of this extension. | yes |
druid.storage.oss.bucket | Storage bucket name. | yes |
druid.storage.oss.prefix | Folder where segments will be published to. druid/segments is recommended. | No |
Currently, Web Console does not support ingestion from OSS, but it could be done by submitting an ingestion task with OSS’s input source configuration.
Below shows the configurations of OSS’s input source.
| property | description | Required |
|---|---|---|
| type | This should be oss. | yes |
| uris | JSON array of URIs where OSS objects to be ingested are located. For example, oss://{your_bucket}/{source_file_path} | uris or prefixes or objects must be set |
| prefixes | JSON array of URI prefixes for the locations of OSS objects to be ingested. Empty objects starting with one of the given prefixes will be skipped. | uris or prefixes or objects must be set |
| objects | JSON array of OSS Objects to be ingested. | uris or prefixes or objects must be set |
| properties | for overriding the default OSS configuration. See below for more information. | no (defaults will be used if not given) |
OSS Object
Properties Object
| Property | Description | Default | Required |
|---|---|---|---|
| accessKey | The Password Provider or plain text string of this OSS InputSource’s access key | None | yes |
| secretKey | The or plain text string of this OSS InputSource’s secret key | None | yes |
| endpoint | The endpoint of this OSS InputSource | None | no |
Say that the file , which can be found under Druid’s quickstart/tutorial directory, has been uploaded to a folder druid in your OSS bucket, the bucket for which your Druid is configured. In this case, the uris property of the OSS’s input source can be used for reading, as shown:
By posting the above ingestion task spec to http://{YOUR_ROUTER_IP}:8888/druid/indexer/v1/task, an ingestion task will be created by the indexing service to ingest.
The spec above tells the ingestion task to read all files under 2020 and 2021 folders.
If you want to read from files in buckets which are different from the bucket Druid is configured, use objects property of OSS’s InputSource for task submission as below:
If the default druid.oss.accessKey is not able to access a bucket, properties could be used to customize these secret information as below:
This properties could be applied to any of uris, objects, prefixes property above.
