Elasticsearch

The Load Node can operate in upsert mode for exchanging UPDATE/DELETE messages with the external system using the primary key defined on the DDL.

If no primary key is defined on the DDL, the Load Node can only operate in append mode for exchanging INSERT only messages with external system.

Supported Version

In order to use the Elasticsearch Load Node the following dependencies are required for both projects using a build automation tool (such as Maven or SBT) and SQL Client with Sort Connectors JAR bundles.

• Elasticsearch 6

• Elasticsearch 7

Usage for SQL API

TODO: It will be supported in the future.

Usage for InLong Manager Client

TODO: It will be supported in the future.

Elasticsearch Load Node Options

Elasticsearch sink can work in either upsert mode or append mode, it depends on whether primary key is defined. If primary key is defined, Elasticsearch sink works in upsert mode which can consume queries containing UPDATE/DELETE messages. If primary key is not defined, Elasticsearch sink works in append mode which can only consume queries containing INSERT only messages.

See CREATE TABLE DDL for more details about PRIMARY KEY syntax.

Dynamic Index

Elasticsearch Load Node supports both static index and dynamic index.

If you want to have a static index, the index option value should be a plain string, e.g. 'myusers', all the records will be consistently written into “myusers” index.

Data Type Mapping