MongoDB-CDC

    Supported Version

    In order to setup the MongoDB CDC connector, the following table provides dependency information for both projects using a build automation tool (such as Maven or SBT) and SQL Client with SQL JAR bundles.

    Setup MongoDB

    • MongoDB version

      MongoDB version >\= 3.6 We use change streams feature (new in version 3.6) to capture change data.

    • Cluster Deployment

      or sharded clusters is required.

    • storage engine is required.

    • Replica set protocol version

      Replica set protocol version 1 is required. Starting in version 4.0, MongoDB only supports pv1. pv1 is the default for all new replica sets created with MongoDB 3.2 or later.

    The example below shows how to create an MongoDB Extract Node with Flink SQL :

    Note

    MongoDB’s change event record doesn’t have update before message. So, we can only convert it to Flink’s UPSERT changelog stream. An upsert stream requires a unique key, so we must declare _id as primary key. We can’t declare other column as primary key, becauce delete operation do not contain’s the key and value besides and sharding key.

    TODO: It will be supported in the future.

    TODO: It will be supported in the future.

    MongoDB Extract Node Options

    The extended CREATE TABLE example demonstrates the syntax for exposing these metadata fields:

    Data Type Mapping