CREATE SYNC JOB

    At present, data synchronization only supports docking with the canal, getting the parsed binlog from the canal server and loading it into Doris.

    You can view the SyncJob’s status by command ‘SHOW SYNC JOB’.

    Syntax:

    1. job_Name is the unique identifier of the SyncJob in the current database. With a specified job name, only one SyncJob can be running at the same time.

    2. channel_desc

      The data channel under the job is used to describe the mapping relationship between the MySQL source table and the Doris target table.

      1. FROM mysql_db.src_tbl INTO des_tbl
      2. [partitions]
      3. [columns_mapping]

      1. mysql_db.src_tbl

        Specify the database and source table on the MySQL side.

      2. des_tbl

        Specify the target table on the Doris side. Only the unique table is supported, and the batch delete feature of the table needs to be enabled.

      3. partitions

        Specify which partitions to be load into in target table. If it is not specified, it will be automatically loaded into the corresponding partition.

        Example:

      4. Specify the mapping relationship between the columns of the MySQL source table and the Doris target table. If not specified, Fe will default that the columns of the source table and the target table correspond one by one in order.

        Columns are not supported in the ‘col_name = expr’ form.

        Example:

        1. Suppose the columns of target table are (K1, K2, V1),
        5. Ignore the fourth column of the source data
        6. COLUMNS(k2, k1, v1, dummy_column)

    3. binlog_desc

      It is used to describe remote data sources. Currently, only canal is supported.

      Syntax:

      1. The attribute related to the canal is prefixed with canal.

        1. canal.server.ip: the address of the canal server
        2. canal.server.port: the port of canal server
        3. canal.destination: Identifier of instance
        4. canal.batchSize: the maximum batch size. The default is 8192
        5. canal.username: the username of instance
        6. canal.password: password of instance
        7. canal.debug: optional. When set to true, the details of each batch and each row will be printed.
      1.  (
      3.  )
      4.  FROM BINLOG 
      5.  (
      6.      "type" = "canal",
      7.      "canal.server.ip" = "127.0.0.1",
      8.      "canal.server.port" = "11111",
      9.      "canal.destination" = "example",
      10.      "canal.username" = "",
      11.      "canal.password" = ""
      12.  );

    2. create a sync job named job1 for multiple target tables in , correspond to multiple MySQL source tables one by one, and explicitly specify column mapping.

    1. CREATE,SYNC,JOB,BINLOG