Deploy TiDB Binlog

    TiDB Binlog is disabled in the TiDB cluster by default. To create a TiDB cluster with TiDB Binlog enabled, or enable TiDB Binlog in an existing TiDB cluster, take the following steps.

    1. Modify the CR file to add the Pump configuration.

      For example:

      Since v1.1.6, TiDB Operator supports passing raw TOML configuration to the component:

      1. spec:
      2.   ...
      3.   pump:
      4.     baseImage: pingcap/tidb-binlog
      5.     version: v5.4.0
      6.     replicas: 1
      7.     storageClassName: local-storage
      8.     requests:
      9.       storage: 30Gi
      10.     schedulerName: default-scheduler
      11.     config: |
      12.       addr = "0.0.0.0:8250"
      13.       gc = 7
      14.       heartbeat-interval = 2

      Edit version, replicas, storageClassName, and requests.storage according to your cluster.

      To deploy Enterprise Edition of Pump, edit the YAML file above to set spec.pump.baseImage to the enterprise image (pingcap/tidb-binlog-enterprise).

      For example:

      1. spec:
      2.   pump:
      3.     baseImage: pingcap/tidb-binlog-enterprise

    2. Set affinity and anti-affinity for TiDB and Pump.

      If you enable TiDB Binlog in the production environment, it is recommended to set affinity and anti-affinity for TiDB and the Pump component; if you enable TiDB Binlog in a test environment on the internal network, you can skip this step.

      By default, the affinity of TiDB and Pump is set to {}. Currently, each TiDB instance does not have a corresponding Pump instance by default. When TiDB Binlog is enabled, if Pump and TiDB are separately deployed and network isolation occurs, and ignore-error is enabled in TiDB components, TiDB loses binlogs.

      In this situation, it is recommended to deploy a TiDB instance and a Pump instance on the same node using the affinity feature, and to split Pump instances on different nodes using the anti-affinity feature. For each node, only one Pump instance is required. The steps are as follows:

      • Configure spec.tidb.affinity as follows:

        1. spec:
        2.   tidb:
        3.     affinity:
        4.       podAffinity:
        5.         preferredDuringSchedulingIgnoredDuringExecution:
        6.         - weight: 100
        7.           podAffinityTerm:
        8.             labelSelector:
        9.               matchExpressions:
        10.               - key: "app.kubernetes.io/component"
        11.                 operator: In
        12.                 values:
        13.                 - "pump"
        14.               - key: "app.kubernetes.io/managed-by"
        15.                 operator: In
        16.                 values:
        17.                 - "tidb-operator"
        18.               - key: "app.kubernetes.io/name"
        19.                 operator: In
        20.                 values:
        21.                 - "tidb-cluster"
        22.               - key: "app.kubernetes.io/instance"
        23.                 operator: In
        24.                 - ${cluster_name}
        25.             topologyKey: kubernetes.io/hostname

      • Configure spec.pump.affinity as follows:

        1. spec:
        2.   pump:
        3.     affinity:
        4.       podAffinity:
        5.         preferredDuringSchedulingIgnoredDuringExecution:
        6.         - weight: 100
        7.           podAffinityTerm:
        8.             labelSelector:
        9.               matchExpressions:
        10.               - key: "app.kubernetes.io/component"
        11.                 operator: In
        13.                 - "tidb"
        14.               - key: "app.kubernetes.io/managed-by"
        15.                 operator: In
        16.                 values:
        17.                 - "tidb-operator"
        18.               - key: "app.kubernetes.io/name"
        19.                 operator: In
        20.                 values:
        21.                 - "tidb-cluster"
        22.               - key: "app.kubernetes.io/instance"
        23.                 operator: In
        24.                 values:
        25.                 - ${cluster_name}
        26.             topologyKey: kubernetes.io/hostname
        27.       podAntiAffinity:
        28.         preferredDuringSchedulingIgnoredDuringExecution:
        29.         - weight: 100
        30.           podAffinityTerm:
        31.             labelSelector:
        32.               matchExpressions:
        33.               - key: "app.kubernetes.io/component"
        34.                 operator: In
        35.                 values:
        36.                 - "pump"
        37.               - key: "app.kubernetes.io/managed-by"
        38.                 operator: In
        39.                 values:
        40.                 - "tidb-operator"
        41.               - key: "app.kubernetes.io/name"
        42.                 operator: In
        43.                 values:
        44.                 - "tidb-cluster"
        45.               - key: "app.kubernetes.io/instance"
        46.                 operator: In
        47.                 values:
        48.                 - ${cluster_name}
        49.             topologyKey: kubernetes.io/hostname

      Note

      If you update the affinity configuration of the TiDB components, it will cause rolling updates of the TiDB components in the cluster.

    To deploy multiple drainers using the tidb-drainer Helm chart for a TiDB cluster, take the following steps:

    1. Make sure that the PingCAP Helm repository is up to date:

      1. helm repo update
      1. helm search repo tidb-drainer -l

    2. Get the default values.yaml file to facilitate customization:

      1. Modify the values.yaml file to specify the source TiDB cluster and the downstream database of the drainer. Here is an example:

        The clusterName and clusterVersion must match the desired source TiDB cluster.

        The initialCommitTs is the starting commit timestamp of data replication when Drainer has no checkpoint. The value must be set as a string type, such as "424364429251444742".

        To deploy Enterprise Edition of Drainer, edit the YAML file above to set baseImage to the enterprise image (pingcap/tidb-binlog-enterprise).

        For example:

        1. ...
        2. clusterVersion: v5.4.0
        3. baseImage: pingcap/tidb-binlog-enterprise
        4. ...

      2. Deploy Drainer:

        1. helm install ${release_name} pingcap/tidb-drainer --namespace=${namespace} --version=${chart_version} -f values.yaml

        If the server does not have an external network, refer to deploy the TiDB cluster to download the required Docker image on the machine with an external network and upload it to the server.

        Deploy TiDB Binlog - 图2Note

        This chart must be installed to the same namespace as the source TiDB cluster.

      Enable TLS between TiDB components

      If you want to enable TLS for the TiDB cluster and TiDB Binlog, refer to Enable TLS between Components.

      After you have created a secret and started a TiDB cluster with Pump, edit the values.yaml file to set the tlsCluster.enabled value to true, and configure the corresponding certAllowedCN:

      1. ...
      2. tlsCluster:
      3.   enabled: true
      4.   # certAllowedCN:
      5.   #  - TiDB
      6. ...

      If you set the downstream database of to mysql/tidb, and if you want to enable TLS between Drainer and the downstream database, take the following steps.

      1. Create a secret that contains the TLS information of the downstream database.

        1. kubectl create secret generic ${downstream_database_secret_name} --namespace=${namespace} --from-file=tls.crt=client.pem --from-file=tls.key=client-key.pem --from-file=ca.crt=ca.pem

        tidb-drainer saves the checkpoint in the downstream database by default, so you only need to configure tlsSyncer.tlsClientSecretName and the corresponding cerAllowedCN:

        1. tlsSyncer:
        2.   tlsClientSecretName: ${downstream_database_secret_name}
        3.   # certAllowedCN:
        4.   #  - TiDB

      2. To save the checkpoint of tidb-drainer to other databases that have enabled TLS, create a secret that contains the TLS information of the checkpoint database:

        1. kubectl create secret generic ${checkpoint_tidb_client_secret} --namespace=${namespace} --from-file=tls.crt=client.pem --from-file=tls.key=client-key.pem --from-file=ca.crt=ca.pem

        Edit the values.yaml file to set the tlsSyncer.checkpoint.tlsClientSecretName value to ${checkpoint_tidb_client_secret}, and configure the corresponding certAllowedCN:

        1. ...
        2. tlsSyncer: {}
        3.   tlsClientSecretName: ${downstream_database_secret_name}
        4.   # certAllowedCN:
        5.   #  - TiDB
        6.   checkpoint:
        7.     tlsClientSecretName: ${checkpoint_tidb_client_secret}
        8.     # certAllowedCN:
        9.     #  - TiDB
        10. ...

      For details on how to maintain the node state of the TiDB Binlog cluster, refer to .

      If you want to remove the TiDB Binlog component completely, it is recommended that you first remove Pump nodes and then remove Drainer nodes.

      If TLS is enabled for the TiDB Binlog component to be removed, write the following content into binlog.yaml and execute kubectl apply -f binlog.yaml to start a Pod that is mounted with the TLS file and the binlogctl tool.

      Scale in Pump nodes

      1. Scale in Pump Pods:

        1. kubectl patch tc ${cluster_name} -n ${namespace} --type merge -p '{"spec":{"pump":{"replicas": ${pump_replicas}}}}'

        In the command above, ${pump_replicas} is the desired number of Pump Pods after the scaling.

        Note

        Do not scale in Pump nodes to 0. Otherwise, .

      2. Wait for the Pump Pods to automatically be taken offline and deleted. Run the following command to observe the Pod status:

        1. watch kubectl get po ${cluster_name} -n ${namespace}

      3. If the offline operation fails, that is, the Pump Pods are not deleted for a long time, you can forcibly mark Pump as offline.

        • If TLS is not enabled for Pump, mark Pump as offline:

          1. kubectl run update-pump-${ordinal_id} --image=pingcap/tidb-binlog:${tidb_version} --namespace=${namespace} --restart=OnFailure -- /binlogctl -pd-urls=http://${cluster_name}-pd:2379 -cmd update-pump -node-id ${cluster_name}-pump-${ordinal_id}:8250 --state offline

        • If TLS is enabled for Pump, mark Pump as offline using the previously started Pod:

          1. kubectl exec binlogctl -n ${namespace} -- /binlogctl -pd-urls=https://${cluster_name}-pd:2379 -cmd update-pump -node-id ${cluster_name}-pump-${ordinal_id}:8250 --state offline -ssl-ca "/etc/binlog-tls/ca.crt" -ssl-cert "/etc/binlog-tls/tls.crt" -ssl-key "/etc/binlog-tls/tls.key"

      Deploy TiDB Binlog - 图4Note

      • Before performing the following steps, you need to have at least one Pump node in the cluster. If you have scaled in Pump nodes to 0, you need to scale out Pump at least to 1 node before you perform the removing operation in this section.
      • To scale out the Pump to 1, execute kubectl patch tc ${tidb-cluster} -n ${namespace} --type merge -p '{"spec":{"pump":{"replicas": 1}}}'.

      1. Before removing Pump nodes, execute kubectl patch tc ${cluster_name} -n ${namespace} --type merge -p '{"spec":{"tidb":{"binlogEnabled": false}}}'. After the TiDB Pods are rolling updated, you can remove the Pump nodes.

        If you directly remove Pump nodes, it might cause TiDB failure because TiDB has no Pump nodes to write into.

      2. Refer to Scale in Pump to scale in Pump to 0.

      3. Execute kubectl patch tc ${cluster_name} -n ${namespace} --type json -p '[{"op":"remove", "path":"/spec/pump"}]' to delete all configuration items of spec.pump.

      4. Execute kubectl delete sts ${cluster_name}-pump -n ${namespace} to delete the StatefulSet resources of Pump.

      5. View PVCs used by the Pump cluster by executing kubectl get pvc -n ${namespace} -l app.kubernetes.io/component=pump. Then delete all the PVC resources of Pump by executing kubectl delete pvc -l app.kubernetes.io/component=pump -n ${namespace}.

      Remove Drainer nodes

      1. Take Drainer nodes offline:

        In the following commands, ${drainer_node_id} is the node ID of the Drainer node to be taken offline. If you have configured drainerName in values.yaml of Helm, the value of ${drainer_node_id} is ${drainer_name}-0; otherwise, the value of ${drainer_node_id} is ${cluster_name}-${release_name}-drainer-0.

        • If TLS is not enabled for Drainer, create a Pod to take Drainer offline:

          1. kubectl run offline-drainer-0 --image=pingcap/tidb-binlog:${tidb_version} --namespace=${namespace} --restart=OnFailure -- /binlogctl -pd-urls=http://${cluster_name}-pd:2379 -cmd offline-drainer -node-id ${drainer_node_id}:8249

        • If TLS is enabled for Drainer, use the previously started Pod to take Drainer offline:

          1. kubectl exec binlogctl -n ${namespace} -- /binlogctl -pd-urls "https://${cluster_name}-pd:2379" -cmd offline-drainer -node-id ${drainer_node_id}:8249 -ssl-ca "/etc/binlog-tls/ca.crt" -ssl-cert "/etc/binlog-tls/tls.crt" -ssl-key "/etc/binlog-tls/tls.key"

        View the log of Drainer by executing the following command:

        1. kubectl logs -f -n ${namespace} ${drainer_node_id}

        If drainer offline, please delete my pod is output, this node is successfully taken offline.

      2. Delete the corresponding Drainer Pod:

        Execute helm uninstall ${release_name} -n ${namespace} to delete the Drainer Pod.

        If you no longer need Drainer, execute kubectl delete pvc data-${drainer_node_id} -n ${namespace} to delete the PVC resources of Drainer.

      3. (Optional) Force Drainer to go offline:

        If the offline operation fails, the Drainer Pod will not output drainer offline, please delete my pod. At this time, you can force Drainer to go offline, that is, taking Step 2 to delete the Drainer Pod and mark Drainer as offline.