- Command line flags
cluster
hot [read | write | store]
member [delete | leader_priority | leader [show | resign | transfer <member_name>]]
ping
region key [--format=raw|encode|hex] <key>
region sibling <region_id>
region store <store_id>
region topwrite [limit]
region topversion [limit]
region check [miss-peer | extra-peer | down-peer | pending-peer | offline-peer | empty-region | hist-size | hist-keys]
log [fatal | error | warn | info | debug]
- Jq formatted JSON output usage
It is recommended that the version of the Control tool you use is consistent with the version of the cluster.
To use PD Control, execute the tiup ctl:<cluster-version> pd -u http://<pd_ip>:<pd_port> [-i]
command.
Download installation package
Download the following installation package where PD Control binary locates.
{version}
indicates the version number of TiKV. For example, if {version}
is v5.0.0
, the package download link is https://download.pingcap.org/tidb-v5.0.0-linux-amd64.tar.gz
.
Compile from source code
- Go Version 1.13 or later because the Go modules are used.
- In the root directory of the , use the
make
ormake pd-ctl
command to compile and generatebin/pd-ctl
.
Single-command mode:
Interactive mode:
./pd-ctl -i -u http://127.0.0.1:2379
Use environment variables:
export PD_ADDR=http://127.0.0.1:2379
./pd-ctl
Use TLS to encrypt:
./pd-ctl -u https://127.0.0.1:2379 --cacert="path/to/ca" --cert="path/to/cert" --key="path/to/key"
--cacert
- Specifies the path to the certificate file of the trusted CA in PEM format
- Default: “”
--cert
- Specifies the path to the certificate of SSL in PEM format
- Default: “”
--detach
/ -d
- Uses the single command line mode (not entering readline)
- Default: true
--help
/ -h
- Outputs the help information
- Default: false
--interact
/ -i
- Uses the interactive mode (entering readline)
- Default: false
--key
- Specifies the path to the certificate key file of SSL in PEM format, which is the private key of the certificate specified by
--cert
- Default: “”
--pd
/ -u
- Specifies the PD address
- Default address:
http://127.0.0.1:2379
- Environment variable:
PD_ADDR
--version
/ -V
- Prints the version information and exit
- Default: false
cluster
Use this command to view the basic information of the cluster.
Usage:
>> cluster // To show the cluster information
{
"id": 6493707687106161130,
"max_peer_count": 3
}
config [show | set <option> <value> | placement-rules]
Use this command to view or modify the configuration information.
Usage:
>> config show // Display the config information of the scheduling
{
"replication": {
"enable-placement-rules": "false",
"isolation-level": "",
"location-labels": "",
"max-replicas": 3,
"strictly-match-label": "false"
},
"schedule": {
"enable-cross-table-merge": "true",
"enable-debug-metrics": "false",
"enable-location-replacement": "true",
"enable-make-up-replica": "true",
"enable-one-way-merge": "false",
"enable-remove-down-replica": "true",
"enable-remove-extra-replica": "true",
"enable-replace-offline-replica": "true",
"high-space-ratio": 0.7,
"hot-region-cache-hits-threshold": 3,
"hot-region-schedule-limit": 4,
"leader-schedule-limit": 4,
"leader-schedule-policy": "count",
"low-space-ratio": 0.8,
"max-merge-region-keys": 200000,
"max-merge-region-size": 20,
"max-pending-peer-count": 16,
"max-snapshot-count": 3,
"max-store-down-time": "30m0s",
"merge-schedule-limit": 8,
"patrol-region-interval": "100ms",
"region-schedule-limit": 2048,
"region-score-formula-version": "v2",
"replica-schedule-limit": 64,
"scheduler-max-waiting-operator": 5,
"split-merge-interval": "1h0m0s",
"store-limit-mode": "manual",
"tolerant-size-ratio": 0
}
}
>> config show all // Display all config information
>> config show replication // Display the config information of replication
{
"max-replicas": 3,
"location-labels": "",
"isolation-level": "",
"strictly-match-label": "false",
"enable-placement-rules": "false"
}
>> config show cluster-version // Display the current version of the cluster, which is the current minimum version of TiKV nodes in the cluster and does not correspond to the binary version.
"5.0.0"
max-snapshot-count
controls the maximum number of snapshots that a single store receives or sends out at the same time. The scheduler is restricted by this configuration to avoid taking up normal application resources. When you need to improve the speed of adding replicas or balancing, increase this value.>> config set max-snapshot-count 16 // Set the maximum number of snapshots to 16
max-pending-peer-count
controls the maximum number of pending peers in a single store. The scheduler is restricted by this configuration to avoid producing a large number of Regions without the latest log in some nodes. When you need to improve the speed of adding replicas or balancing, increase this value. Setting it to 0 indicates no limit.>> config set max-pending-peer-count 64 // Set the maximum number of pending peers to 64
max-merge-region-size
controls the upper limit on the size of Region Merge (the unit is M). WhenregionSize
exceeds the specified value, PD does not merge it with the adjacent Region. Setting it to 0 indicates disabling Region Merge.>> config set max-merge-region-size 16 // Set the upper limit on the size of Region Merge to 16M
max-merge-region-keys
controls the upper limit on the key count of Region Merge. WhenregionKeyCount
exceeds the specified value, PD does not merge it with the adjacent Region.>> config set max-merge-region-keys 50000 // Set the the upper limit on keyCount to 50000
split-merge-interval
controls the interval between thesplit
andmerge
operations on a same Region. This means the newly split Region won’t be merged within a period of time.>> config set split-merge-interval 24h // Set the interval between `split` and `merge` to one day
enable-one-way-merge
controls whether PD only allows a Region to merge with the next Region. When you set it tofalse
, PD allows a Region to merge with the adjacent two Regions.>> config set enable-one-way-merge true // Enables one-way merging.
enable-cross-table-merge
is used to enable the merging of cross-table Regions. When you set it tofalse
, PD does not merge the Regions from different tables. This option only works when key type is “table”.>> config set enable-cross-table-merge true // Enable cross table merge.
key-type
specifies the key encoding type used for the cluster. The supported options are [“table”, “raw”, “txn”], and the default value is “table”.- If no TiDB instance exists in the cluster,
key-type
will be “raw” or “txn”, and PD is allowed to merge Regions across tables regardless of theenable-cross-table-merge
setting. - If any TiDB instance exists in the cluster,
key-type
should be “table”. Whether PD can merge Regions across tables is determined byenable-cross-table-merge
. Ifkey-type
is “raw”, placement rules do not work.
>> config set key-type raw // Enable cross table merge.
- If no TiDB instance exists in the cluster,
region-score-formula-version
controls the version of the Region score formula. The value options arev1
andv2
. The version 2 of the formula helps to reduce redundant balance Region scheduling in some scenarios, such as taking TiKV nodes online or offline.>> config set region-score-formula-version v2
patrol-region-interval
controls the execution frequency thatreplicaChecker
checks the health status of Regions. A shorter interval indicates a higher execution frequency. Generally, you do not need to adjust it.max-store-down-time
controls the time that PD decides the disconnected store cannot be restored if exceeded. If PD does not receive heartbeats from a store within the specified period of time, PD adds replicas in other nodes.>> config set max-store-down-time 30m // Set the time within which PD receives no heartbeats and after which PD starts to add replicas to 30 minutes
leader-schedule-limit
controls the number of tasks scheduling the leader at the same time. This value affects the speed of leader balance. A larger value means a higher speed and setting the value to 0 closes the scheduling. Usually the leader scheduling has a small load, and you can increase the value in need.>> config set leader-schedule-limit 4 // 4 tasks of leader scheduling at the same time at most
region-schedule-limit
controls the number of tasks of scheduling Regions at the same time. This value avoids too many Region balance operators being created. The default value is2048
which is enough for all sizes of clusters, and setting the value to0
closes the scheduling. Usually, the Region scheduling speed is limited bystore-limit
, but it is recommended that you do not customize this value unless you know exactly what you are doing.>> config set region-schedule-limit 2 // 2 tasks of Region scheduling at the same time at most
controls the number of tasks scheduling the replica at the same time. This value affects the scheduling speed when the node is down or removed. A larger value means a higher speed and setting the value to 0 closes the scheduling. Usually the replica scheduling has a large load, so do not set a too large value.
>> config set replica-schedule-limit 4 // 4 tasks of replica scheduling at the same time at most
merge-schedule-limit
controls the number of Region Merge scheduling tasks. Setting the value to 0 closes Region Merge. Usually the Merge scheduling has a large load, so do not set a too large value.>> config set merge-schedule-limit 16 // 16 tasks of Merge scheduling at the same time at most
hot-region-schedule-limit
controls the hot Region scheduling tasks that are running at the same time. Setting its value to0
means to disable the scheduling. It is not recommended to set a too large value, otherwise it might affect the system performance.>> config set hot-region-schedule-limit 4 // 4 tasks of hot Region scheduling at the same time at most
hot-region-cache-hits-threshold
is used to set the number of minutes required to identify a hot Region. PD can participate in the hotspot scheduling only after the Region is in the hotspot state for more than this number of minutes.tolerant-size-ratio
controls the size of the balance buffer area. When the score difference between the leader or Region of the two stores is less than specified multiple times of the Region size, it is considered in balance by PD.low-space-ratio
controls the threshold value that is considered as insufficient store space. When the ratio of the space occupied by the node exceeds the specified value, PD tries to avoid migrating data to the corresponding node as much as possible. At the same time, PD mainly schedules the remaining space to avoid using up the disk space of the corresponding node.config set low-space-ratio 0.9 // Set the threshold value of insufficient space to 0.9
high-space-ratio
controls the threshold value that is considered as sufficient store space. This configuration takes effect only whenregion-score-formula-version
is set tov1
. When the ratio of the space occupied by the node is less than the specified value, PD ignores the remaining space and mainly schedules the actual data volume.config set high-space-ratio 0.5 // Set the threshold value of sufficient space to 0.5
cluster-version
is the version of the cluster, which is used to enable or disable some features and to deal with the compatibility issues. By default, it is the minimum version of all normally running TiKV nodes in the cluster. You can set it manually only when you need to roll it back to an earlier version.config set cluster-version 1.0.8 // Set the version of the cluster to 1.0.8
replication-mode
controls the replication mode of Regions in the dual data center scenario. See for details.leader-schedule-policy
is used to select the scheduling strategy for the leader. You can schedule the leader according tosize
orcount
.enable-remove-down-replica
is used to enable the feature of automatically deleting DownReplica. When you set it tofalse
, PD does not automatically clean up the downtime replicas.enable-replace-offline-replica
is used to enable the feature of migrating OfflineReplica. When you set it tofalse
, PD does not migrate the offline replicas.enable-make-up-replica
is used to enable the feature of making up replicas. When you set it tofalse
, PD does not add replicas for Regions without sufficient replicas.enable-remove-extra-replica
is used to enable the feature of removing extra replicas. When you set it tofalse
, PD does not remove extra replicas for Regions with redundant replicas.enable-location-replacement
is used to enable the isolation level checking. When you set it tofalse
, PD does not increase the isolation level of a Region replica through scheduling.enable-debug-metrics
is used to enable the metrics for debugging. When you set it totrue
, PD enables some metrics such asbalance-tolerant-size
.enable-placement-rules
is used to enable placement rules.store-limit-mode
is used to control the mode of limiting the store speed. The optional modes areauto
andmanual
. Inauto
mode, the stores are automatically balanced according to the load (experimental).
config placement-rules [disable | enable | load | save | show | rule-group]
For the usage of config placement-rules [disable | enable | load | save | show | rule-group]
, see .
health
Use this command to view the health information of the cluster.
Usage:
>> health // Display the health information
[
{
"name": "pd",
"member_id": 13195394291058371180,
"client_urls": [
"http://127.0.0.1:2379"
......
],
"health": true
}
......
]
Use this command to view the hot spot information of the cluster.
Usage:
>> hot read // Display hot spot for the read operation
>> hot write // Display hot spot for the write operation
>> hot store // Display hot spot for all the read and write operations
label [store <name> <value>]
Use this command to view the label information of the cluster.
Usage:
>> label // Display all labels
>> label store zone cn // Display all stores including the "zone":"cn" label
member [delete | leader_priority | leader [show | resign | transfer <member_name>]]
Use this command to view the PD members, remove a specified member, or configure the priority of leader.
Usage:
>> member // Display the information of all members
{
"header": {......},
"members": [......],
"leader": {......},
"etcd_leader": {......},
}
>> member delete name pd2 // Delete "pd2"
Success!
>> member delete id 1319539429105371180 // Delete a node using id
Success!
>> member leader show // Display the leader information
{
"name": "pd",
"member_id": 13155432540099656863,
"peer_urls": [......],
"client_urls": [......]
}
>> member leader resign // Move leader away from the current member
......
>> member leader transfer pd3 // Migrate leader to a specified member
......
operator [check | show | add | remove]
Use this command to view and control the scheduling operation.
Usage:
>> operator show // Display all operators
>> operator show admin // Display all admin operators
>> operator show leader // Display all leader operators
>> operator show region // Display all Region operators
>> operator add add-peer 1 2 // Add a replica of Region 1 on store 2
>> operator add add-learner 1 2 // Add a learner replica of Region 1 on store 2
>> operator add remove-peer 1 2 // Remove a replica of Region 1 on store 2
>> operator add transfer-leader 1 2 // Schedule the leader of Region 1 to store 2
>> operator add transfer-region 1 2 3 4 // Schedule Region 1 to stores 2,3,4
>> operator add transfer-peer 1 2 3 // Schedule the replica of Region 1 on store 2 to store 3
>> operator add merge-region 1 2 // Merge Region 1 with Region 2
>> operator add split-region 1 --policy=approximate // Split Region 1 into two Regions in halves, based on approximately estimated value
>> operator add split-region 1 --policy=scan // Split Region 1 into two Regions in halves, based on accurate scan value
>> operator remove 1 // Remove the scheduling operation of Region 1
>> operator check 1 // Check the status of the operators related to Region 1
The splitting of Regions starts from the position as close as possible to the middle. You can locate this position using two strategies, namely “scan” and “approximate”. The difference between them is that the former determines the middle key by scanning the Region, and the latter obtains the approximate position by checking the statistics recorded in the SST file. Generally, the former is more accurate, while the latter consumes less I/O and can be completed faster.
ping
Use this command to view the time that ping
PD takes.
Usage:
>> ping
time: 43.12698ms
region <region_id> [--jq="<query string>"]
Use this command to view the Region information. For a jq formatted output, see .
Usage:
>> region // Display the information of all Regions
{
"count": 1,
"regions": [......]
}
>> region 2 // Display the information of the Region with the ID of 2
{
"id": 2,
"start_key": "7480000000000000FF1D00000000000000F8",
"end_key": "7480000000000000FF1F00000000000000F8",
"epoch": {
"conf_ver": 1,
"version": 15
},
"peers": [
{
"id": 40,
"store_id": 3
}
],
"leader": {
"id": 40,
"store_id": 3
},
"written_bytes": 0,
"read_bytes": 0,
"written_keys": 0,
"read_keys": 0,
"approximate_size": 1,
"approximate_keys": 0
}
region key [--format=raw|encode|hex] <key>
Use this command to query the Region that a specific key resides in. It supports the raw, encoding, and hex formats. And you need to use single quotes around the key when it is in the encoding format.
Hex format usage (default):
>> region key 7480000000000000FF1300000000000000F8
{
"region": {
"id": 2,
......
}
}
Raw format usage:
>> region key --format=raw abc
{
"region": {
"id": 2,
......
}
}
Encoding format usage:
>> region key --format=encode 't\200\000\000\000\000\000\000\377\035_r\200\000\000\000\000\377\017U\320\000\000\000\000\000\372'
{
"region": {
"id": 2,
......
}
}
region scan
Use this command to get all Regions.
Usage:
>> region scan
{
"count": 20,
"regions": [......],
}
region sibling <region_id>
Use this command to check the adjacent Regions of a specific Region.
Usage:
>> region sibling 2
{
"count": 2,
}
Use this command to query all Regions starting from a key.
Usage:
>> region startkey --format=raw abc
{
"count": 16,
"regions": [......],
}
region store <store_id>
Use this command to list all Regions of a specific store.
>> region store 2
{
"count": 10,
"regions": [......],
}
region topread [limit]
Use this command to list Regions with top read flow. The default value of the limit is 16.
Usage:
>> region topread
{
"count": 16,
"regions": [......],
}
region topwrite [limit]
Use this command to list Regions with top write flow. The default value of the limit is 16.
Usage:
>> region topwrite
{
"count": 16,
"regions": [......],
}
region topconfver [limit]
Use this command to list Regions with top conf version. The default value of the limit is 16.
Usage:
>> region topconfver
{
"count": 16,
"regions": [......],
}
Use this command to list Regions with top version. The default value of the limit is 16.
Usage:
>> region topversion
{
"count": 16,
"regions": [......],
}
region topsize [limit]
Use this command to list Regions with top approximate size. The default value of the limit is 16.
Usage:
region check [miss-peer | extra-peer | down-peer | pending-peer | offline-peer | empty-region | hist-size | hist-keys]
Use this command to check the Regions in abnormal conditions.
Description of various types:
- miss-peer: the Region without enough replicas
- extra-peer: the Region with extra replicas
- down-peer: the Region in which some replicas are Down
- pending-peer:the Region in which some replicas are Pending
Usage:
>> region check miss-peer
{
"count": 2,
"regions": [......],
}
scheduler [show | add | remove | pause | resume | config]
Use this command to view and control the scheduling policy.
Usage:
>> scheduler show // Display all schedulers
>> scheduler add grant-leader-scheduler 1 // Schedule all the leaders of the Regions on store 1 to store 1
>> scheduler add evict-leader-scheduler 1 // Move all the Region leaders on store 1 out
>> scheduler config evict-leader-scheduler // Display the stores in which the scheduler is located since v4.0.0
>> scheduler add shuffle-leader-scheduler // Randomly exchange the leader on different stores
>> scheduler add shuffle-region-scheduler // Randomly scheduling the regions on different stores
>> scheduler remove grant-leader-scheduler-1 // Remove the corresponding scheduler, and `-1` corresponds to the store ID
>> scheduler pause balance-region-scheduler 10 // Pause the balance-region scheduler for 10 seconds
>> scheduler pause all 10 // Pause all schedulers for 10 seconds
>> scheduler resume balance-region-scheduler // Continue to run the balance-region scheduler
>> scheduler resume all // Continue to run all schedulers
>> scheduler config balance-hot-region-scheduler // Display the configuration of the balance-hot-region scheduler
scheduler config balance-hot-region-scheduler
Use this command to view and control the balance-hot-region-scheduler
policy.
Usage:
>> scheduler config balance-hot-region-scheduler // Display all configuration of the balance-hot-region scheduler
{
"min-hot-byte-rate": 100,
"min-hot-key-rate": 10,
"max-zombie-rounds": 3,
"max-peer-number": 1000,
"byte-rate-rank-step-ratio": 0.05,
"key-rate-rank-step-ratio": 0.05,
"count-rank-step-ratio": 0.01,
"great-dec-ratio": 0.95,
"minor-dec-ratio": 0.99,
"src-tolerance-ratio": 1.02,
"dst-tolerance-ratio": 1.02
}
min-hot-byte-rate
means the smallest byte counted, which is usually 100.>> scheduler config balance-hot-region-scheduler set min-hot-byte-rate 100
min-hot-key-rate
means the smallest key counted, which is usually 10.>> scheduler config balance-hot-region-scheduler set min-hot-key-rate 10
max-zombie-rounds
means the maximum number of heartbeats with which an operator can be considered as the pending influence. If you set it to a larger value, more operators might be included in the pending influence. Usually, you do not need to adjust its value. Pending influence refers to the operator influence that is generated during scheduling but still has an effect.>> scheduler config balance-hot-region-scheduler set max-zombie-rounds 3
max-peer-number
means the maximum number of peers to be solved, which prevents the scheduler from being too slow.>> scheduler config balance-hot-region-scheduler set max-peer-number 1000
byte-rate-rank-step-ratio
,key-rate-rank-step-ratio
, andcount-rank-step-ratio
respectively mean the step ranks of byte, key, and count. The rank step ratio decides the step when the rank is calculated.great-dec-ratio
andminor-dec-ratio
are used to determine thedec
rank. Usually, you do not need to modify these items.>> scheduler config balance-hot-region-scheduler set byte-rate-rank-step-ratio 0.05
src-tolerance-ratio
anddst-tolerance-ratio
are configuration items for the expectation scheduler. The smaller thetolerance-ratio
, the easier it is for scheduling. When redundant scheduling occurs, you can appropriately increase this value.>> scheduler config balance-hot-region-scheduler set src-tolerance-ratio 1.05
store [delete | label | weight | remove-tombstone | limit | limit-scene] <store_id> [--jq="<query string>"]
Use this command to view the store information or remove a specified store. For a jq formatted output, see jq-formatted-json-output-usage.
Usage:
>> store // Display information of all stores
{
"count": 3,
"stores": [...]
}
>> store 1 // Get the store with the store id of 1
......
>> store delete 1 // Delete the store with the store id of 1
......
>> store label 1 zone cn // Set the value of the label with the "zone" key to "cn" for the store with the store id of 1
>> store weight 1 5 10 // Set the leader weight to 5 and Region weight to 10 for the store with the store id of 1
>> store remove-tombstone // Remove stores that are in tombstone state
>> store limit // Show the speed limit of adding-peer operations and the limit of removing-peer operations per minute in all stores
>> store limit add-peer // Show the speed limit of adding-peer operations per minute in all stores
>> store limit remove-peer // Show the limit of removing-peer operations per minute in all stores
>> store limit all 5 // Set the limit of adding-peer operations to 5 and the limit of removing-peer operations to 5 per minute for all stores
>> store limit 1 5 // Set the limit of adding-peer operations to 5 and the limit of removing-peer operations to 5 per minute for store 1
>> store limit all 5 add-peer // Set the limit of adding-peer operations to 5 per minute for all stores
>> store limit 1 5 add-peer // Set the limit of adding-peer operations to 5 per minute for store 1
>> store limit 1 5 remove-peer // Set the limit of removing-peer operations to 5 per minute for store 1
>> store limit all 5 remove-peer // Set the limit of removing-peer operations to 5 per minute for all stores
>> store limit-scene // Show all limit scenarios (experimental)
{
"Idle": 100,
"Low": 50,
"Normal": 32,
"High": 12
}
>> store limit-scene idle 100 // set rate to 100 in the idle scene (experimental)
When you use the store limit
command, the original region-add
and region-remove
are deprecated. Use add-peer
and remove-peer
instead.
log [fatal | error | warn | info | debug]
Use this command to set the log level of the PD leader.
Usage:
>> log warn
tso
Use this command to parse the physical and logical time of TSO.
Usage:
>> tso 395181938313123110 // Parse TSO
system: 2017-10-09 05:50:59 +0800 CST
logic: 120102
Simplify the output of store
» store --jq=".stores[].store | { id, address, state_name}"
{"id":1,"address":"127.0.0.1:20161","state_name":"Up"}
{"id":30,"address":"127.0.0.1:20162","state_name":"Up"}
...
Query the remaining space of the node
» store --jq=".stores[] | {id: .store.id, available: .status.available}"
{"id":1,"available":"10 GiB"}
{"id":30,"available":"10 GiB"}
...
Query all nodes whose status is not Up
» store --jq='.stores[].store | select(.state_name!="Up") | { id, address, state_name}'
{"id":1,"address":"127.0.0.1:20161""state_name":"Offline"}
{"id":5,"address":"127.0.0.1:20162""state_name":"Offline"}
...
Query the distribution status of the Region replicas
» region --jq=".regions[] | {id: .id, peer_stores: [.peers[].store_id]}"
{"id":2,"peer_stores":[1,30,31]}
{"id":4,"peer_stores":[1,31,34]}
...
Filter Regions according to the number of replicas
For example, to filter out all Regions whose number of replicas is not 3:
» region --jq=".regions[] | {id: .id, peer_stores: [.peers[].store_id] | select(length != 3)}"
{"id":12,"peer_stores":[30,32]}
{"id":2,"peer_stores":[1,30,31,32]}
Filter Regions according to the store ID of replicas
For example, to filter out all Regions that have a replica on store30:
» region --jq=".regions[] | {id: .id, peer_stores: [.peers[].store_id] | select(any(.==30))}"
{"id":6,"peer_stores":[1,30,31]}
{"id":22,"peer_stores":[1,30,32]}
...
You can also find out all Regions that have a replica on store30 or store31 in the same way:
» region --jq=".regions[] | {id: .id, peer_stores: [.peers[].store_id] | select(any(.==(30,31)))}"
{"id":16,"peer_stores":[1,30,34]}
{"id":28,"peer_stores":[1,30,32]}
{"id":12,"peer_stores":[30,32]}
...
Look for relevant Regions when restoring data
For example, when [store1, store30, store31] is unavailable at its downtime, you can find all Regions whose Down replicas are more than normal replicas:
» region --jq=".regions[] | {id: .id, peer_stores: [.peers[].store_id] | select(length as $total | map(if .==(1,30,31) then . else empty end) | length>=$total-length) }"
{"id":2,"peer_stores":[1,30,31,32]}
{"id":12,"peer_stores":[30,32]}
{"id":14,"peer_stores":[1,30,32]}
...
Or when [store1, store30, store31] fails to start, you can find Regions where the data can be manually removed safely on store1. In this way, you can filter out all Regions that have a replica on store1 but don’t have other DownPeers:
» region --jq=".regions[] | {id: .id, remove_peer: [.peers[].store_id] | select(length>1) | map(if .==(30,31) then . else empty end) | select(length==1)}"
{"id":12,"remove_peer":[30]}
{"id":4,"remove_peer":[31]}
{"id":22,"remove_peer":[30]}
...