[PD] delete store node failed

[Xavier1994]

Please answer these questions before submitting your issue. Thanks!

1. What did you do?
   If possible, provide a recipe for reproducing the error.

I delete a tikv node from pd-ctl, but after i delete the node, the node state is offline, I expect it is tomstone. and my region still on this store

2. What did you expect to see?

3. What did you see instead?

4. What version of PD are you using (pd-server -V)?

[Xavier1994]

I wait for a long time(> 1h), still this state, and i restart pd && tikv, it is still the same state

[disksing]

Hi @Xavier1994 , we need to move all replica on store5 to other stores before set it to tombstone. Do you have enough up tikv-servers with > 80% available space? Could you check the output of "store" command? You may also check if there are any log in PD formatted as "store %v may not turn into Tombstone, there are no extra up node has enough space to accommodate the extra replica".

[Xavier1994]

@disksing ok, but the region 6905 have no leader is reasonable? i hava extra 7 * tikv node. because the region 6905 have no leader, so my request to that region failed

[Xavier1994]

and the "region check down-peer" command's output is also confusing

seems there are many down peer, but pd-ctl can't display them

[Xavier1994]

And I think my store capacity is enough

[disksing]

This does not look normal.
For the region 6905, it seems that something wrong with leader election, so that no heartbeat is sent to the PD. You can check the log related to this region by using grep "region 6905" in the corresponding tikv log files.
For the stuck offline process, you may check the grafana panels "Scheduler/ReplicaChecker" and "Opeator/Schedule Operator Create" in PD page to see if anything is wrong.
For the strange output of "region check down-peer", @nolouch do you have any clue?
In addition, if possible, you could provide the cluster access method (grafana/ssh) to us for help. You can communicate privately via email at menglong AT pingcap.com.

[nolouch]

and @Xavier1994 Could you show me the result about pdctl>> config show all?

[Xavier1994]

@nolouch I have destroyed my cluster because the machine is used for another sake. but I just change some config to speeded up the rebalance. max-snapshot: 16, max-pending-peer: 64, region-schedule-limit 64, leader-schedule-limit 16. and others use default config

[nolouch]

@Xavier1994 thanks for your feedback. If this problem reappears, you can contact us immediately.