Tidb-server is still up but don't work after down deploy disk

[mayjiang0203]

Bug Report

What did you do?

1、Deploy tidb in one nvme disk via tiup;
2、Down the nvme disk network (nvme over tcp, down the tcp network to down the nvme );

[root@172 hongm]# iptables -A INPUT -p tcp --dport 4420 -j DROP;env LANG=en_US.UTF-8 date;
Fri May 27 18:50:17 CST 2022

[root@172 hongm]# env LANG=en_US.UTF-8 date;ps -ef|grep tidb-server
Fri May 27 19:05:15 CST 2022
root 15872 15853 99 5月26 ? 2-15:42:38 /tidb-server --store=tikv --advertise-address=tc-tidb-0.tc-tidb-peer.glh-610-mjxt6.svc --host=0.0.0.0 --path=tc-pd:2379 --config=/etc/tidb/tidb.toml
tidb 26080 1 28 18:41 ? 00:06:40 bin/tidb-server -P 4000 --status=10080 --host=0.0.0.0 --advertise-address=172.16.6.34 --store=tikv --path=172.16.6.34:23791,172.16.6.34:23792,172.16.6.34:23793 --log-slow-query=/data/nvme2n1/tidb-deploy/tidb-4000/log/tidb_slow_query.log --config=conf/tidb.toml --log-file=/data/nvme2n1/tidb-deploy/tidb-4000/log/tidb.log

What did you expect to see?

Tidb instance should panic or work fine.

What did you see instead?

tidb-server instance still running but can't work.

4. What is your TiDB version? (Required)

5.2.4

[mayjiang0203]

This test was request by one import customer.
/severity Major

[jebter]

@AstroProfundis @bb7133 PTAL

[tiancaiamao]

What does the tidb log look like?

less /data/nvme2n1/tidb-deploy/tidb-4000/log/tidb.log

[tiancaiamao]

Maybe it's writing log, and the disk is not available ... then it hang there.
I think we can check the tidb log to verify this.

[tiancaiamao]

Maybe it's writing log, and the disk is not available ... then it hang there. I think we can check the tidb log to verify this.

Otherwise, what does the goroutine stack look like?

curl http://127.0.0.1:10080/debug/pprof/goroutine?debug=2 > goroutine.txt

We can check the goroutine stack to see where it blocks.

[tiancaiamao]

The block is caused by a mutex when the code access log writing
The log library need to hold a lock for it's operation.
If the operation can't finish, the mutex won't be free, so the other log writing blocks.
The operation can't finish because the io is blocked... on the OS level

So I don't think this is a bug of TiDB