Is there a way to locate why the "tcp reset" is actively being sent?

[Jamlee]

Version

v12.13.1

Platform

Linux VM-0-10-centos 3.10.0-1127.19.1.el7.x86_64 #1 SMP Tue Aug 25 17:23:54 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

Subsystem

No response

What steps will reproduce the bug?

The http server has a 99.95% probability of actively sending reset. The timeout is not triggered, but there is a 99.95% probability of sending a connection reset, Is there a way to locate why the "tcp reset" is actively being sent?

var server = http.createServer()
    .on('clientError', handleClientError)
    .on('request', handleRequest)
    .on('connect', handleConnect)
    .on('upgrade', handleUpgrade)
    .on('connection', function (socket) {
        socket.on('error', () => { });
        socket.setNoDelay();
        socket.setKeepAlive(true, 600000);
    });
server.timeout = 120 * 1000;
server.keepAliveTimeout = 120 * 100;
server.on('timeout', (socket) => {
    socket.destroy(new Error('timeout'));
});
server.listen(8080, '0.0.0.0', 1023, function () {
    console.log('Started Server At: ' + bindIP + ':' + port);
});

select a connection:

How often does it reproduce? Is there a required condition?

No response

What is the expected behavior?

Every reset should have its reason.

What do you see instead?

by systemtap , i found the reset is send at net/ipv4/tcp.c:2054"

tcp_v4_send_reset: kernel.statement("tcp_v4_rcv@net/ipv4/tcp_ipv4.c:1764")
tcp_v4_send_reset: kernel.statement("tcp_v4_rcv@net/ipv4/tcp_ipv4.c:1764")
tcp_send_active_reset: kernel.statement("tcp_close@net/ipv4/tcp.c:2054")
tcp_send_active_reset: kernel.statement("tcp_close@net/ipv4/tcp.c:2054")
tcp_send_active_reset: kernel.statement("tcp_close@net/ipv4/tcp.c:2054")
tcp_send_active_reset: kernel.statement("tcp_close@net/ipv4/tcp.c:2054")
tcp_send_active_reset: kernel.statement("tcp_close@net/ipv4/tcp.c:2054")
tcp_send_active_reset: kernel.statement("tcp_close@net/ipv4/tcp.c:2054")
tcp_send_active_reset: kernel.statement("tcp_close@net/ipv4/tcp.c:2054")
tcp_send_active_reset: kernel.statement("tcp_close@net/ipv4/tcp.c:2054")
tcp_send_active_reset: kernel.statement("tcp_close@net/ipv4/tcp.c:2054")
tcp_v4_send_reset: kernel.statement("tcp_v4_rcv@net/ipv4/tcp_ipv4.c:1764")
tcp_v4_send_reset: kernel.statement("tcp_v4_rcv@net/ipv4/tcp_ipv4.c:1764")
tcp_send_active_reset: kernel.statement("tcp_close@net/ipv4/tcp.c:2054")
tcp_send_active_reset: kernel.statement("tcp_close@net/ipv4/tcp.c:2054")
tcp_send_active_reset: kernel.statement("tcp_close@net/ipv4/tcp.c:2054")
tcp_send_active_reset: kernel.statement("tcp_close@net/ipv4/tcp.c:2054")
tcp_send_active_reset: kernel.statement("tcp_close@net/ipv4/tcp.c:2054")
tcp_send_active_reset: kernel.statement("tcp_close@net/ipv4/tcp.c:2054")
tcp_send_active_reset: kernel.statement("tcp_close@net/ipv4/tcp.c:2054")
tcp_send_active_reset: kernel.statement("tcp_close@net/ipv4/tcp.c:2054")
tcp_send_active_reset: kernel.statement("tcp_close@net/ipv4/tcp.c:2054")
tcp_v4_send_reset: kernel.statement("tcp_v4_rcv@net/ipv4/tcp_ipv4.c:1764")
tcp_send_active_reset: kernel.statement("tcp_close@net/ipv4/tcp.c:2054")
tcp_v4_send_reset: kernel.statement("tcp_v4_rcv@net/ipv4/tcp_ipv4.c:1764")
tcp_send_active_reset: kernel.statement("tcp_close@net/ipv4/tcp.c:2054")
tcp_send_active_reset: kernel.statement("tcp_close@net/ipv4/tcp.c:2054")
tcp_send_active_reset: kernel.statement("tcp_close@net/ipv4/tcp.c:2054")
tcp_send_active_reset: kernel.statement("tcp_close@net/ipv4/tcp.c:2054")
tcp_send_active_reset: kernel.statement("tcp_close@net/ipv4/tcp.c:2054")
tcp_v4_send_reset: kernel.statement("tcp_v4_rcv@net/ipv4/tcp_ipv4.c:1764")
tcp_v4_send_reset: kernel.statement("tcp_v4_rcv@net/ipv4/tcp_ipv4.c:1764")
tcp_send_active_reset: kernel.statement("tcp_close@net/ipv4/tcp.c:2054")
tcp_send_active_reset: kernel.statement("tcp_close@net/ipv4/tcp.c:2054")
tcp_send_active_reset: kernel.statement("tcp_close@net/ipv4/tcp.c:2054")
tcp_v4_send_reset: kernel.statement("tcp_v4_rcv@net/ipv4/tcp_ipv4.c:1764")
tcp_v4_send_reset: kernel.statement("tcp_v4_rcv@net/ipv4/tcp_ipv4.c:1764")
tcp_send_active_reset: kernel.statement("tcp_close@net/ipv4/tcp.c:2054")
tcp_send_active_reset: kernel.statement("tcp_close@net/ipv4/tcp.c:2054")
tcp_send_active_reset: kernel.statement("tcp_close@net/ipv4/tcp.c:2054")
tcp_send_active_reset: kernel.statement("tcp_close@net/ipv4/tcp.c:2054")
tcp_send_active_reset: kernel.statement("tcp_close@net/ipv4/tcp.c:2054")
tcp_send_active_reset: kernel.statement("tcp_close@net/ipv4/tcp.c:2054")
tcp_v4_send_reset: kernel.statement("tcp_v4_rcv@net/ipv4/tcp_ipv4.c:1764")
tcp_v4_send_reset: kernel.statement("tcp_v4_rcv@net/ipv4/tcp_ipv4.c:1764")

the kernel code

if (unlikely(tcp_sk(sk)->repair)) {
		sk->sk_prot->disconnect(sk, 0);
	} else if (data_was_unread) {
		/* Unread data was tossed, zap the connection. */
		NET_INC_STATS_USER(sock_net(sk), LINUX_MIB_TCPABORTONCLOSE);
		tcp_set_state(sk, TCP_CLOSE);
		tcp_send_active_reset(sk, sk->sk_allocation);   // this line
	} else if (sock_flag(sk, SOCK_LINGER) && !sk->sk_lingertime) {
		/* Check zero linger _after_ checking for unread data. */
		sk->sk_prot->disconnect(sk, 0);
		NET_INC_STATS_USER(sock_net(sk), LINUX_MIB_TCPABORTONDATA);
	} else if (tcp_close_state(sk)) {

the code is closely：https://elixir.bootlin.com/linux/v3.10/source/net/ipv4/tcp.c#L2099.

Additional information

my stap script

probe kernel.statement("*@net/ipv4/tcp.c:2054") {
   if (execname() == "node") {
        printf("tcp_send_active_reset: %s\n", pp())
        //print_backtrace()
    }
}

[bnoordhuis]

Converted to a discussion as this is not a bug or feature request.

There are many possible reasons for the behavior you're seeing. Perhaps the client sends a RST, perhaps there's a MiTM TCP proxy in your network that sends the RST, etc.

[Jamlee]

it is impossible that other program send reset, because i filted by program name if (execname() == "node")

probe kernel.statement("*@net/ipv4/tcp.c:2054") {
   if (execname() == "node") {
        printf("tcp_send_active_reset: %s\n", pp())
        //print_backtrace()
    }
}