test_sharded_ingest timeouts during pageserver shutdown

[jcsp]

Example:
https://neon-github-public-dev.s3.amazonaws.com/reports/pr-9678/11802652055/index.html#suites/c4b5b5cc329a950d1fae768ab6cbdaf5/6f2d1e56efd29eba

The backtrace in test code indicates that it's setting immediate=True which should be a SIGKILL, but the pageserver logs look like a graceful shutdown and it reports getting a SIGTERM:

2024-11-12T18:53:55.701009Z  INFO Got signal SIGTERM. Terminating gracefully in fast shutdown mode

This test might be exposing a bug in how we do shutdowns in test teardown?

[erikgrinaker]

It works just fine manually at least:

$ kill -s QUIT 45673
$ tail .neon/pageserver_1/pageserver.log
2024-12-02T15:48:40.018876Z  INFO Got signal SIGQUIT. Terminating in immediate shutdown mode

$ cargo neon pageserver stop --id 1 -m immediate
Stopping pageserver with pid 46608 immediately...
pageserver stopped
$ tail .neon/pageserver_1/pageserver.log
2024-12-02T15:50:05.741718Z  INFO Got signal SIGQUIT. Terminating in immediate shutdown mode

I'll dig a bit further.

[erikgrinaker]

It's trying to stop it twice -- first with SIGTERM, then with SIGQUIT. This won't work, because there's a common signal handler for SIGTERM and SIGQUIT, and the SIGTERM got there first so it's busy with that.

The test runner may expect it to be responsive to both. I'll check.

2024-11-12 18:54:05.712 INFO [neon_cli.py:137] Run ['/tmp/neon/bin/neon_local', 'pageserver', 'stop', '--id=1'] failed:
  stdout:
    Stopping pageserver with pid 1441 gracefully.......
    pageserver has not stopped yet, continuing to wait.....
  stderr:
    pageserver stop failed: pageserver with pid 1441 did not stop in 10s seconds

2024-11-12 18:54:05.713 INFO [neon_cli.py:72] Running command "/tmp/neon/bin/neon_local storage_broker stop"
2024-11-12 18:54:05.820 INFO [neon_cli.py:72] Running command "/tmp/neon/bin/neon_local pageserver stop --id=1 -m immediate"
2024-11-12 18:54:15.841 INFO [neon_cli.py:137] Run ['/tmp/neon/bin/neon_local', 'pageserver', 'stop', '--id=1', '-m', 'immediate'] failed:
  stdout:
    Stopping pageserver with pid 1441 immediately.......
    pageserver has not stopped yet, continuing to wait.....
  stderr:
    pageserver stop failed: pageserver with pid 1441 did not stop in 10s seconds

[erikgrinaker]

Ok, so NeonEnv.stop() first does a graceful SIGTERM stop of pageservers, then an immediate SIGQUIT stop if the first one fails. I'll fix the signal handler to still be responsive to SIGQUIT after receiving a SIGTERM.

neon/test_runner/fixtures/neon_fixtures.py

Lines 1238 to 1246 in 5330122

	try:
	pageserver.stop(immediate=immediate)
	except RuntimeError:
	stop_later.append(pageserver)
	self.broker.stop()
	# TODO: for nice logging we need python 3.11 ExceptionGroup
	for ps in stop_later:
	ps.stop(immediate=True)