Epic: Improve sk<->ps connection observability

[petuhovskiy]

Motivation

During one of the deploys we saw some projects were stuck for several minutes. There were errors like this in the logs:

query handler for 'basebackup X Y 0/1EF9A10 --gzip' failed: Timed out while waiting for WAL record at LSN 0/1EF9A10 to arrive, last_record_lsn 0/14EEA60 disk consistent LSN=0/14EEA60, WalReceiver status: Not active

I tried to find something relevant in the logs and metrics, but they were mostly empty without any hints.

DoD

I think we should add more context in the logs, more metrics and print broker status in the logs.

Implementation ideas

• Initialize manager_status right after WalReceiver creation
• Don't write WalReceiver status: Not active, instead write the timestamp of the last message received from the broker
• Add global metrics about the broker to the pageserver (is the pageserver connected to broker, when was the last message received from the broker)
• When pageserver disconnects from the safekeeper, safekeeper should log first and last LSN that was sent/received
• Safekeeper should log hostname/NodeId of the connected pageserver in the logs

Tasks

Preview Give feedback

No tasks being tracked yet.

Links

• stuck project thread https://neondb.slack.com/archives/C03F5SM1N02/p1709230475014849