Restart scheduler on error

[mkindahl]

If the scheduler receives an error, it will never restart again since bgw_restart_time is set to BGW_NEVER_RESTART, which will prevent all jobs from executing.

This commit adds the GUC timescaledb.bgw_scheduler_restart_time that can be set to the restart time for the scheduler. It defaults to 60 seconds, which is the default restart interval for background workers PostgreSQL defines.

It also adds timescaledb.debug_bgw_scheduler_exit_status to be able to shutdown the scheduler with a non-zero exit status, which allows the restart functionality to be tested.

It also ensures that backend_type is explicitly set up rather than copied from application_name and add some more information to application_name. It also updates the tests to use backend_type where applicable.

Disable-check: loader-change

[mkindahl]

Compared to #6195 this adds code to the start of the launcher that handles the status for any schedulers that existed prior to the restart by killing them. Reproducing it in a test is difficult because of how the schedulers are used in tests, but this short script demonstrates the issue:

create view tsdb_bgw as
       select datname, pid, backend_type, application_name from pg_stat_activity
       where backend_type like '%TimescaleDB%';

alter system set timescaledb.debug_bgw_scheduler_exit_status to 1;
alter system set timescaledb.bgw_scheduler_restart_time to 10;
select pg_reload_conf();

-- Kill the schedulers
select pid, pg_terminate_backend(pid) from pg_stat_activity where backend_type like '%Scheduler%';
select pg_sleep(1);
select datname, backend_type, count(*) from tsdb_bgw group by datname, backend_type;

-- Wait for the schedulers to restart.
select pg_sleep(10);
select datname, backend_type, count(*) from tsdb_bgw group by datname, backend_type;

-- Kill the launcher, it should restart immediately.
select pid, pg_terminate_backend(pid) from pg_stat_activity where backend_type like '%Launcher%';
select pg_sleep(1);

-- This should only show one scheduler for each database.
select datname, backend_type, count(*) from tsdb_bgw group by datname, backend_type;

[codecov]

Codecov Report

Attention: Patch coverage is 57.57576% with 14 lines in your changes missing coverage. Please review.

Project coverage is 81.76%. Comparing base (59f50f2) to head (bb093c2).
Report is 662 commits behind head on main.

Files with missing lines	Patch %	Lines
src/loader/bgw_launcher.c	50.00%	12 Missing and 2 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #7527      +/-   ##
==========================================
+ Coverage   80.06%   81.76%   +1.69%     
==========================================
  Files         190      229      +39     
  Lines       37181    43072    +5891     
  Branches     9450    10844    +1394     
==========================================
+ Hits        29770    35216    +5446     
- Misses       2997     3457     +460     
+ Partials     4414     4399      -15     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.