Crashed supervisor leader initially starts as leader without elections

[jsirex]

I have 3 supervisors running 3 postgresql servers in leader topology:

• database-01
• database-02
• database-03

After all servers started leader is elected. I also have suitability hooks.
Let database-01 is leader with suitability: 2
Let database-02 and database-03 are followers with suitability: 1

steps to reproduce

1. Stop leader supervisor -> new election is triggered
2. Wait and see that hook[suitability] is triggered and new leader is elected
3. Start old leader supervisor

expected result (updated)

1. Supervisor by default starts like follower
2. Hooks rendered only once before service init/start

actual result

Hard to say what I'm seeing. Looks like:

1. Supervisor immediately start all services and reports "I'm the leader" from cache
2. Some hooks re-rendered and init hook get executed
3. No suitability hook were triggered at all (probably b/o Evaluate what conditions should automatically trigger a service election #7486)
4. During init hook supervisor receives rumor and figured out he is not leader
5. Hooks re-rendered and one more restart is happened
6. Finally supervisor reports he is not leader

[jsirex]

I tried add sleep 5 in the begging of the run hook and actual results is slightly different:

1. Supervisor immediately start all services and reports "I'm the leader" from cache
2. Some hooks re-rendered and init hook executed
3. Suitability hook ONLY on database-01 is triggered and "2" is reported
4. During init hook supervisor receives rumor and figured out he is not leader
5. Hooks re-rendered and one more restart is happened
6. Finally supervisor reports he is not leader

[christophermaier]

Interesting... seems like we might want to privilege incoming rumors over cached rumors in some cases 🤔

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. We value your input and contribution. Please leave a comment if this issue still affects you.

[stale]

This issue has been automatically closed after being stale for 400 days. We still value your input and contribution. Please re-open the issue if desired and leave a comment with details.