Decide deployment services check

[EDsCODE]

**Initial draft changes so discussion can be done here

Description

based on https://posthog.slack.com/archives/C05034MPYA0/p1679596107221869, the decide pods still rely postgres because of the startup checks. This means that if postgres is down, decide pods will be down and error out regardless of caching

Discussion points

• Possibly, split up wait checks (postgres // rest of the services) so there's not repeated check scripts for each deployment
• If possible, I think the decide deployment should wait briefly for postgres and migrations and timeout after X amount of time. Not sure the exact mechanics here but we don't want decide deployment to never connect to postgres/pgbouncer.
• Main point is that postgres isn't mission critical for decide deployment to operate as it can use a populated redis cache so we should not have the deployment be blocked by postgres being up

[neilkakkar]

Don't need Clickhouse either?

[neilkakkar]

If possible, I think the decide deployment should wait briefly for postgres and migrations and timeout after X amount of time. Not sure the exact mechanics here but we don't want decide deployment to never connect to postgres/pgbouncer.

This just checks the current status of services to determine if the pod can come up, so it doesn't mean we'll never connect to postgres/pgbouncer.

I think instead we'd want something like an "OR" check between redis and postgres? If either is up, we're good to go.

[neilkakkar]

split up wait checks (postgres // rest of the services) so there's not repeated check scripts for each deployment

Good idea, not clear enough about things here, would want @PostHog/team-infra to give their input 😅

[tiina303]

Related: looked at the readiness and startup and liveness probes:

1. readiness & startup are both _readyz?role=decide (here), which doesn't check PG, but does require cache (here), not sure we'd want to (Redis down PG available situations)?
2. liveliness check is here - just verifying service is fine 👍

[hazzadous]

Decide still makes queries to PostgreSQL right? So we still need to wait for them to have been applied?

[tiina303]

I think I might have misunderstood earlier & mislead the ff team ... this would only apply on new container setup, not on restart? If that's true and considering we do rolling deploys this is fine to keep as is. Sorry.

[hazzadous]

Yeah if you don't include the migration check, you'll end up with a SELECT all, the, fields, including, ones, that, don't, exist, yet FROM posthog_persons which will error and if we're lucky fall back to redis, or just straight error out. Either way I don't think that's desired.

[frankh]

just confirmed that init containers only run when the pod is created

pods never restart, only the containers within them

[tiina303]

Ok, then we should investigate, why FF went down, when PG was unavailable (as this was the initial hypothesis iirc and based on what Frank said it's not the case)