Improve kube deploy process.

[lmossman]

What

Resolves #13144

As the issue linked above describes, if a user tries to perform a rolling update of a kube deployment of airbyte, they may run into multiple issues: it may throw an error saying that the bootloader pod cannot be edited, and if a new db pod starts up it could cause the underlying db to be permanently broken. Even if users follow our upgrade instructions in our docs, i.e. in a non-rolling fashion, they could still run into both issues depending on how quickly they try to execute the commands.

This PR attempts to fix both issues

How

I tried a few strategies to fix the bootloader problem:

• I tried using generateName instead of name so that the bootloader pod would always have a unique name. This didn't work because kubectl apply cannot be used on a kube resource that does not have a name field, and kubectl apply is what our docs currently instruct users to use and may be important in the future for rolling deploys.
• I tried switching the bootloader to be of kind Deployment instead of Pod. This was bad because it caused the bootloader to be ran repeatedly; Deployment is not the right resource type to use for a one-time process.
• I tried switching the bootloader to be of kind Job instead of Pod. This still had the issue where running kubectl apply with some changes to env variables would throw this error:

  The Job "airbyte-bootloader" is invalid: spec.template: Invalid value: core.PodTemplateSpec{ObjectMeta:v1.ObjectMeta{Name:"", GenerateName:"", Namespace:"", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{"controller-uid":"96239008-b40f-432c-bcdc-fb258208b81e", "job-name":"airbyte-bootloader"}, Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:core.PodSpec{Volumes:[]core.Volume(nil), InitContainers:[]core.Container(nil), Containers:[]core.Container{core.Container{Name:"airbyte-bootloader-container", Image:"airbyte/bootloader:dev", Command:[]string(nil), Args:[]string(nil), WorkingDir:"", Ports:[]core.ContainerPort(nil), EnvFrom:[]core.EnvFromSource(nil), Env:[]core.EnvVar{core.EnvVar{Name:"AIRBYTE_VERSION", Value:"", ValueFrom:(*core.EnvVarSource)(0xc00cca1e60)}, core.EnvVar{Name:"DATABASE_HOST", Value:"", ValueFrom:(*core.EnvVarSource)(0xc00cca1e80)}, core.EnvVar{Name:"DATABASE_PORT", Value:"", ValueFrom:(*core.EnvVarSource)(0xc00cca1ea0)}, core.EnvVar{Name:"DATABASE_PASSWORD", Value:"", ValueFrom:(*core.EnvVarSource)(0xc00cca1ec0)}, core.EnvVar{Name:"DATABASE_URL", Value:"", ValueFrom:(*core.EnvVarSource)(0xc00cca1ee0)}, core.EnvVar{Name:"DATABASE_USER", Value:"", ValueFrom:(*core.EnvVarSource)(0xc00cca1f00)}}, Resources:core.ResourceRequirements{Limits:core.ResourceList(nil), Requests:core.ResourceList(nil)}, VolumeMounts:[]core.VolumeMount(nil), VolumeDevices:[]core.VolumeDevice(nil), LivenessProbe:(*core.Probe)(nil), ReadinessProbe:(*core.Probe)(nil), StartupProbe:(*core.Probe)(nil), Lifecycle:(*core.Lifecycle)(nil), TerminationMessagePath:"/dev/termination-log", TerminationMessagePolicy:"File", ImagePullPolicy:"IfNotPresent", SecurityContext:(*core.SecurityContext)(nil), Stdin:false, StdinOnce:false, TTY:false}}, EphemeralContainers:[]core.EphemeralContainer(nil), RestartPolicy:"Never", TerminationGracePeriodSeconds:(*int64)(0xc00889bb38), ActiveDeadlineSeconds:(*int64)(nil), DNSPolicy:"ClusterFirst", NodeSelector:map[string]string(nil), ServiceAccountName:"", AutomountServiceAccountToken:(*bool)(nil), NodeName:"", SecurityContext:(*core.PodSecurityContext)(0xc00c4cc700), ImagePullSecrets:[]core.LocalObjectReference(nil), Hostname:"", Subdomain:"", SetHostnameAsFQDN:(*bool)(nil), Affinity:(*core.Affinity)(nil), SchedulerName:"default-scheduler", Tolerations:[]core.Toleration(nil), HostAliases:[]core.HostAlias(nil), PriorityClassName:"", Priority:(*int32)(nil), PreemptionPolicy:(*core.PreemptionPolicy)(nil), DNSConfig:(*core.PodDNSConfig)(nil), ReadinessGates:[]core.PodReadinessGate(nil), RuntimeClassName:(*string)(nil), Overhead:core.ResourceList(nil), EnableServiceLinks:(*bool)(nil), TopologySpreadConstraints:[]core.TopologySpreadConstraint(nil)}}: field is immutable
  

• I tried adding a ttlSecondsAfterFinished to the bootloader job, so that the bootloader pod would be automatically deleted after it completes. This fixed the above issue and allowed me to use kubectl apply to freely switch between stable and dev (as long as I waited for the bootloader pod to be automatically deleted).

For the db pod issue, adding the Recreate strategy to the db deployment manifest seems to have fully fixed the issue, as it causes kube to first terminate the existing db pod before spinning up a new one.

Recommended reading order

any

[lmossman]

As mentioned in the PR description, this is necessary to avoid an error being about the airbyte-bootloader job being immutable. This isn't an ideal solution, because this means that the bootloader pod is deleted after it completes, making its logs inaccessible through kube.

This was the only solution I could come up with that allowed us to still use kubectl apply without issue though, so it may be a worthwhile tradeoff.

Definitely open to feedback here if there are any other options I haven't considered.

[cgardens]

can you add a comment explaining why we do this, please?

[lmossman]

Sure thing, done!

[cgardens]

nice!

[davinchia]

Kube jobs only have best effort parallelism guarantees, which is why I don't really like using them for crucial workflows. Took a look and confirmed this is probably the best way of doing this with Kustomize. Can we add a comment here that we generally want to use Pod (our Helm charts use Pod) for the best exactly-once execution guarantees and cannot do so because Kustomize does not support generateName? Want to prevent confusion in the future.

If Kustomize did support generateName, we should be able to instruct users to run kubectl create on initial create and replace on subsequent runs.

This happens relatively infrequently so risk is low. In the long term, I think we'll consolidate the Kube deploys into Helm so I think this is fine for now.

[davinchia]

I appreciate the thoroughness and the detailed PR description. One note to explain why we are using a job here. Otherwise looks good!