"Credential Not Found" Error When the GKE Cluster Restarted

[bfg-jian]

Our spinnaker instance in GKE has been experiencing a "Credential Not Found" error whenever the GKE Cluster got restarted. The restart can be due to gcp's auto maintenance, upgrade, or manual restarting. When we hit the "Credential Not Found" error, all the accounts we created seem to disappeared.

I found that this is caused by clouddriver service start in the wrong order when the GKE cluster restarted. Currently, to get around this problem, I would manually scale the clouddriver pod to 0 then back to 1. Is there a better way to resolve this? Or is there a way to enforce the order in which the services get started?