deployments should block scale operations

[tgross]

In an internal discussion with @schmichael and @cgbaker, we realized that autoscaling can cause some unexpected behaviors when combined with deployments. When a Job.Scale RPC comes in, a new deployment is created and that marks any deployments in flight as DeploymentStatusCancelled. If you're a user running a deployment, you probably don't expect that the autoscaler is going to come along in the middle of things and stop you. This is going to be particularly bad when we consider the metrics an APM source might get for applications that have expensive startups or have to rebalance connections.

This issue is to add a check to Job.Scale that verifies that the most recent deployment for a job is not active (there's already an IsActive() method that correctly checks terminal states).

[cgbaker]

@tgross, do you think that we should create a ScalingEvent for a blocked deployment? can always add this later.

[tgross]

Those are informational, right? It's probably a good idea to add one if we do it for successful/failed deployments today.

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.