New variable property to help control when new versions will be used / deployed

[ystros]

Hi folks!

We are currently exploring ways to be better control updates to variables during BOSH deploys in order to facilitate continuous certificate rotation. We want a way to control the roll out of new, rotated certificate versions to times where a long deploy is already expected.

Currently, when a manifest is deployed, the BOSH Director will query CredHub for the latest version of each variable in the manifest. This ensures that deployments always use the current configuration stored in CredHub.

This is a problem for being able to automatically rotate certificates - if a new version of a certificate is in CredHub, the deploy will use it. Normally this would be desirable, but since the rotation is done without users taking a direct action, they may not be aware that there are updated certs and thus will be surprised by a longer-than-normal deploy if they have no other changes. Predicting when there's going to be a longer-than-normal deploy is important for strict change / maintenance windows.

We're thinking about adding a new key to variable declarations that will prevent BOSH from asking for the latest version unless a new stemcell is also being deployed. When deploying a new stemcell, you are already expecting a longer deploy time because the VMs must be recreated, making it a great time to roll out new certificates. The new key on variables should have two potential values:

• on-deploy - BOSH will use the current behavior of looking up the latest variable value on each deploy (default)
• on-stemcell-change - BOSH will only look up the latest variable value when a deploy is occurring with a new stemcell

Example:

variables:
- name: /some/certificate-authority
  type: certificate
  update:
    strategy: on-stemcell-change
  options:
    ...

Note: this would be distinct from the existing update_mode property, which controls when BOSH asks CredHub to regenerate a variable based on its options changing. i.e. using a new version already in CredHub vs generating new version using CredHub.

While our use case is for certificates, it could be applied to any type of variable where delaying rollout could be beneficial. Additional strategies could be added in the future (e.g. update variables on other manifest changes) if there is a use case / value.

We are currently exploring this idea and seeing what other implications the feature may have on BOSH.

[beyhan]

Sunds good to me. I just was wandering which convention for the new option we should follow. It looks like we have the convention of using underscore like update_mode or common_name. Is there a special reason why you decide for the current proposal over update_strategy?

[ystros]

@beyhan I believe the idea was to have it more consistent with the update blocks that appear in other places with BOSH. It would also allow more modifiers to be added to the update section in the future if needed rather than a top level key. As you point out, it is definitely inconsistent with the existing update_mode, so update_strategy might make more sense for the final key name.

[mvach]

Hmmm isn't on-stemcell-change only one of multiple strategies like on-deploy, on-cloud-config-change or on-vm-recreation? If so, then maybe the strategy property should be of type array so that one can register to multiple of theses events.

Something like:

update:
  strategy:
    - on-stemcell-change
    - on-vm-recreation

[rkoster]

so more like an update trigger instead of a strategy.

[selzoc]

I would say that this a possible future enhancement, but it's actually quite tricky to detect all of those scenarios in a reliable way.

[mvach]

@selzoc My intention was not to implement such scenarios now, but to provide a configuration that can be extended in future when others want to add more strategies. Therefore I suggested to use an array as parameter type.

@rkoster Yes, trigger is probably the better naming here since a strategy is usually something different.
And it looks expressive from my perspective:

variables:
- name: /some/certificate-authority
  type: certificate
  update:
    triggers:
    - stemcell-change
    - deployment
    - foo_bar  
  options:
    ...

[selzoc]

Is the idea "go fetch the latest version of this variable from the config server" if any of these triggers are true? With an implicit deployment in every place where the triggers array is empty?

As a thought experiment, what if there was a variable with the following configuration:

update:
  triggers:
  - vm-recreation

In the case of a regular deployment, you would not expect this variable to get updated. In the case of a stemcell roll-out, it would. In the case of a bosh recreate of an instance, you would expect it to get updated - even though it may then be inconsistent across instances?

I'm trying to tease out if it's worth the extra complexity of not having an enum-type value which are all mutually exclusive vs the (possible) extensibility of having an array where all the triggers could interact.

[mvach]

Hmm, good points ... may be it is not worth the extra complexity.

[max-soe]

And we should also invest in better output for this case. For example, if the deploy running without a stemcell-change and there are new credentials, we should print an information that we wait to apply this credentials as long as there is no new stemcell...

[selzoc]

There are currently task debug logs that show which version of each variable is being interpolated into templates. The plan is to expand those logs to indicate the strategy being used for on-stemcell-change variables (in addition to the default on-deploy variables)

[mvach]

I guess @max-soe point is something different. If certificates are rotated, then bosh is not displaying any diff in the console (like for manifest changes) and therefore people wonder why a deployment is taking that long because from their perspective nothing has changed and it should be quick. So it is more about creating transparency why the deployment is taking that long.

[selzoc]

I agree, having that information would be great. Practically, though, I think it would be out of the scope of this effort. The diff output is calculated in an entirely different place (a different endpoint even) than the place in the code where the decisions for how the changes in the diff affect the deployment.

[jpalermo]

VMs getting updated without any diff output has been an issue since the introduction of the config server since variable changes from there won’t show up in the diff.

it would be nice to solve somehow, but it does seem like a separate set of work since it is much bigger than the variables being changed as part of this work.

[ystros]

Thanks for the feedback so far! We've submitted an initial PR for the implemented feature here: #2460