Idea: transform intent into runnable behavior

[jahkeup]

Yeah, I think this is the best place to start improvements (alongside other coordination improvements). I really liked the idea you proposed of using Behavior Trees and really would like to see what an inspired implementation looks like. I think some hybrid that produced predefined Actions (with concrete types behind them) would be slick and enable some testing of responses other than of the subsequent mutations of the annotations.

I don't think we'll be able to get away from the 3 variables - for many reasons as we discussed: we need to be able to observe pending/active actions taken (for ACKs and monitoring) and also to be able to discern between current (or reached state) and the desired next state. We may be able to collapse these into 2 annotations thought that may severely overload them, I'm open to the discussion and changing of these annotations. I'm interested to hear some alternative ideas we could put in place here.

Originally posted by @jahkeup in bottlerocket-os/bottlerocket#239

[jahkeup]

@patraw can you post your awesome prototype you showed me here?

[patraw]

https://gist.github.com/patraw/4ee5e7f7c5c10b0757c7307ca8524cd0

There is a comment in that gist explaining how to run it and how it works. We can leave this as a possible avenue to pursue in the future. I got it to only use two variables, and I haven't been able to think of any cases where this would lead to an ambiguous/erroneous situation.

[cbgbt]

I think we've ended up solving some of the problems discussed here by using the BottlerocketShadow custom resources in 0.2.0 release.

Intents are defined via the spec of the Shadow, and the current state is encoded in the status. We model node-level changes by thinking of the brupop agent as a single-node control loop which drives the status towards the spec, and the cluster level controller as modifying specs to drive the overarching cluster to the desired state.

For more details, check out the design doc.