post-run hook doesn't re-run on exit 1

[jamessewell]

Running hab from hab-0.22.1-20170509234454-x86_64-linux on CentOS 7.2, deploying into Docker.

It looks like when a unsuccessful code is encountered from the post-run hook script the script is not run again, like we see with init, run etc...

The use case for this would be:

1. Start the application in run
2. Perform some once off config which requires the application to be up in post-run

The two options I can see here are block in run until my service becomes available for config, or have the post-run script accept an exit 1 as requiring another post-run pass.

If this is the intended functionality ideas are welcome!

[srenatus]

I, too, have just expected it to be re-run, and was surprised it didn't. So, that's another teeny-tiny data point 😉

[jamessewell]

Hi, Yeah I've seen a few people put post-run functionality into health-check with some logic to only run once, not pretty. I've actually got a PR for this which does the following: - Don't run the post-run hook till there has been a successful health check - Re-run the post-run hook until success I'll clean it up and try to get it through today. Cheers, James

…

On Mon, Aug 14, 2017 at 10:31 PM, Stephan Renatus ***@***.***> wrote: I, too, have just expected it to be re-run, and was surprised it didn't. So, that's another teeny-tiny data point 😉 — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#2364 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABKRo0fvMACqD-TI-Syd8rFo30k-8oCqks5sYD4ngaJpZM4NajO0> .

[rsertelon]

@jamessewell any news on that PR? Is this something still relevant today? thanks!

[themightychris]

Relevant case: habitat-sh/core-plans#1674 (comment)

[jsirex]

This really must have.

Use-case:

1. I'm starting a cluster.
2. But habitat discovery is not happened yet (no members, they are starting)
3. post-run hook get executed but without any sense - cluster not ready
4. discovery happened
5. Service restarted
6. We are ready for the post-run hook but it will not be executed

Also it is not clear how much to wait in step 3? 1 second, 15 seconds, 8 minutes? Will service actually restart while post-run hook is running? Or hook gets killed? Or service will wait? Too many questions.

Immediate exit with non-zero code is better.

[christophermaier]

Taking some questions posed by @davidMcneil and putting them here (along with some thoughts) for posterity and broader discussion. As usual, all this stuff is open to discussion with anyone that has thoughts to contribute!

Should post-run only run after a successful health check?

I'd say "no" at this point, since it's conceivable (even likely?) that what is done in post-run could be required for a successful health check.

How many times should post-run be tried?

We don't currently have a way of restricting restarts of services, which makes me want something like Erlang's supervisor restart intensity configuration. Implementing something like that in a holistic way seems like a feature on its own, and adding ad-hoc partial throttling right now seems like it'd complicate things necessarily.

Should we use status codes to indicate if it should be rerun?

0: successful do not retry
1: failed retry
2: failed do not retry

That sounds like a great idea, with the observation that the "failed do not retry" case seems linked to the "how many times should we retry" question above.

Should there be a delay between tries or a backoff strategy?

Could be a good idea. It seems like we could ultimately model the post-run execution similar to how we do the health-check (i.e., run repeatedly, with a potentially configurable duration between runs, until we get the outcome we want). That provides a nice place to localize delay and backoff logic. If implemented properly, it would be easy to drop this in later, pending further product investigation (i.e., do we want to allow for these parameters to be customized on a per-service basis, or are they just hard-coded aspects of how the Supervisor operates?) (though @jsirex's comment above suggests that per-service customization could indeed be useful)

Should failure to run a lifecycle hook influence the result of a health check?

Possibly, though I think an argument can be made that if a given hook fails to do something important for the health of a service, then the health check hook should be explicitly checking for that.

Given that, I'm inclined to leave out any explicit connection between other hooks and health checking for right now. If we get user feedback to the contrary, we can certainly revisit it.

Should similar rules apply to other lifecycle hooks?

Probably so, but depending on the specifics of that hook. It would be interesting to see if we could eventually model all hooks as being customized implementations of some general "hook prototype" that has customization points for these behaviors you've mentioned.

[christophermaier]

Also, @jsirex's comments above:

Will service actually restart while post-run hook is running? Or hook gets killed? Or service will wait? Too many questions.

suggest that some kind of overall state machine for a service will be important in order to properly track these kinds of transitions.