agent should remember state of checks across restart

[blalor]

When upgrading my consul agents from 0.5.0 to 0.5.2, I noticed that TTL checks that were previously passing are now showing as critical. This causes a massive disruption in the force our monitoring system, especially since we're exposing ~1000 services on some hosts. I understand the reasoning behind defaulting the state of a check to critical, but I think it'd be reasonable to have the agent use the last-known state for a check when restarting. For a TTL check, this may just be using the last-known state if the agent restarts before (a multiple of?) the TTL has expired.

[blalor]

This is especially important when a check may have a long TTL (like in the 10s of minutes).

[ryanuber]

Hey @blalor, this sounds like it could be a regression. I'll crack into this one very soon. Thanks for reporting it!

[ryanuber]

@blalor I remembered wrong, we were never restoring the previous status up to this point. I've started a branch for this. I think it will need to be optional and off by default so as to not disrupt clusters which depend on the current functionality. I'll update here when I'm closer to having something usable.

[blalor]

That's great, thank you. I'm ok with opting-in, although I'm not sure that's the behavior I'd expect by default. :-)

[ryanuber]

@blalor the above merged PR should fix the TTL checks and restore their status on later agent start. No config option needed for this case since it is only the TTL checks that get this new behavior.

[blalor]

Excellent! Thanks!