Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

logging: Attempt to recover logmon failures #5577

Merged
merged 2 commits into from
Apr 22, 2019

Commits on Apr 18, 2019

  1. loggging: Attempt to recover logmon failures

    Currently, when logmon fails to reattach, we will retry reattachment to
    the same pid until the task restart specification is exhausted.
    
    Because we cannot clear hook state during error conditions, it is not
    possible for us to signal to a future restart that it _shouldn't_
    attempt to reattach to the plugin.
    
    Here we revert to explicitly detecting reattachment seperately from a
    launch of a new logmon, so we can recover from scenarios where a logmon
    plugin has failed.
    
    This is a net improvement over the current hard failure situation, as it
    means in the most common case (the pid has gone away), we can recover.
    
    Other reattachment failure modes where the plugin may still be running
    could potentially cause a duplicate process, or a subsequent failure to launch
    a new plugin.
    
    If there was a duplicate process, it could potentially cause duplicate
    logging. This is better than a production workload outage.
    
    If there was a subsequent failure to launch a new plugin, it would fail
    in the same (retry until restarts are exhausted) as the current failure
    mode.
    endocrimes committed Apr 18, 2019
    Configuration menu
    Copy the full SHA
    269e2c0 View commit details
    Browse the repository at this point in the history

Commits on Apr 22, 2019

  1. tweak logging level for failed log line

    Co-Authored-By: notnoop <mahmood@notnoop.com>
    schmichael and notnoop committed Apr 22, 2019
    Configuration menu
    Copy the full SHA
    0f91277 View commit details
    Browse the repository at this point in the history