Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

client: Return empty values when host stats fail #6349

Merged
merged 3 commits into from
Nov 20, 2019
Merged

Commits on Sep 18, 2019

  1. client: Return empty values when host stats fail

    Currently, there is an issue when running on Windows whereby under some
    circumstances the Windows stats API's will begin to return errors (such
    as internal timeouts) when a client is under high load, and potentially
    other forms of resource contention / system states (and other unknown
    cases).
    
    When an error occurs during this collection, we then short circuit
    further metrics emission from the client until the next interval.
    
    This can be problematic if it happens for a sustained number of
    intervals, as our metrics aggregator will begin to age out older
    metrics, and we will eventually stop emitting various types of metrics
    including `nomad.client.unallocated.*` metrics.
    
    However, when metrics collection fails on Linux, gopsutil will in many cases
    (e.g cpu.Times) silently return 0 values, rather than an error.
    
    Here, we switch to returning empty metrics in these failures, and
    logging the error at the source. This brings the behaviour into line
    with Linux/Unix platforms, and although making aggregation a little
    sadder on intermittent failures, will result in more desireable overall
    behaviour of keeping metrics available for further investigation if
    things look unusual.
    endocrimes committed Sep 18, 2019
    Configuration menu
    Copy the full SHA
    5b183e5 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    c8ba938 View commit details
    Browse the repository at this point in the history

Commits on Sep 19, 2019

  1. Configuration menu
    Copy the full SHA
    4ba87cc View commit details
    Browse the repository at this point in the history