Support number of threads in each state #2164

wenlxie · 2021-10-08T09:17:03Z

Node exporter support node_processes_state metrics to expose number of processes in each state, but this is not enough for the whole system. This PR is to support number of threads in each state.

Signed-off-by: wenlxie <wenlxie@ebay.com>

discordianfish · 2021-10-08T09:26:41Z

Thanks for the PR! But first we need to decide whether we want to include these days. This is increasing cardinality a lot. @SuperQ wdyt?

Beside that, we should try to iterate only once about AllProcs.

wenlxie · 2021-10-09T00:52:10Z

@discordianfish Thanks for the reply. From feature perspective, just have node_processes_state metrics is really not enough. Thread in D or R states means lots to system load, so it is helpful when user want to check for why the system load is high. I'd think this should be supported by node_exporter.

SuperQ · 2021-10-09T03:59:13Z

This might be better suited to the process-exporter.

SuperQ · 2021-10-09T04:01:10Z

@discordianfish I don't think this changes the cardinality much. This is also not a default collector.

discordianfish · 2021-10-11T10:38:49Z

@SuperQ Oh I see, it's one dimension per state not thread. Id' be fine to include this but don't feel strongly either way.

wenlxie · 2021-10-12T01:06:59Z

@discordianfish @SuperQ
Thanks for the reply.

We use node_exporter to monitor k8s nodes. There is one scenario that we want to monitor the D state tasks in the node, because large number of D state tasks always indicates that the node may have issues. But node_processes_state metrics can't be used to do this. So I think this PR is for common requirements.
When check for node's states, user can't just care about the processes state only but ignore the threads states in most scenarios. Because threads states also impact the host as processes state.

SuperQ · 2021-10-12T02:46:07Z

We use node_exporter to monitor k8s nodes. There is one scenario that we want to monitor the D state tasks in the node, because large number of D state tasks always indicates that the node may have issues.

This is not a great assumption, and has the same issues as using "load average" for any real monitoring. Side effect gauges are prone to produce bad signals.

You might consider looking at the IO data from the pressure collector instead. See the kernel docs.

discordianfish · 2021-10-18T10:37:44Z

@SuperQ So beside the question whether this is useful for this particular scenario. Is this something we want to include?

SuperQ · 2021-10-18T12:30:55Z

Yes, I think it's reasonable.

SuperQ

LGTM

discordianfish

LGTM

NOTE: In order to support globs in the textfile collector path, filenames exposed by `node_textfile_mtime_seconds` now contain the full path name. * [CHANGE] Add path label to rapl collector #2146 * [FEATURE] Add support for monitoring GPUs on Linux #1998 * [FEATURE] Add os release collector #2094 * [FEATURE] Add netdev.address-info collector #2105 * [ENHANCEMENT] Support glob textfile collector directories #1985 * [ENHANCEMENT] ethtool: Expose node_ethtool_info metric #2080 * [ENHANCEMENT] Use include/exclude flags for ethtool filtering #2165 * [ENHANCEMENT] Add flag to disable guest CPU metrics #2123 * [ENHANCEMENT] Add threads metrics to processes collector #2164 * [BUGFIX] ethtool: Sanitize metric names #2093 * [BUGFIX] Fix ethtool collector for multiple interfaces #2126 * [BUGFIX] Fix possible panic on macOS #2133 * [BUGFIX] Collect flag_info and bug_info only for one core #2156 Signed-off-by: Ben Kochie <superq@gmail.com>

wenlxie · 2021-10-20T01:43:27Z

@SuperQ @discordianfish thanks

NOTE: In order to support globs in the textfile collector path, filenames exposed by `node_textfile_mtime_seconds` now contain the full path name. * [CHANGE] Add path label to rapl collector #2146 * [CHANGE] Exclude filesystems under /run/credentials #2157 * [FEATURE] Add support for monitoring GPUs on Linux #1998 * [FEATURE] Add Darwin thermal collector #2032 * [FEATURE] Add os release collector #2094 * [FEATURE] Add netdev.address-info collector #2105 * [ENHANCEMENT] Support glob textfile collector directories #1985 * [ENHANCEMENT] ethtool: Expose node_ethtool_info metric #2080 * [ENHANCEMENT] Use include/exclude flags for ethtool filtering #2165 * [ENHANCEMENT] Add flag to disable guest CPU metrics #2123 * [ENHANCEMENT] Add DMI collector #2131 * [ENHANCEMENT] Add threads metrics to processes collector #2164 * [ENHANCMMENT] Reduce timer GC delays in the Linux filesystem collector #2169 * [BUGFIX] ethtool: Sanitize metric names #2093 * [BUGFIX] Fix ethtool collector for multiple interfaces #2126 * [BUGFIX] Fix possible panic on macOS #2133 * [BUGFIX] Collect flag_info and bug_info only for one core #2156 Signed-off-by: Ben Kochie <superq@gmail.com>

NOTE: In order to support globs in the textfile collector path, filenames exposed by `node_textfile_mtime_seconds` now contain the full path name. * [CHANGE] Add path label to rapl collector #2146 * [CHANGE] Exclude filesystems under /run/credentials #2157 * [FEATURE] Add darwin powersupply collector #1777 * [FEATURE] Add support for monitoring GPUs on Linux #1998 * [FEATURE] Add Darwin thermal collector #2032 * [FEATURE] Add os release collector #2094 * [FEATURE] Add netdev.address-info collector #2105 * [ENHANCEMENT] Support glob textfile collector directories #1985 * [ENHANCEMENT] ethtool: Expose node_ethtool_info metric #2080 * [ENHANCEMENT] Use include/exclude flags for ethtool filtering #2165 * [ENHANCEMENT] Add flag to disable guest CPU metrics #2123 * [ENHANCEMENT] Add DMI collector #2131 * [ENHANCEMENT] Add threads metrics to processes collector #2164 * [ENHANCMMENT] Reduce timer GC delays in the Linux filesystem collector #2169 * [BUGFIX] ethtool: Sanitize metric names #2093 * [BUGFIX] Fix ethtool collector for multiple interfaces #2126 * [BUGFIX] Fix possible panic on macOS #2133 * [BUGFIX] Collect flag_info and bug_info only for one core #2156 Signed-off-by: Ben Kochie <superq@gmail.com>

NOTE: In order to support globs in the textfile collector path, filenames exposed by `node_textfile_mtime_seconds` now contain the full path name. * [CHANGE] Add path label to rapl collector #2146 * [CHANGE] Exclude filesystems under /run/credentials #2157 * [FEATURE] Add lnstat collector for metrics from /proc/net/stat/ #1771 * [FEATURE] Add darwin powersupply collector #1777 * [FEATURE] Add support for monitoring GPUs on Linux #1998 * [FEATURE] Add Darwin thermal collector #2032 * [FEATURE] Add os release collector #2094 * [FEATURE] Add netdev.address-info collector #2105 * [ENHANCEMENT] Support glob textfile collector directories #1985 * [ENHANCEMENT] ethtool: Expose node_ethtool_info metric #2080 * [ENHANCEMENT] Use include/exclude flags for ethtool filtering #2165 * [ENHANCEMENT] Add flag to disable guest CPU metrics #2123 * [ENHANCEMENT] Add DMI collector #2131 * [ENHANCEMENT] Add threads metrics to processes collector #2164 * [ENHANCMMENT] Reduce timer GC delays in the Linux filesystem collector #2169 * [BUGFIX] ethtool: Sanitize metric names #2093 * [BUGFIX] Fix ethtool collector for multiple interfaces #2126 * [BUGFIX] Fix possible panic on macOS #2133 * [BUGFIX] Collect flag_info and bug_info only for one core #2156 Signed-off-by: Ben Kochie <superq@gmail.com>

NOTE: In order to support globs in the textfile collector path, filenames exposed by `node_textfile_mtime_seconds` now contain the full path name. * [CHANGE] Add path label to rapl collector prometheus#2146 * [CHANGE] Exclude filesystems under /run/credentials prometheus#2157 * [FEATURE] Add lnstat collector for metrics from /proc/net/stat/ prometheus#1771 * [FEATURE] Add darwin powersupply collector prometheus#1777 * [FEATURE] Add support for monitoring GPUs on Linux prometheus#1998 * [FEATURE] Add Darwin thermal collector prometheus#2032 * [FEATURE] Add os release collector prometheus#2094 * [FEATURE] Add netdev.address-info collector prometheus#2105 * [ENHANCEMENT] Support glob textfile collector directories prometheus#1985 * [ENHANCEMENT] ethtool: Expose node_ethtool_info metric prometheus#2080 * [ENHANCEMENT] Use include/exclude flags for ethtool filtering prometheus#2165 * [ENHANCEMENT] Add flag to disable guest CPU metrics prometheus#2123 * [ENHANCEMENT] Add DMI collector prometheus#2131 * [ENHANCEMENT] Add threads metrics to processes collector prometheus#2164 * [ENHANCMMENT] Reduce timer GC delays in the Linux filesystem collector prometheus#2169 * [BUGFIX] ethtool: Sanitize metric names prometheus#2093 * [BUGFIX] Fix ethtool collector for multiple interfaces prometheus#2126 * [BUGFIX] Fix possible panic on macOS prometheus#2133 * [BUGFIX] Collect flag_info and bug_info only for one core prometheus#2156 Signed-off-by: Ben Kochie <superq@gmail.com>

support thread state

7a08b57

Signed-off-by: wenlxie <wenlxie@ebay.com>

wenlxie force-pushed the upstream.master.supportthreadstate branch from 47eaaaa to 7a08b57 Compare October 8, 2021 09:20

SuperQ approved these changes Oct 18, 2021

View reviewed changes

discordianfish approved these changes Oct 19, 2021

View reviewed changes

discordianfish merged commit 11bb2f8 into prometheus:master Oct 19, 2021

SuperQ mentioned this pull request Oct 27, 2021

Release 1.3.0 #2168

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support number of threads in each state #2164

Support number of threads in each state #2164

wenlxie commented Oct 8, 2021

discordianfish commented Oct 8, 2021

wenlxie commented Oct 9, 2021 •

edited

Loading

SuperQ commented Oct 9, 2021

SuperQ commented Oct 9, 2021

discordianfish commented Oct 11, 2021

wenlxie commented Oct 12, 2021

SuperQ commented Oct 12, 2021

discordianfish commented Oct 18, 2021

SuperQ commented Oct 18, 2021

SuperQ left a comment

discordianfish left a comment

wenlxie commented Oct 20, 2021

Support number of threads in each state #2164

Support number of threads in each state #2164

Conversation

wenlxie commented Oct 8, 2021

discordianfish commented Oct 8, 2021

wenlxie commented Oct 9, 2021 • edited Loading

SuperQ commented Oct 9, 2021

SuperQ commented Oct 9, 2021

discordianfish commented Oct 11, 2021

wenlxie commented Oct 12, 2021

SuperQ commented Oct 12, 2021

discordianfish commented Oct 18, 2021

SuperQ commented Oct 18, 2021

SuperQ left a comment

Choose a reason for hiding this comment

discordianfish left a comment

Choose a reason for hiding this comment

wenlxie commented Oct 20, 2021

wenlxie commented Oct 9, 2021 •

edited

Loading