Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

doc: initial cut at support tiers for diag tools #21870

Closed
wants to merge 9 commits into from
Closed

doc: initial cut at support tiers for diag tools #21870

Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
19ee126
doc: initial cut at support tiers for diag tools
mhdawson Jul 18, 2018
345d17f
squash: partially address first set of comments
mhdawson Aug 7, 2018
57bef97
squash: address more comments
mhdawson Aug 7, 2018
19120b2
squash: address additional comments.
mhdawson Aug 7, 2018
c31bf8a
squash: address additional comments
mhdawson Aug 7, 2018
ee6b81d
squash: address rest of comments
mhdawson Aug 7, 2018
bb4ec69
squash: address comments
mhdawson Aug 8, 2018
1e46bd3
squash: fix linting issues from comments
mhdawson Aug 9, 2018
11bd549
squash: address comments
mhdawson Aug 10, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
147 changes: 147 additions & 0 deletions doc/guides/diagnostic-tooling-support-tiers.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,147 @@
# Diagnostic Tooling Support Tiers

Diagnostic tooling is important to the consumers of Node.js. It is used both
in development and in production in order to investigate problems. The failure
of one of these tools may be as big a problem for an end user as a bug within
the runtime itself.

The Node.js project has assessed the tools and the APIs which support those
tools. Each of the tools and APIs has been put into one of
the following tiers.

* Tier 1 - Must always be working(CI tests passing) for all
Current and LTS Node.js releases. A release will not be shipped if the test
suite for the tool/API is not green. To be considered for inclusion
in this tier it must have a good test suite and that test suite and a job
must exist in the Node.js CI so that it can be run as part of the release
process. Tests on master will be run nightly when possible to provide
early warning of potential issues. No commit to the current and LTS
release branches should break this tool/API if the next major release
is within 1 month. In addition:
* The maintainers of the tool must remain responsive when there
are problems;
* The tool must be actively used by the ecosystem;
* The tool must be heavily depended on;
* The tool must have a guide or other documentation in the Node.js GitHub
organization or website;
* The tool must be working on all supported platforms;
* The tool must only be using APIs exposed by Nodejs as opposed to
its dependencies; and
* The tool must be open source.

* Tier 2 - Must be working(CI tests passing) for all
LTS releases. An LTS release will not be shipped if the test
suite for the tool/API is not green. To be considered for inclusion
in this tier it must have a good test suite and that test suite and a job
must exist in the Node.js CI so that it can be run as part of the release
process. In addition:
* The maintainers of the tool must remain responsive when
there are problems;
* The tool must be actively used by the ecosystem;
* The tool must be heavily depended on;
* The tool must have a guide or other documentation in the Node.js GitHub
organization or website;
* The tool must be open source.

* Tier 3 - If possible its test suite
will be run at least nightly in the Node.js CI and issues opened for
failures. Does not block shipping a release.

* Tier 4 - Does not block shipping a release.

* Unclassified - tool/API is new or does not have the required testing in the
Node.js CI in order to qualify for a higher tier.

The choice of which tier a particular tool will be assigned to, will be a
collaborative decision between Diagnostics WG and Release WG. Some of the
criteria considered might be:

* If the tool fits into a key category as listed below.
* Whether the tool is actively used by the ecosystem.
* The availability of alternatives.
* Impact to the overall ecosystem if the tool is not working.
* The availability of reliable test suite that can be integrated into our CI.
* The availability of maintainer or community collaborator who will help
resolve issues when there are CI failures.
* If the tool is maintained by the Node.js Foundation GitHub organization.

The current categories of tools/APIs that fall under these Tiers are:

* FFDC (F) - First failure data capture, easy to consume initial diagnostic
information.
* Tracing (T) - use of logging to provide information about execution flow.
* Memory (M) - tools that provide additional information about memory
used in the Heap or by native code.
* Profiling (P) - tools that provide additional information about where
CPU cycles are being spent.
* AsyncFlow (A) - tools that provide additional insight into asynchronous
execution flow.

## Adding a tool to this list

Any tool that might be used to investigate issues when running Node.js can
be added to the list. If there is a new tool that should be added to the
list, it should start by being added to the "Not yet classified" or
"Tier 4" lists. Once it has been added to the list "promotion" to Tier 3
through Tier 1 requires that the requirements listed above be met AND
have agreement from Diagnostics WG and Release WG based on the criteria
listed above.

## Tiers

The tools are currently assigned to Tiers as follows:

## Tier 1

| Tool Type | Tool/API Name | Regular Testing in Node.js CI | Integrated with Node.js | Target Tier |
|-----------|---------------------------|-------------------------------|-------------------------|-------------|
| | | | | |

## Tier 2

| Tool Type | Tool/API Name | Regular Testing in Node.js CI | Integrated with Node.js | Target Tier |
|-----------|---------------------------|-------------------------------|-------------------------|-------------|
| | | | | |


## Tier 3

| Tool Type | Tool/API Name | Regular Testing in Node.js CI | Integrated with Node.js | Target Tier |
|-----------|---------------------------|-------------------------------|-------------------------|-------------|
| | | | | |

## Tier 4

| Tool Type | Tool/API Name | Regular Testing in Node.js CI | Integrated with Node.js | Target Tier |
|-----------|---------------------------|-------------------------------|-------------------------|-------------|
| | | | | |

## Not yet classified

| Tool Type | Tool/API Name | Regular Testing in Node.js CI | Integrated with Node.js | Target Tier |
|-----------|---------------------------|-------------------------------|-------------------------|-------------|
| FFDC | node-report | No | No | 1 |
| Memory | mdb_V8 | No | No | 4 |
| Memory | node-heapdump | No | No | 2 |
| Memory | V8 heap profiler | No | Yes | 1 |
| Memory | V8 sampling heap profiler | No | Yes | 1 |
| AsyncFlow | Async Hooks (API) | ? | Yes | 1 |
| Debugger | V8 Debug protocol (API) | No | Yes | 1 |
| Debugger | Command line Debug Client | ? | Yes | 1 |
| Debugger | llnode | ? | No | 2 |
| Debugger | Chrome Dev tools | ? | No | 3 |
| Debugger | Chakracore - time-travel | No | Data source only | too early |
| Tracing | trace_events (API) | No | Yes | 1 |
| Tracing | DTrace | No | Partial | 3 |
| Tracing | LTTng | No | Removed? | N/A |
| Tracing | ETW | No | Partial | 3 |
| Tracing | Systemtap | No | Partial | ? |
| Profiling | V8 CPU profiler (--prof) | No | Yes | 1 |
| Profiling | Linux perf | No | Partial | ? |
| Profiling | DTrace | No | Partial | 3 |
| Profiling | Windows Xperf | No | ? | ? |
| Profiling | 0x | No | No | 4 |
| Profiling | node-clinic | No | No | to early |
| F/P/T | appmetrics | No | No | ? |
| M/T | eBPF tracing tool | No | No | ? |