Skip to content

Commit

Permalink
docs: improve ncu-ci docs (nodejs#453)
Browse files Browse the repository at this point in the history 
* docs: improve ncu-ci docs

* Fixup malformed ncu-ci walk <type> section
  • Loading branch information
@codebytere
codebytere authored Jul 4, 2020
1 parent be9e3ff commit 158efdf
Showing 1 changed file with 170 additions and 17 deletions.
187 changes: 170 additions & 17 deletions docs/ncu-ci.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -30,47 +30,200 @@ Options:
--help Show help [boolean]
```

## Example
### `ncu-ci rate <type>`

Get the CI results of PR 12345 (including latest results of each type of
supported CI) and copy the summaries into clipboard:
`ncu-ci rate <type>` calculate the success rate for CI jobs in the last 100 runs per [CI Health History](https://github.com/nodejs/reliability#ci-health-history), where `<type>` can be either `pr` for `node-test-pull-request` or `commit` for `node-test-commit`.

```
ncu-ci url https://github.com/nodejs/node/pull/12345 --copy
```
Examples:

Get the results of job #12345 of `node-test-pull-request`:
```sh
node on git:master ❯ ncu-ci rate pr
--------------------------------------------------------------------------------
[1/1] Running health
--------------------------------------------------------------------------------
✔ Done
| UTC Time | RUNNING | SUCCESS | UNSTABLE | ABORTED | FAILURE | Green Rate |
| ---------------- | ------- | ------- | -------- | ------- | ------- | ---------- |
| 2020-07-03 16:28 | 1 | 7 | 34 | 10 | 48 | 7.87% |
```

```sh
node on git:master ❯ ncu-ci rate commit
--------------------------------------------------------------------------------
[1/1] Running health
--------------------------------------------------------------------------------
✔ Done
| UTC Time | RUNNING | SUCCESS | UNSTABLE | ABORTED | FAILURE | Green Rate |
| ---------------- | ------- | ------- | -------- | ------- | ------- | ---------- |
| 2020-07-03 16:28 | 2 | 8 | 27 | 5 | 58 | 8.60% |
```
ncu-ci pr 12345

### `ncu-ci walk <type>`

`ncu-ci walk <type>` walks CI and displays failures, where `<type>` can be either `pr` for `node-test-pull-request` or `commit` for `node-test-commit`.

Example
```sh
node on git:master ❯ ncu-ci walk commit
✔ Done--------------------------------------------------------------------------------
[1/60] Running health
--------------------------------------------------------------------------------
| UTC Time | RUNNING | SUCCESS | UNSTABLE | ABORTED | FAILURE | Green Rate |
| ---------------- | ------- | ------- | -------- | ------- | ------- | ---------- |
| 2020-07-03 16:50 | 1 | 8 | 27 | 5 | 59 | 8.51% |

--------------------------------------------------------------------------------
[2/60] Running https://ci.nodejs.org/job/node-test-commit/39446/
--------------------------------------------------------------------------------
✔ Build data downloaded
✔ Data downloaded
----------------------------------- Summary ------------------------------------
Result FAILURE
URL https://ci.nodejs.org/job/node-test-commit/39446/
Source https://api.github.com/repos/libuv/ci-tmp-libuv-node/git/refs/heads/jenkins-libuv-in-node-141
Commit [b205f29c5b4d] Replace libuv version with refs/heads/v1.x from libuv/libuv
Date 2020-07-03 15:44:21 +0000
Author ci <ci@iojs.org>
-------------------------------- freebsd11-x64 ---------------------------------
URL https://ci.nodejs.org/job/node-test-commit-freebsd/nodes=freebsd11-x64/34327/console
Type JENKINS_FAILURE
Built On test-digitalocean-freebsd11-x64-2
Reason
Build timed out (after 6 minutes). Marking the build as failed.
--------------------------------- Other builds ---------------------------------
Unstable https://ci.nodejs.org/job/node-test-commit-linux/35929/
Unstable https://ci.nodejs.org/job/node-test-commit-arm-fanned/15212/
--------------------------------------------------------------------------------
[3/60] Running https://ci.nodejs.org/job/node-test-commit/39444/
...etc
```

Possible use cases:

Walk the CI for the latest 100 runs of `node-test-pull-request`,
aggregate the failures, then write the results into a Markdown file,
and cache the responses from Jenkins so that the next time the command
1. Walk CI for the latest 100 runs of `node-test-pull-request`,
aggregate failures, write the results into a Markdown file,
and then cache the responses from Jenkins so that the next time the command
is run, it picks up cached data written on disk for jobs whose results
are known.

Note: results are cached in `${ncu_intallation_path}/.ncu/cache`, you
Note: results are cached in `${ncu_intallation_path}/.ncu/cache`, so you
may want to clean it up from time to time.

```
ncu-ci walk pr --stats --cache --markdown results.md
```

Walk the CI for the latest 100 runs of `node-test-pull-request`, write the
failures into a JSON file
2. Walk CI for the latest 100 runs of `node-test-pull-request`, and then write the
failures into a JSON file named database.json.

```
ncu-ci walk pr --json database.json
```

Calculate the green rate of the CI for
[CI Health History](https://github.com/nodejs/reliability#ci-health-history)
### `ncu-ci pr <jobid>`

`ncu-ci pr <jobid>` returns information about the results of a `node-test-pull-request` job.

For example, if you would like to see the results of `node-test-pull-request` for https://github.com/nodejs/node/pull/34127, you would visit the PR and note that the `node-test-pull-request` job can be found at https://ci.nodejs.org/job/node-test-pull-request/32158, and therefore the `<jobid>` is `32158`, not `34127`.

Example:
```sh
node on git:master ❯ ncu-ci pr 32158
--------------------------------------------------------------------------------
[1/1] Running PR: 32158
--------------------------------------------------------------------------------
✔ Build data downloaded
----------------------------------- Summary ------------------------------------
Result UNSTABLE
URL https://ci.nodejs.org/job/node-test-pull-request/32158/
Source https://github.com/nodejs/node/pull/34127/
Commit [faa808326cb0] lib: handle missing callbackMap in esm logic
Date 2020-06-29 16:51:58 -0700
Author Shelley Vohr <shelley.vohr@gmail.com>
--------------------------------- Other builds ---------------------------------
Unstable https://ci.nodejs.org/job/node-test-commit-arm-fanned/15155/
```

### `ncu-ci commit <jobid>`

`ncu-ci commit <jobid>` returns information about the results of a specified `node-test-commit` job.

For example, if you would like to see the results of `node-test-commit` for https://github.com/nodejs/node/pull/34086, you would visit the PR and note that the `node-test-commit` job can be found at https://ci.nodejs.org/job/node-test-commit/39377, and therefore the `<jobid>` is `39377`, not `34086`.

Example:
```sh
node on git:master ❯ ncu-ci commit 39377
--------------------------------------------------------------------------------
[1/1] Running COMMIT: 39377
--------------------------------------------------------------------------------
✔ Build data downloaded
----------------------------------- Summary ------------------------------------
Result UNSTABLE
URL https://ci.nodejs.org/job/node-test-commit/39377/
Source https://github.com/undefined/undefined/pull/34086/
Commit [725a1df7b849] quic: remove redundant cast
Date 2020-06-27 22:14:09 +0800
Author Jiawen Geng <technicalcute@gmail.com>
--------------------------------- Other builds ---------------------------------
Unstable https://ci.nodejs.org/job/node-test-commit-linux/35868/
Unstable https://ci.nodejs.org/job/node-test-commit-arm-fanned/15148/
```

### `ncu-ci url <url>`

`ncu-ci url <url>` takes a url corresponding to a PR on `nodejs/node`, detects the CI type (either `node-test-commit` or `node-test-pull-request`) and corresponding job id for the latest run, and returns a summary of results about the job run.

Example:
```sh
node on git:master ❯ ncu-ci url https://github.com/nodejs/node/pull/34127
--------------------------------------------------------------------------------
[1/1] Running PR: 32158
--------------------------------------------------------------------------------
✔ Build data downloaded
----------------------------------- Summary ------------------------------------
Result UNSTABLE
URL https://ci.nodejs.org/job/node-test-pull-request/32158/
Source https://github.com/nodejs/node/pull/34127/
Commit [faa808326cb0] lib: handle missing callbackMap in esm logic
Date 2020-06-29 16:51:58 -0700
Author Shelley Vohr <shelley.vohr@gmail.com>
--------------------------------- Other builds ---------------------------------
Unstable https://ci.nodejs.org/job/node-test-commit-arm-fanned/15155/
```
ncu-ci rate pr

### `ncu-ci benchmark <jobid>`

`ncu-ci benchmark <jobid>` displays the results of a specified `benchmark-node-micro-benchmarks` CI job.

Example:
```sh
node on git:master ❯ ncu-ci benchmark 636
--------------------------------------------------------------------------------
[1/1] Running BENCHMARK: 636
--------------------------------------------------------------------------------
✔ Data downloaded
confidence improvement accuracy (*) (**) (***)
http2/compat.js duration=5 benchmarker='test-double-http2' clients=2 streams=100 requests=100 NA NaN % NA NA NA
http2/compat.js duration=5 benchmarker='test-double-http2' clients=2 streams=10 requests=100 NA NaN % NA NA NA
http2/compat.js duration=5 benchmarker='test-double-http2' clients=2 streams=10 requests=1000 NA NaN % NA NA NA
http2/compat.js duration=5 benchmarker='test-double-http2' clients=2 streams=1 requests=100 NA NaN % NA NA NA
http2/compat.js duration=5 benchmarker='test-double-http2' clients=2 streams=1 requests=1000 NA NaN % NA NA NA
http2/compat.js duration=5 benchmarker='test-double-http2' clients=2 streams=200 requests=100 NA NaN % NA NA NA
http2/compat.js duration=5 benchmarker='test-double-http2' clients=2 streams=20 requests=100 NA NaN % NA NA NA
http2/compat.js duration=5 benchmarker='test-double-http2' clients=2 streams=20 requests=1000 NA NaN % NA NA NA
http2/compat.js duration=5 benchmarker='test-double-http2' clients=2 streams=40 requests=100 NA NaN % NA NA NA

Be aware that when doing many comparisons the risk of a false-positive
result increases. In this case there are 9 comparisons, you can thus
expect the following amount of false-positive results:
0.45 false positives, when considering a 5% risk acceptance (*, **, ***),
0.09 false positives, when considering a 1% risk acceptance (**, ***),
0.01 false positives, when considering a 0.1% risk acceptance (***)
++ mv output030720-170033.csv /w/bnch-comp
Collecting metadata...
Metadata collection done.
Notifying upstream projects of job completion
Finished: SUCCESS
```
## Caveats
Expand Down

0 comments on commit 158efdf

Please sign in to comment.