feat: allow citgm job comparison #455
Conversation
Codecov Report
@@ Coverage Diff @@
## master #455 +/- ##
==========================================
- Coverage 76.66% 76.41% -0.26%
==========================================
Files 21 27 +6
Lines 1603 1717 +114
==========================================
+ Hits 1229 1312 +83
- Misses 374 405 +31
Continue to review full report at Codecov.
|
codebytere
force-pushed
the
allow-citgm-comparison
branch
2 times, most recently
from
dd48ae2
to
3cd93b9
Compare
|
Just realized that we might want to compare both citgm + citgm-nobuild jobs. Is that possible with this implementation |
|
@MylesBorins it does not at present - this pulls
I'll make an issue and add that as a followup TODO, if that sounds good to you! |
|
Also - for more effective reviewing, just d22d5e1 is the actual new change introduced in this PR. |
codebytere
force-pushed
the
allow-citgm-comparison
branch
2 times, most recently
from
8b99bba
to
1fc9c06
Compare
|
Trying to make some sense of this: This shows, for example, 'torrent-stream-v1.2.0' for |
|
@richardlau spotted the error - fix incoming! |
|
@richardlau you should see: node-core-utils on git:allow-citgm-comparison ❯ ncu-ci citgm 2421 2417
--------------------------------------------------------------------------------
[1/1] Running CITGM: 2421
--------------------------------------------------------------------------------
✔ Summary data downloaded
✔ Results data downloaded
✔ Summary data downloaded
✔ Results data downloaded
----------------------------------- Summary ------------------------------------
Result FAILURE
URL https://ci.nodejs.org/job/citgm-smoker/2421/
Source https://api.github.com/repos/nodejs/node/git/refs/heads/v10.x
Commit [0bd44b4a88b7] Working on v10.21.1
Date 2020-06-02 18:57:33 +0100
Author Bethany.Griggs <Bethany.Griggs@uk.ibm.com>
----------------------------------- Summary ------------------------------------
Result FAILURE
URL https://ci.nodejs.org/job/citgm-smoker/2417/
Source https://api.github.com/repos/nodejs/node/git/refs/heads/v10.22.0-proposal
Commit [c5215d0b3da0] 2020-07-21, Version 10.22.0 'Dubnium' (LTS)
Date 2020-07-15 11:27:18 -0400
Author Richard Lau <riclau@uk.ibm.com>
----------------------------------- Results ------------------------------------
FAILURE: 9 failures in 2417 not present in 2421
┌────────────────────────┬─────────────────────┬──────────────────┐
│ (index) │ 0 │ 1 │
├────────────────────────┼─────────────────────┼──────────────────┤
│ aix71-ppc64 │ 'ember-cli-v3.19.0' │ │
│ ubuntu1804-64 │ │ │
│ fedora-last-latest-x64 │ 'spawn-wrap-v2.0.0' │ 'winston-v3.3.3' │
│ osx1014 │ 'fastify-v3.0.3' │ │
│ ubuntu1404-64 │ │ │
│ rhel7-s390x │ 'uglify-js-v3.10.0' │ │
│ osx1015 │ │ │
│ centos7-ppcle │ 'ember-cli-v3.19.0' │ 'multer-v1.4.2' │
│ ubuntu1604-64 │ 'clinic-v6.0.2' │ 'yargs-v15.4.1' │
│ debian9-64 │ │ │
│ fedora-latest-x64 │ │ │
└────────────────────────┴─────────────────────┴──────────────────┘now which makes far more sense! |
codebytere
force-pushed
the
allow-citgm-comparison
branch
2 times, most recently
from
843b259
to
0dbe375
Compare
codebytere
force-pushed
the
allow-citgm-comparison
branch
from
0ec2b1f
to
91f107e
Compare
|
At a cursory glance it seems like the Could we potentially also allow a small grammar here, rather than a flag... perhaps e.g.
thoughts? |
|
@MylesBorins the way i've written it at present is that for comparison jobs, if so if one passed |
|
@codebytere as long as it is documented and possible that seems like a good way to move forward... and we could revisit later if we want to improve the dx |
Follow-up to #454.
Closes #352.
Extends
ncu-ci cigtm <jobid>toncu-cu cigtm <jobid> [jobid2], allowing for direct comparisons between two CITGM runs to see if new failures were introduced.I also did a bit of a refactor here, since there were a million classes in the result parser file and it was getting wildly unwieldy. I tried to do the minimum necessary to make this work but I plan to follow it up with the remainder of the the class split-out work.
Display:
Sample JSON Output
[ { "baseBuild": { "source": "https://api.github.com/repos/nodejs/node/git/refs/heads/v12.x", "upstream": "https://ci.nodejs.org/job/citgm-smoker/2390/" }, "comparisonBuild": { "source": "https://github.com/nodejs/node/pull/33811/", "upstream": "https://ci.nodejs.org/job/citgm-smoker/2392/" }, "debian9-64": [ { "name": "express-session-v1.17.1", "status": "FAILED" }, { "name": "yeoman-generator-v4.10.1", "status": "FAILED" } ], "centos7-ppcle": [ { "name": "ember-cli-v3.18.0", "status": "FAILED" }, { "name": "multer-v1.4.2", "status": "FAILED" }, { "name": "torrent-stream-v1.2.0", "status": "FAILED" } ], "rhel7-s390x": [ { "name": "torrent-stream-v1.2.0", "status": "FAILED" } ], "fedora-latest-x64": [ { "name": "spawn-wrap-v2.0.0", "status": "FAILED" } ], "ubuntu1804-64": [], "fedora-last-latest-x64": [], "ubuntu1604-64": [] } ]Sample Markdown Output
cc @richardlau (since i can't request your review)