Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a "CI" mode for versioned tests #2122

Closed
jsumners-nr opened this issue Apr 4, 2024 · 1 comment · Fixed by #2145
Closed

Add a "CI" mode for versioned tests #2122

jsumners-nr opened this issue Apr 4, 2024 · 1 comment · Fixed by #2145
Assignees
Labels
points: 5 1-2 weeks

Comments

@jsumners-nr
Copy link
Contributor

jsumners-nr commented Apr 4, 2024

Description

The versioned test runner currently outputs a lot of text, making it very difficult to review errors during CI runs. See https://github.com/newrelic/node-newrelic/actions/runs/8542101178/job/23404883851#step:6:1 for an example. Basically, it writes out a status report for every completed child test suite for every parent test suite that looks like:

Running versioned tests output
   amqplib
 • bluebird/regressions.tap.js run 1 of 28): waiting
   bunyan
   cassandra-driver
   cjs-in-esm
   cls
 • connect/error-intercept.tap.js run 1 of 20): waiting
   director
   elastic
   esm-package
 • express/app-use.tap.js run 1 of 170): installing
   express-esm
 • fastify/add-hook.tap.js run 1 of 130): installing
   generic-pool
 • grpc/client-unary.tap.js run 1 of 56): waiting
   grpc-esm
 • hapi/capture-params.tap.js run 1 of 70): waiting
   ioredis
   langchain
 • mongodb/bulk.tap.js run 1 of 84): waiting
 • mongodb-esm/bulk.tap.mjs run 1 of 70): waiting
 • mysql/basic-pool.tap.js run 1 of 40): waiting
 • mysql2/basic-pool.tap.js run 1 of 50): waiting
   nestjs
 • openai/chat-completions.tap.js run 1 of 30): waiting
 • pg/force-native.tap.js run 1 of 66): waiting
 • pg-esm/force-native.tap.mjs run 1 of 66): waiting
   pino
   prisma
   redis
 • restify-post-7/capture-params.tap.js run 1 of 70): waiting
 • restify-pre-7/capture-params.tap.js run 1 of 42): waiting
   undici
   winston
 • winston-esm/winston.tap.mjs run 1 of 80): waiting

Notice that there are 84 possible tests for mongodb/bulk.tapjs. That means there will be at least 84 of those blocks printed to the test run output for that one suite.

The versioned test runner currently supports a -p <mode> switch, where mode can be either simple or pretty and defaults to pretty (which is what our CI is printing). We could switch our CI runs to use the "simple" mode and it would greatly improve things, but in this mode the test runner still outputs a lot of extraneous text. In simple mode, it prints a . character in place of the above quoted pretty status table. While this would make things easier, I think we can do better.

We should be able to add a "quiet" mode that will not print any status, except for bubbling up tap errors for failed tests, until such time as the full versioned test suite has finished. At that point, the tool should print its normal output that looks like:

Final versioned tests output
 ✖ cassandra-driver/query.tap.js run 1 of 2): failure
 ✓ generic-pool (2) 3s
 ✓ express (34) 1m
 ✓ grpc-esm (2) 5s
 ✓ grpc (16) 1m
 ✓ mysql2 (15) 1m
 ✓ pino (2) 6s
 ✓ cls (1) 2s
 ✓ cjs-in-esm (1) 3s
 ✓ director (1) 2s
   langchain
 ✓ esm-package (1) 1s
 ✓ openai (3) 19s
 ✓ winston-esm (1) 2s
 ✓ winston (1) 3s
 ✓ restify-post-7 (35) 1m
 ✓ prisma (1) 8s
 ✓ undici (2) 15s
 ✓ mongodb (35) 2m
 ✓ nestjs (3) 1m
Versions executed

Folder: amqplib
         * amqplib(1): 0.10.3
Folder: bluebird
         * bluebird(2): 2.11.0, 3.7.2
Folder: bunyan
         * bunyan(2): 1.8.15, 2.0.5
Folder: cassandra-driver
         * cassandra-driver(2): 3.6.0, 4.7.2
Folder: cjs-in-esm
         * express(1): 4.18.2
         * swagger-ui-express(1): 5.0.0
         * axios(1): 1.4.0
Folder: cls
         * continuation-local-storage(1): 3.2.1
         * async-listener(1): 0.6.10
Folder: connect
         * connect(2): 2.30.2, 3.7.0
Folder: director
         * director(1): 1.2.8
         * express(1): 4.16
Folder: elastic
         * @elastic/elasticsearch(3): 7.13.0, 7.17.13, 8.13.0
         * @elastic/transport(1): 8.4.1
Folder: esm-package
         * parse-json(1): 6.0.2
Folder: express-esm
         * express(2): 4.18.2, 4.19.2
         * express-enrouten(1): 1.1
         * ejs(1): 2.5.9
Folder: express
         * express(2): 4.18.2, 4.19.2
         * express-enrouten(1): 1.1
         * ejs(1): 2.5.9
Folder: fastify
         * fastify(3): 2.15.3, 3.29.5, 4.26.2
         * middie(1): 5.3.0
Folder: generic-pool
         * generic-pool(2): 3.9.0, 2.5.4
Folder: grpc-esm
         * @grpc/grpc-js(1): 1.10.6
Folder: grpc
         * @grpc/grpc-js(2): >=1.4.0 <1.8.0, 1.10.6
Folder: hapi
         * ejs(1): 2.5.5
         * @hapi/hapi(2): 20.3.0, 21.3.8
         * @hapi/vision(1): 7.0.3
Folder: ioredis
         * ioredis(3): 3.2.2, 4.28.5, 5.3.2
Folder: langchain(SKIPPED)
Folder: mongodb-esm
         * mongodb(3): 2.2.36, 3.7.4, 4.17.2
Folder: mongodb
         * mongodb(5): 2.2.36, 3.7.4, 4.17.2, 5.9.2, 6.5.0
Folder: mysql
         * mysql(1): 2.18.1
         * generic-pool(1): 2.4
Folder: mysql2
         * mysql2(3): 1.7.0, 2.3.3, 3.9.3
         * generic-pool(1): 2.4 <2.5
Folder: nestjs
         * @nestjs/cli(3): 8.2.8, 9.5.0, 10.3.2
Folder: openai
         * openai(1): 4.32.1
Folder: pg-esm
         * pg(2): >=8.2 <8.8, 8.11.5
         * pg-native(2): 2.2.0, 3.0.1
Folder: pg
         * pg(2): >=8.2 <8.8, 8.11.5
         * pg-native(2): 2.2.0, 3.0.1
Folder: pino
         * pino(2): 7.11.0, 8.19.0
         * flush-write-stream(1): 2.0.0
         * split2(1): 4.1.0
Folder: prisma
         * @prisma/client(1): 5.12.0
         * prisma(1): 5.12.0
Folder: redis
         * redis(6): >=2.0.0 < 2.3.0, >=2.3.0 < 2.4.0, >=2.4.0 < 2.6.0, 2.8.0, 3.1.2, 4.6.13
Folder: restify-post-7
         * restify(5): 7.7.0, 8.6.1, 9.1.0, 10.0.0, 11.1.0
         * express(1): 4.16
         * restify-errors(1): 6.1.1
Folder: restify-pre-7
         * restify(2): 5.2.1, 6.4.0
         * express(1): 4.16
         * restify-errors(1): 6.1.1
Folder: undici
         * undici(2): 4.16.0, 5.28.4
Folder: winston-esm
         * winston(1): 3.13.0
         * winston-transport(1): 4.7.0
Folder: winston
         * winston(1): 3.13.0
===============================================================
===============================================================
FAIL (3)
   packages: pg@>=8.2 <8.8, pg-native@2.2.0 file: /Users/jsumners/Projects/team-repos/node-newrelic/test/versioned/pg-esm/force-native.tap.mjs
   packages: pg@>=8.2 <8.8, pg-native@2.2.0 file: /Users/jsumners/Projects/team-repos/node-newrelic/test/versioned/pg/force-native.tap.js
   packages: cassandra-driver@3.6.0 file: /Users/jsumners/Projects/team-repos/node-newrelic/test/versioned/cassandra-driver/query.tap.js

Acceptance Criteria

  1. Update the versioned tests runner to support a "quiet" output mode
  2. Update CI in this repo to utilize the new "quiet" output mode
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
points: 5 1-2 weeks
Projects
Archived in project
Development

Successfully merging a pull request may close this issue.

2 participants