Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

add smoke tests #2959

Merged
merged 1 commit into from
Mar 25, 2021
Merged

add smoke tests #2959

merged 1 commit into from
Mar 25, 2021

Conversation

ruyadorno
Copy link
Contributor

@ruyadorno ruyadorno commented Mar 25, 2021 •
edited
Loading

Lightweight smoke test suite that runs common commands so that we can
also have a holistic check during CI.

Overview

smoke-tests/server.js

Copy-pasted mock server from arborist

smoke-tests/content/

Package/manifest fixtures for usage along with smoke-tests/server.js

smoke-tests/index.js

Serialized, snapshot-based smoke tests that run various npm cli commands asserting the changes in state after each step.

Times

  • Average of 4s in my local, intel-based macbook pro
  • Average of 32s in GH actions ubuntu latest
  • Average of 1min in GH actions windows latest
  • Average of 50s in GH actions macos latest

@ruyadorno ruyadorno requested a review from a team as a code owner March 25, 2021 04:48
@ruyadorno ruyadorno force-pushed the add-smoke-tests branch 10 times, most recently from cead93b to 248602b Compare March 25, 2021 05:50
@ruyadorno ruyadorno added Release 7.x work is associated with a specific npm 7 release release: next These items should be addressed in the next release labels Mar 25, 2021
@ruyadorno ruyadorno added this to the OSS - Sprint 27 milestone Mar 25, 2021
@ruyadorno
Copy link
Contributor Author

ruyadorno commented Mar 25, 2021 •
edited
Loading

NOTE: added more info in the PR description providing context on how it works.

That said, I have two items to follow up with that may or not be addressed before merging:

  • remove the [10.1, 12.1, 14.1] targets from CI (since they're likely to be very redundant in the context of smoke tests as unit tests will catch these oddities much better)
  • add smoke-tests as a step to be run during the checks phase of make publish

@ljharb
Copy link
Contributor

ljharb commented Mar 25, 2021

@ruyadorno as long as they’re tested then it’s fine, but smoke tests are typically slow and flaky, whereas unit tests are fast and reliable, so it seems useful to keep both.

@ruyadorno
add smoke tests
97b4152 
Lightweight smoke test suite that runs common commands so that we can
also have a holistic check during CI.

PR-URL: npm#2959
Credit: @ruyadorno
Close: npm#2959
Reviewed-by: @nlf
@ruyadorno ruyadorno changed the base branch from latest to release-next March 25, 2021 21:41
@ruyadorno ruyadorno merged commit 97b4152 into npm:release-next Mar 25, 2021
This was referenced Mar 25, 2021
Merged
Closed
feelepxyz added a commit to dependabot/dependabot-core that referenced this pull request Apr 16, 2021
@feelepxyz
Update npm from 7.7.4 to 7.10.0
a6d3c7f 
Changes since 7.7.4: https://github.com/npm/cli/blob/latest/CHANGELOG.md

## v7.10.0 (2021-04-15)

### FEATURES

* [`f9b639eb6`](npm/cli@f9b639e)
  [#3052](npm/cli#3052)
  feat(bugs): fall back to email if provided
  ([@Yash-Singh1](https://github.com/Yash-Singh1))
* [`8c9e24778`](npm/cli@8c9e247)
  [#3055](npm/cli#3055)
  feat(version): add workspace support
  ([@wraithgar](https://github.com/wraithgar))

### DEPENDENCIES

* [`f1e6743a6`](npm/cli@f1e6743)
  `libnpmversion@1.2.0`
    * feat(retrieve-tag): retrieve unannotated git tags
    * fix(retrieve-tag): use semver to look for semver
* [`3b476a24c`](npm/cli@3b476a2)
  `@npmcl/git@2.0.8`
    * fix(git): do not use shell when calling git
* [`dfcd0c1e2`](npm/cli@dfcd0c1)
  [#3069](npm/cli#3069)
  `tap@15.0.2`

### DOCUMENTATION

* [`90b61eda9`](npm/cli@90b61ed)
  [#3053](npm/cli#3053)
  fix(contributing.md): explicitely outline dep updates
  ([@darcyclarke](https://github.com/darcyclarke))

## v7.9.0 (2021-04-08)

### FEATURES

* [`1f3e88eba`](npm/cli@1f3e88e)
  [#3032](npm/cli#3032)
  feat(dist-tag): add workspace support
  ([@nlf](https://github.com/nlf))
* [`6e31df4e7`](npm/cli@6e31df4)
  [#3033](npm/cli#3033)
  feat(pack): add workspace support
  ([@wraithgar](https://github.com/wraithgar))

### DEPENDENCIES

* [`ba4f7fea8`](npm/cli@ba4f7fe)
  `licensee@8.2.0`

## v7.8.0 (2021-04-01)

### FEATURES


* [`8bcc5d73f`](npm/cli@8bcc5d7)
  [#2972](npm/cli#2972)
  feat(workspaces): add repo and docs
  ([@wraithgar](https://github.com/wraithgar))
* [`ec520ce32`](npm/cli@ec520ce)
  [#2998](npm/cli#2998)
  feat(set-script): implement workspaces
* [`32717a60e`](npm/cli@32717a6)
  [#3001](npm/cli#3001)
  feat(view): add workspace support
  ([@wraithgar](https://github.com/wraithgar))
* [`7b177e43f`](npm/cli@7b177e4)
  [#3014](npm/cli#3014)
  feat(config): add 'envExport' flag
  ([@isaacs](https://github.com/isaacs))

### BUG FIXES

* [`4c4252348`](npm/cli@4c42523)
  [#3016](npm/cli#3016)
  fix(usage): specify the key each time for multiples
  ([@isaacs](https://github.com/isaacs))
* [`9237d375b`](npm/cli@9237d37)
  [#3013](npm/cli#3013)
  fix(docs): add workspaces configuration
  ([@wraithgar](https://github.com/wraithgar))
* [`cb6eb0d20`](npm/cli@cb6eb0d)
  [#3015](npm/cli#3015)
  fix(ERESOLVE): better errors when current is missing
  ([@isaacs](https://github.com/isaacs))

### DEPENDENCIES

* [`61da39beb`](npm/cli@61da39b)
  `@npmcli/config@2.1.0`
  * feat(config): add support for envExport:false
* [`fb095a708`](npm/cli@fb095a7)
  `@npmcli/arborist@2.3.0`:
  * [#2896](npm/cli#2896) Provide currentEdge in
  ERESOLVE if known, and address self-linking edge case.
  * Add/remove dependencies to/from workspaces when set, not root project
  * Only reify the portions of the dependency graph identified by the
  `workspace` configuration value.
  * Do not recursively `chown` the project root path.

## v7.7.6 (2021-03-29)

### BUG FIXES

* [`9dd2ed518`](npm/cli@9dd2ed5)
  fix empty newline printed to stderr
  ([@ruyadorno](https://github.com/ruyadorno))
* [`9d391462a`](npm/cli@9d39146)
  [#2973](npm/cli#2973)
  fix spelling in workspaces.md file
  ([@sethomas](https://github.com/sethomas))
* [`4b100249a`](npm/cli@4b10024)
  [#2979](npm/cli#2979)
  change 'maxsockets' default value back to 15
  ([@wallrat](https://github.com/wallrat))

### DEPENDENCIES

* [`a28f89572`](npm/cli@a28f895)
  `libnpmversion@1.1.0`
    * fix reading `script-shell` config on `npm version` lifecycle scripts
* [`03734c29e`](npm/cli@03734c2)
  `npm-packlist@2.1.5`
    * fix packaging `bundledDependencies`
* [`80ce2a019`](npm/cli@80ce2a0)
  `@npmcli/metavuln-calculator@1.1.1`
    * fix error auditing package documents with missing dependencies

## v7.7.5 (2021-03-25)

### BUG FIXES

* [`95ba87622`](npm/cli@95ba876)
  [#2949](npm/cli#2949)
  fix handling manual indexes in `npm help`
  ([@dmchurch](https://github.com/dmchurch))
* [`59cf37962`](npm/cli@59cf379)
  [#2958](npm/cli#2958)
  always set `npm.command` to canonical command name
  ([@isaacs](https://github.com/isaacs))
* [`1415b4bde`](npm/cli@1415b4b)
  [#2964](npm/cli#2964)
  fix(config): properly translate user-agent
  ([@wraithgar](https://github.com/wraithgar))
* [`59271936d`](npm/cli@5927193)
  [#2965](npm/cli#2965)
  fix(config): tie save-exact/save-prefix together
  ([@wraithgar](https://github.com/wraithgar))

### TESTS

* [`97b415287`](npm/cli@97b4152)
  [#2959](npm/cli#2959)
  add smoke tests
  ([@ruyadorno](https://github.com/ruyadorno))
@feelepxyz feelepxyz mentioned this pull request Apr 16, 2021
Merged
This was referenced Jun 29, 2021
Open
Open
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@nlf nlf nlf approved these changes

Assignees
No one assigned
Labels
release: next These items should be addressed in the next release Release 7.x work is associated with a specific npm 7 release
Projects
None yet
Milestone
OSS - Sprint 27
Development

Successfully merging this pull request may close these issues.
4 participants
@ruyadorno @ljharb @nlf @jimmywarting