0000-dependency-selector-syntax.md

npm query Command & Dependency Selector Syntax

Summary

Introduce a new npm query commmand which exposes a new dependency selector syntax (informed by & respecting many aspects of the CSS Selectors 4 Spec).

Motivation

• Standardize the shape of, & querying of, dependency graphs with a robust object model, metadata & selector syntax
• Leverage existing, known language syntax & operators from CSS to make disperate package information broadly accessible
• Unlock the ability to anwser complex, multi-faceted questions about dependencies, their relationships & associative metadata
• Consolidate redundant logic of similar query commands in npm (ex. npm fund, npm ls, npm outdated, npm audit ...)

Detailed Explanation

This RFC's spec & implementation should closely mimic the capabilities of existing CSS Selector specifications. Notably, we'll introduce limited net-new classes, states & syntax to ensure the widest adoption & understanding of paradigms. When deviating, we'll explicitely state why & how.

Implementation

• Arborist's Node Class will have a new .querySelectorAll() method
  • this method will return a filtered, flattened dependency Arborist Node list based on a valid query selector
• Introduce a new command, npm query, which will take a dependency selector & output a flattened dependency Node list (output is in json by default, but configurable)

Dependency Selector Syntax

Overview:

• there is no "type" or "tag" selectors (ex. div, h1, a) as a dependency/target is the only type of Node that can be queried
• the term "dependencies" is in reference to any Node found in the idealTree returned by Arborist

Selectors

• * universal selector
• #<name> dependency selector (equivalent to [name="..."])
• #<name>@<version> (equivalent to [name=<name>]:semver(<version>))
• , selector list delimiter
• . class selector
• : pseudo class selector
• > direct decendent/child selector
• ~ sibling selector

Attribute Selectors

• [] attribute selector (ie. existence of attribute)
• [attribute=value] attribute value is equivalant...
• [attribute~=value] attribute value contains word...
• [attribute*=value] attribute value contains string...
• [attribute|=value] attribute value is equal to or starts with...
• [attribute^=value] attribute value begins with...
• [attribute$=value] attribute value ends with...

Pseudo Selectors

• :not(<selector>)
• :has(<selector>)
• :where(<selector list>)
• :is(<selector list>)
• :root matches the root node/dependency
• :scope matches node/dependency it was queried against
• :empty when a dependency has no dependencies
• :private when a dependency is private
• :link when a dependency is linked
• :deduped when a dependency has been deduped
• :override when a dependency is an override
• :extraneous when a dependency could be but is not deduped
• :outdated when a dependency is not latest
• :vulnerable when a dependency has a CVE
• :semver(<spec>) matching a valid node-semver spec
• :path(<path>) glob matching based on dependencies path relative to the project
• :realpath(<path>) glob matching based on dependencies realpath
• :type(<type>) based on currently recognized types

Generic Pseudo Selectors:

A standardized pseudo selector pattern will be used for Objects, Arrays or Arrays of Objects accessible via Arborist's Node.package metadata. Pseudo classes generated in this way will allow for iterative attribute selection.

Arrays specifically use a special value keyword in place of a typical attribute name. Arrays also support exact value matching when a String is passed to the selector. See examples below:

Example of an Object:

/* return dependencies that have a `scripts.test` containing `"tap"` */
*:scripts([test~=tap])

Example of an Array Attribute Selection:

/* return dependencies that have a keyword that begins with "react" */
*:keywords([value^="react"])

Example of an Array matching directly to a value:

/* return dependencies that have the exact keyword "react" */
/* this is equivalent to `*:keywords([value="react"])` */
*:keywords("react")

Example of an Array of Objects:

/* returns */
*:contributors([email="ruyadorno@github.com"])`

Classes

Given that Arborist will control our understanding of the DOM (Dependency Object Model), claseses are predefined by their relationships to their ancestors & are specific to their dependency type. Dependencies may have, & are allowed to have, multiple classifications (ex. a workspace may also be a dev dependency).

• .prod
• .dev
• .optional
• .peer
• .bundled
• .workspace

Attributes

There are several known attributes that are normalized & queryable living in Node.package (aka. package.json).

• [name]
• [version]
• [description]
• [homepage]
• [bugs]
• [author]
• [license]
• [funding]
• [files]
• [main]
• [browser]
• [bin]
• [man]
• [directories]
• [repository]
• [scripts]
• [config]
• [workspaces]
• [dependencies]
• [devDependencies]
• [optionalDependencies]
• [bundledDependencies]
• [peerDependencies]
• [peerDependenciesMeta]
• [engines]
• [os]
• [cpu]
• [keywords]

Command

npm query "<selector>" (alias q)

Options:

• query-output
  • Default: json
  • Type: json, list, explain, outdated, funding, audit, duplicates, file

Response Output

• an array of dependency objects is returned which can contain multiple copies of the same package which may or may not have been linked or deduped

[
  {
    "name": "",
    "version": "",
    "description": "",
    "homepage": "",
    "bugs": {},
    "author": {},
    "license": {},
    "funding": {},
    "files": [],
    "main": "",
    "browser": "",
    "bin": {},
    "man": [],
    "directories": {},
    "repository": {},
    "scripts": {},
    "config": {},
    "dependencies": {},
    "devDependencies": {},
    "optionalDependencies": {},
    "bundledDependencies": {},
    "peerDependencies": {},
    "peerDependenciesMeta": {},
    "engines": {},
    "os": [],
    "cpu": [],
    "workspaces": {},
    "keywords": [],
    "ancenstry": "",
    "path": "",
    "realpath": "",
    "parent": "",
    "vulnerabilities": [],
    "cwe": []
  },
  ...

Usage

npm query ":root > .workspace > *"  # get all workspace direct deps

Extended Example Queries & Use Cases

// all deps
*

// all direct deps
:root > *

// direct production deps
:root > .prod

// direct development deps
:root > .dev 

// any peer dep of a direct deps
:root > * > .peer  
  
// any workspace dep
.workspace

// all workspaces that depend on another workspace
.workspace > .workspace
    
// all workspaces that have peer deps
.workspace:has(*.peer)

// any dep named "lodash"
// equivalent to [name="lodash"]
#lodash 
  
// any deps named "lodash" & within semver range ^"1.2.3"
#lodash@^1.2.3
// equivalent to...
[name="lodash"]:semver(^1.2.3)

// get the hoisted node for a given semver range
#lodash@^1.2.3:not(:deduped)

// querying deps with a specific version
#lodash@2.1.5
// equivalent to...
[name="lodash"][version="2.1.5"]
  
// all deps living alongside vulnerable deps
*:vulnerable ~ *:not(:vulnerable)

// has any deps
*:has(*)

// has any vulnerable deps
*:has(*:vulnerable)

// deps with no other deps (ie. "leaf" nodes)
*:empty

// all vulnerable deps that aren't dev deps & that aren't vulnerable to CWE-1333
*:vulnerable:not(.dev:cwe(1333))
  
// manually querying git dependencies
*[repository^="github:"],
*[repository^="git:"],
*[repository^="https://github.com"],
*[repository^="http://github.com"],
*[repository^="https://github.com"],
*[repository^="+git:..."]
  
// querying for all git dependencies
*:type(git)

// find all references to "install" scripts
*[scripts=install], 
*[scripts=postinstall],
*[scripts=preinstall]

// get production dependencies that aren't also dev deps
.prod:not(.dev)
    
// get dependencies with specific licenses
*[license="MIT"], *[license="ISC"]

// find all packages that have @ruyadorno as a contributor
*:contributors([email=ruyadorno@github.com])

Next Steps

Command Mapping to query-output

Previous commands with similar behaivours will now be able to utilize Aborist Node.querySelectorAll() under-the-hood & will fast-follow the npm query implementation.

npm list

npm list # equivalent to...
npm query ":root > *"

npm list --all # equivalent to...
npm query "*" --query-output list

npm list <pkg> # equivalent to...
npm query "#<pkg>" --query-output list

npm explain

npm explain <pkg> # equivalent to...
npm query "#<pkg>" --query-output explain

npm outdated

npm outdated # equivalent to...
npm query ":root > *:outdated" --query-output outdated

npm outdated --all # equivalent to...
npm query "*:outdated" --query-output outdated

npm fund

npm fund # equivalent to...
npm query ":root > *[funding]" --query-output funding

npm fund --all # equivalent to...
npm query "*[funding]" --query-output funding

npm fund <pkg> # equivalent to...
npm query "#<pkg>" --query-output funding

npm audit

npm audit # equivalent to...
npm query ":root > *:vulnerable" --query-output audit

npm audit --all # equivalent to...
npm query "*:vulnerable" --query-output audit

npm audit <pkg> # equivalent to...
npm query "#<pkg>" --query-output audit

npm find-dupes

npm find-dupes # equivalent to...
npm query "*:deduped" --query-output duplicates

npm view

npm view # equivalent to...
npm query ":root" --query-output view

npm view <pkg> # equivalent to...
npm query "#<pkg>" --query-output view

Commands could read from stdin

In a future RFC, & major version bump, npm could begin reading from stdin to chain commands together with a common understand of a dependency object. All of the below commands would add the ability to execute as if they were passed package specs (our current defulat representation of packages/dependencies).

audit, bugs, ci, config, deprecate, diff, dist-tag, docs, edit, exec, 
explain, explore, find-dupes, fund, get, install, install-ci-test, 
install-test, link, list, outdated, pack, pkg, prune, publish, rebuild, 
repo, restart, run-script, set, set-script, shrinkwrap, star, stars,
start, stop, team, test, token, uninstall, unpublish, unstar, update,
version, view

Example of piping from npm query to other commands

# list workspaces w/ peer deps
npm query ".workspace:has(.peer)" | npm ls

# list outdated direct dev deps
npm query ":root > .dev:outdated" | npm outdated

# install the same dev deps across all workspaces
npm query ":root > .dev" | npm install --workspaces

# show audit details for dependencies with a specific vulnerability/CWE
npm query "*:cwe(1333)" | npm audit

# show audit details for vulnerable deps that aren't ReDoS dev deps
npm query "*:vulnerable:not(.dev:cwe(1333))" | npm audit

Prior Art

• HTML & DOM Specifications
• CSS, Selectors & Pseudo Class Specifications
• AST Selector Libraries/Parsers
  • estree
  • abstract-syntax-tree
  • postcss-selector-parser / API
  • css-selector-parser
• pnpm's --filter Flag
• Gzemnid Package Querying

F.A.Q.

• Q. Is there any such thing as a bare specifier?
  • A. No. Unlike CSS Selectors, there's no "element", or "dependency" in this context, equivalent. The selector syntax presuposes all entities are packages.
• Q. Should this syntax cover all possible queries of a dependency graph?
  • A. No. This spec is meant to provide a sufficently mature mechanism/building block for answering the majority of questions end-users have about their depndencies (re. 80/20 rule applies here)

Unresolved Questions & Bikeshedding

• N/A