Infra: Switch monorepo tooling to wireit and pnpm

[ryan-roemer]

This PR is a big infrastructure overhaul to switch us from lerna + yarn to wireit + pnpm.

Why?

Our existing setup of yarn + lerna has the following undesirable things:

1. Installs are slow on yarn
2. Yarn1 isn't updated anymore and simple package dependency updates don't work out of the box. It's a huge pain to update anything.
3. Lerna is getting old, and while the NX team has agreed to support it now, our build is (1) slow, (2) not cached at all, and (3) opaque as to what you need to run task-wise to support other tasks.

So, let's welcome PNPM + Wireit!

1. PNPM is a fast installer and well supported for normal things like package upgrades.
2. Wireit allows fine-grained subtask caching and dependency specifications to allow us to: (1) specify dependencies so when you run pnpm run jest all the other things that need to happen magically happen, (2) caches sub-tasks so things that don't have changed file inputs don't re-run, and (3) has full GH actions CI support!

To get a sense of how much faster our build has become take a look at the CI times for this branch in https://github.com/FormidableLabs/victory/actions?query=branch%3Ainfra%2Fpnpm-wireit++ -- when no input files change our CI times are about 1 minute. When some things change, a couple minutes. When all things (or our base scripts) change, we take the full hit of a comparable existing Victory CI time of like 15-16 mins.

For the average Victory developer, if you're working in just one package, you can just run the project-level pnpm run check and have like a full build and everything work reasonably fast without needing to know more or run subtasks!

Check it out

$ npm install -g pnpm
$ pnpm install
# This will be slow!
$ pnpm run check

# ... but the second time will be super fast! And as you change things subsequent runs should be very fast!
$ pnpm run check

Implementation notes

• High level:

  • Dependency graph architecture: All of our tasks now use wireit to identify and run/cache dependencies. So, if you want to run jest, you just run pnpm run jest and don't need to worry about "what else needs to be built?" before that.
  • Parallelization: To best use wireit, we've refactored our tasks to be more package-specific where possible. E.g., we run lint in incremental mode per-package meaning that both Wireit (at package level) and eslint (at file level within package) re-execute on the narrowest unit of "changed files" possible.

• Demos: The demos originally had imports from victory*/src/index (source) which wasn't efficient or consistent because the transitive deps on other victory packages was on built files. I refactored these to be normal victory(-<NAME>) imports using built files.

• src:

  • Since we did a "fresh" install with pnpm, there were some updates in lint packages that meant source or tests needed small tweaks to continue passing/building.
  • Refactor self-references within a package to not use package name.
  • Fixed missing package dependencies uncovered as pnpm-based installs will fail on missing dependencies that are flattened and "accidentally work" in yarn/npm.

• Configs:

  • Webpack resolve.alias: Since we no longer hoist victory packages to root, we now add aliases for our webpack configs and storybook (which uses webpack) programatically.
  • Babel, Jest configs: Normalized the naming and location of these across normal and native ones.

• Incremental caching:

  • eslint

Remaining work in this PR

Big tasks:

• Victory Native: We need to check that everything still works for reals in RN and for our demos.
• Chromatic: We use chromatic mainly through GH actions which don't need changes. But we also have localdev tasks to test things out. Should be straightforward.
• Docs: This is still an independent install via yarn. Should work, but let's check.
• Watch support: Something reasonable for most workflows out of the gate. Ticket the rest.
  • start*
  • check
  • build: Might combine with check (?)
• Fix scripts: We have to pass different "fix" flags to things. Let's aggregate to individual and meta-tasks.

All the rest:

• Update all tasks, workflows, and guides in CONTRIBUTING.md
• Configure wireit package locks Needed customization for PNPM and we should identify which subtasks should have a package log dep. Add pnpm-lock.yaml as files field for some tasks.
• Search on
  • yarn
  • lerna
  • nps (direct uses)
  • TODO(wireit)
  • TODO(pnpm)

Remaining work (ticketed)

Incremental caching: Investigate opportunities for tool-level incremental caching. #2375

• build:dist:*: See https://webpack.js.org/configuration/cache/
• format: Need to update to v2.7. https://prettier.io/blog/2022/06/14/2.7.0.html
• types
• jest

Various things

• Publish support: Publish with changesets. Infra: Publish with changesets #2373
• Update dependencies! This is a huge pain with yarn, so we've been avoiding. After PNPM lands and everything works, let's finally update everything! Infra: Get all deps updated #2236
• Add code coverage, split out in check:ci and upload to CodeCov. Infra: Add coverage reporting to CI #2348
• Revisit how we're doing build watches for start and start:ts. Infra: Review how wireit watches are working. #2374
• Review legacy sub-projects that still use yarn: Infra: Review existing yarn installs #2376

[scottrippey]

Looking good so far! So excited about the initial results.
Excellent idea to automate the wireit config!

[scottrippey]

Well I did a spot-check of several of the --watch workflows, and I'm thrilled!
It's awesome how we can just jump into packages/victory-line and run pnpm run jest and it just works! Pulls from all previous cache's. Detects changes instantly, and reruns very fast.

LGTM!

[gksander]

This is AWESOME. 🎈