Restructure Repository to streamline development & releasing

[hmans]

Introduction & Important Notes

This is a big PR that changes a lot of things around in both the repository itself, and also in how the package is built. It implements a setup that I've been successfully using in my projects and been recommending to others, and is meant to be read as primarily a suggestion. There's a lot of stuff going on and I would understand 100% if you decide against adopting this; but do give it a try and see if it would work for you. If you have questions, I'd be happy to answer them.

First and foremost, this PR makes three major changes to the project:

• It restructures the repository into a Yarn Workspaces-based monorepo. In this repository, where just a single package is published, this is mostly done because it is a prerequisite for the other two packages, both of which make use of Yarn Workspaces, handling both the package and the different example apps as individual workspaces.
• It adds Preconstruct to the monorepo. Preconstruct is a monorepo-leaning build system (sitting on top of rollup) that has one amazing feature that comes in extremely handy during development: it provides a preconstruct dev task that will wire up all the different workspaces that internally depend on each other (like the examples apps depend on the CSM package) to make any kind of manual linking unnecessary; in fact it even does this in a way that workspaces can consume packages' Typescript sources (so you don't even need to run a yarn watch or similar task. You can just write code and immediately get type warnings, HMR, the works. It's great!)
• It adds Changesets to help with release and changelog management. Changesets allows you to express changes that should trigger a new release (patch, minor, or major) as a small Markdown file; once it's time to release a new version, the changeset version script will accumulate all these into updated CHANGELOG.md files, and changeset publish will publish changed packages to NPM. You can even set up a GitHub Action that will do all of these for you by way of automated Pull Requests -- just merge and it'll publish. It's really convenient.

Development

One of the benefits all this provides is that hacking on the code now just requires these three steps:

1. Clone the repository
2. yarn install
3. yarn dev

The last step will fire up the waves example, and you can immediately start making changes to the CSM package's TypeScript code with reloading, HMR, and so on.

Releasing

Given at least one unreleased changeset in the repository, the process is now the following:

1. Run yarn changeset version -- this will accumulate the unreleased changesets into updated CHANGELOG files and ask you to commit these changes (after review.)
2. Run yarn release -- this will run the yarn ci task (which does integrity tests and builds the package) followed by changeset publish, which will publish the new version you created in the previous step to NPM.

All of this can be wrapped in a GitHub action that I'd be happy to set up in a separate PR.

Caveats

• The way that the CSM package provides its main and vanilla entry points is now based on a different configuration from before; Preconstruct uses the sub-package approach for these for maximum compatibility; before, the package was using the new-ish package.json exports feature. Preconstruct is opinionated in this regard and can probably not reconfigured to use another approach, but the approach it's chosen has proven to be very reliable in the past. From where I'm standing, I don't see an issue with this switch, but YMMV.
• Since the repository now has the ability to let example apps directly import the source TypeScript code during development, I stumbled over the apparently widespread issue of CRA, even if configured to use Typescript, refusing to transpile TS sources from outside of its local src folder. I solved this by customizing each CRA apps' configuration through craco. I wish this hadn't been necessary, but it was the most pragmatic way forward. Please keep in mind that this is only necessary for the local development mode, it does not affect regular CSM usage in CRA apps.

Other minor changes:

• Adds manypkg to quickly identify and fix inconsistencies in the version requirements across the different example applications and packages
• Adds a yarn ci task that performs a bunch of integrity tests and builds all packages (if we ever get tests, we can add them here)
• Adds a yarn release task that will perform the ci task followed by changeset publish.
• Moves README.md and LICENSE.md into the package workspace, but symlinks both into the project root.
• Adds a tiny stackblitz configuration in package.json to make SB fire up an example on startup
• Tweaks to the GH Action for automated tests:
  • It now runs the yarn ci script mentioned above
  • It will also run for PRs now
  • Uses the yarn cache action for faster runtimes

Todo:

• Restructure repository
• Add changesets and preconstruct
• Change package structure to meet preconstruct's requirements, keeping the two distinct entrypoints intact
• Test if everything links up correctly
  • caustics
  • debug
  • instances
  • points
    • The example itself is reporting a shader compilation error (also on master)
  • vanilla
  • waves
• Remove the scripts we no longer need
• Add development and release documentation to the README

[FarazzShaikh]

This looks great to me, i am able to run it by just using yarn dev no linking etc required! Amazing! Couple notes:

• I dont really like packages/three-custom-shader-material, since there will only ever be one package in this repo, can it not just be called something like package? Or is it a limitation of the build system
• What would be the flow for releases? Also how do i make pre-releases? Please share some guides if you have any Never mind just read the disc

[hmans]

I dont really like packages/three-custom-shader-material, since there will only ever be one package in this repo, can it not just be called something like package? Or is it a limitation of the build system

We can probably change it to just ./package/ or similar. I will give it a go. Having ./packages/* is simply the typical monorepo convention (even if you're just dealing with a single package), but on a technical level, it won't make a difference.

What would be the flow for releases? Also how do i make pre-releases? Please share some guides if you have any

Sure! The tl;dr is:

1. Make commits with changesets in them (changesets are just files that describe a change, and signal what kind of release -- minor, major, patch -- should be done next to release them.) Changesets are typically part of the commits that implement changes (or merge PRs implementing changes.)
2. Once you're ready to do a release, yarn changeset version. This will accumulate all existing changesets, update the CHANGELOG.md, and bump all versions, but still wait for you to commit these updates (a useful feature because it gives you a chance to make some manual changes to the changelogs if necessary.)
3. At this point, the version numbers in package.json will have been bumped, but the new versions will not have been released to NPM. To do this, you'll do yarn release, which will build the package, run the non-existing tests, and finally push the unreleased versions to NPM.
4. Repeat. :-)

Note that steps 2 and 3 can be handled using a GitHub Action that creates PRs for you (see this one for an example) that you just merge whenever you want to release, and the action will perform everything for you (including the publishing.) Setting this up is recommended (because it's so convenient), but entirely optional. In fact, if you're new to changesets, I'd recommend doing a couple of releases manually (as described above), so you understand how changesets ticks, before automating them.

Let me know if you have more questions!

[hmans]

Addendum to the release process: it's important to note that you can still do releases the old-fashioned way (with the new updates in this PR, essentially yarn build, a manual edit of package.json, and yarn publish), if the changesets-driven approach is feeling intimidating.

[hmans]

I've made the change you requested about moving the package into a top-level package folder, and tested that everything still works as intended.

[hmans]

I've also added a GH action configuration for the changesets stuff. When merged, this will start creating release PRs as soon as new changesets are added to the repository, but it will never publish anything without your action. (It wouldn't be able to, anyway, since it doesn't have an NPM token yet.) But it'll be a good opportunity to look at what these things look like.