Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Migrate to ES Module structure/runtime - Node 20, Vite, Vitest, Storybook 8 #272

Merged
merged 55 commits into from
May 14, 2024

Conversation

twelch
Copy link
Contributor

@twelch twelch commented Mar 7, 2024

Dev

Key Requirements/Decisions

  • Switch from Jest to Vitest, supports ESM out of the box and is mostly Jest compatible API
  • Switch from webpack to esbuild, supports ESM out of the box and more easily configurable.
  • Switch from ts-node to tsx. Better node 20 and esm support out of the box.
  • upgrade flatgeobuf, and upgrade the docker-gp-workspace to the latest GDAL
  • request fix to geoblaze to work in ESM environment, v2.6.1 - https://github.com/GeoTIFF/geoblaze/commits/master/

ToDo

  • Migrate packages/geoprocessing
  • Swap __dirname and __filename to use node:url
  • Run test suite
  • Update top-level tsconfig.templates.json
  • Migrate other packages
  • Experimental publish
  • Test with Are the types right? https://arethetypeswrong.github.io/?p=%40seasketch%2Fgeoprocessing%406.1.3-experimental-esm.5
  • Test geoprocessing init
  • clean up import of CJS dependencies that don't have a true default import
  • migrate use of require, enough to get init working
  • flatgeobuf
    • figure out how to produce a flatgeobuf that js lib will read from disk. Why does countries.fgb work? But not world_new.fgb? Installing latest QGIS and regenerating flatgeobuf seems to have fixed. Also fixing node-fetch polyfill seems to have fixed.
    • Add tests to flatgeobuf.test.e2e.ts
  • VectorDatasource not fetching properly. Fixing node-fetch polyfill seems to have fixed it
  • Upgrade to node 20.11.1 and use new import.meta.dirname
  • esbuild
  • Add support for deploying Node 20 with ESModule code to AWS using ESM
    • Example project - data import, precalc, build, deploy, publish
  • Translation
    • Fix translation:install for when running init in a monorepo (can't find i18n directory because is at top-level of monorepo). build a more robust script instead of an inline command in package.json
  • Vite storybook
  • Vite build
    • replace esbuild with Vite. This must allow build to be done from GP_ROOT with entry point over in PROJECT_ROOT. esbuild doesn't currently allow this, specifically the html plugin
  • Update styled-components to 6.x and use {styled} named import to be able to access class properties.
  • Upgrade slugify
  • Upgrade turf to 7.0 prerelease (wasn't importing types properly for 6.5)
  • Switch to dynamic import approach for loading translations in client
  • Update start:client
  • Upgrade zx
  • Upgrade to latest Typescript
  • Update dependencies that are non-breaking
  • Make sure territorial waters image loads, currently still has require but esbuild seems to replace

Testing

  • Add flatgeobuf.ts test using local file
  • Verify devcontainer is working with unstable release of docker-gp-workspace

Release

  • Init a new project from scratch and go through to deploy - create sketch and collection, import data, test, deploy, publish data.
  • Migrate an existing gp project
  • Write migration doc
    • Add "type": "module" to package.json
    • Update tsconfig.json
    • Restart vscode for it to start recognizing use of esm mode
    • update package.json scripts, dependences, devDependencies. Can copy and overwrite directly from [insert link to release
    • install:scripts, then translation:install to upgrade
      file] and add back any that you added just for the project, if any.
  • Update docs
    • ANALYZE and MINIFY vars for build command
    • Note use of nvm alias default v20.12.1 and nvm use default in vscode terminal if your vscode terminal keeps not using the right version of node
    • devcontainer upgrade, to get access to node20, latest GDAL, etc.
    • writeup how CLI commands work. They run in gp space using gp dependencies, but run against the project space. Build commands additionally bundle the report and gp/pp functions first in project space?
    • Use of unstable devcontainer and how to pull the latest docker pull seasketch/geoprocessing-workspace:unstable
    • named export required for report component in order for example-reports to work
    • Document static asset handling, and importing images - https://vitejs.dev/guide/assets.html#static-asset-handling

Follow-on tickets:

  • Basic smoke test fails for ecoregions, FAIL src/functions/simpleFunctionSmoke.test.ts > Basic smoke tests > simpleFunction

  • fix snapshot update via test:scripts:update, need to migrate away from jest-based solution

  • Do sideEffects need to be added back for fetchPolyfill?

  • Vitest - leftovers from Switch from Jest to Vitest #273

  • Once migrate to aws-sdk v3, bring back the dynamodb tests - https://daaru.medium.com/testing-lambda-functions-with-vitest-fc713caa975d

  • Once migrate to aws-sdk v3, filter it and aws-cdk out of the function builds using externals

  • upgrade Typescript and switch from assert type JSON to with

  • Update base translations

  • Merge client-ui and client-core

  • Reduce project dependencies by bringing CLI commands into GP space (via cd GP_PATH) so that GP dependencies are used. translation:extract is one example where babel and plugin should be able to be moved.

  • Try to use top-level turf library

  • Consolidate src and scripts into one tsc build step (assuming pure esm)

  • Enable eslint - with warnings

  • Add monorepolint like turf uses for template packages

  • Convert "G extends Geometry" to "G extends Geometry | null" to support null geometry case?

  • Update React

  • Update i18next

  • Migrate from polygon-clipping back to turf clip if possible
    v7 release:

  • Publish new base EN translations and translate them to all languages

Old - will not do

Questions

  • What does the top-level tsconfig.json do? Should it be updated or deleted?
    • Doesn't seem to be used so deleted it. See if it breaks anything.
  • Should I use "module": "node16" like sindresorhus suggests instead of "module": "nodenext" like Typescript suggests?
    • Switched to nodenext, to get json assert support, which new project init was throwing an error on for import of all json files, even though node16. 🤷
    • node16 should work best with everything in toolchain, and it will be stable/fixed. nodenext can/will change when Typescript is upgraded.
  • Is "target": "es2022" for both src and scripts correct?
    • Yes, assuming webpack can handle it at the project build stage. This should ensure that the most modern esm code is generated by the typescript transpiler.
  • Should I have esModuleInterop and allowSyntheticDefaultImports enabled?
    • Let's leave them enabled if they already are for now.
  • What to replace require.resolve with in CJS files?

Notes on steps

- update typescript and ts-node config
- file path and extension changes
- migrate away from styled-icons/boxicons due to typing issue with esm
- update flatgeobuf, remove old workarounds
- add skipLibCheck to resolve flatgeobuf internal import error
- update turf to v7 prerelease, migrate to exporting latest geojson types, tighten up geometry generics
- ignore rbush type import errors
- switch from ts-node to tsx for prepare script
- migrating use of __dirname to new method
@twelch twelch marked this pull request as draft March 7, 2024 03:04
@twelch twelch self-assigned this Mar 13, 2024
…port. Migrate use of require in some places, enough to get init working.
* start switch to vitest

* drop jest stuff, other dirname and import dep fixes, reset lerna.json

* use default-import to use and extend rbush

* add vitest dep formally, add vitest-setup loading jest-dom

* Migrate more tests, make them runnable standalone in vscode debug with direct import

* update geoblaze, update overlap output changes related to turf area. skip many remaining tests

* fix prepare

* comment out and skip tests with jest mocks

* initial migrate of scripts to vitest, skipping failing precalc datasource tests

* stub vitest project runner

* clean up vitest config type error, and punt on browser test mock error and bump vi* versions

* skip remaining failing tests, add back test commit gate

* try making vitest regular dependency

* update runner to config project vitest

* merge cli options

* fix vitest cli command

* switch to TEST_ROOT env var

* shift jsdom dep for project use

* fix TEST_ROOT

* simplify test commands, tune test config

* update vitest comments

* add data server back to test command

* vitest comments

* drop ts-node for tsx

* add testing of base and template packages, skipping most.  Migrate remaining commands to vitest. Migrate to turf 7 in projects
* convert function builder to esbuild

* fix other function output dir, cleanup negative zero in cleanCoords

* add buildClient using esbuild with html plugin, refactor App to receive lazy loaded report clients, fix loose client-ui exports

* add esbuild polyfill, address esbuild strict module import - pare down projectClient import paths, switch client away from using top-level gp lib import, shift top-level indexes to prevent pollution of node code in ui code, guard use of process env var

* Switch to generating ReportApp.tsx in .build-web with lazy load of report clients

* on build copy language assets to .build-web
* upgrade aws-cdk-lib and aws-sdk, add package.json to build to get esm enabled in lambda, move esbuild dep to top of monorepo, add node 20 runtime

* update data scripts to work in monorepo

* fix precalc script env

* build - create isolated mini package per function with own entry point. Add type assertion to language json import
@twelch twelch changed the title Migrate to ES Module structure Migrate to ES Module structure, Node 20, Vitest, esbuild Apr 17, 2024
@twelch
Copy link
Contributor Author

twelch commented Apr 20, 2024

Upgrade WSL Ubunto to 22.04 Jammy

@twelch twelch mentioned this pull request May 9, 2024
91 tasks
* switch to transient prop for duration ProgressBar

* start conversion to Vite for client build, still need to fix import attributes and add dev server

* Migrate start:client to Vite, language dynamic import not working or keeping with:json import assertion

* Migrate gp lib to storybook 8 using Vite, missing project storybook still

* drop babel from gp vite config, causing require error

* add back storybook to build docs

* migrate from start-storybook to just storybook

* refactor report stories to use code generator with story-cache

* add story-cache to base project gitignore

* change report-stories to example-stories. Migrate templates to use them.  reports required to use named export

* migrate create:report to node 20 patterns.  Fix classColumn type error

* glob delete story caches on startup, add vite types using d.ts file for asset import

* switch to Vite glob loader for language modules

* no need to copy language modules into build-web anymore

* add vite to base-project

* export pathUtils, simplify tsconfig

* Drop unused TranslatorSync component

* shift storybook to dependencies for project user

* Change story folder structure

* Add turf/helpers to base-project since common project use

* add all to dependencies to try and solve storybook start error

* export datasource helpers, remove unused vitest workspace, update min node and npm in package.json

* Make project storybook+vite independent of gp version

* make styled-components props transient, drop unused main from base-project package.json

* vite configure web worker

* remove vite worker config since web worker not being used

* fix story sort order

* fix blank-project functions

* attempt fix to favicon install command

* add storybook install script for migraters.  Add initial Migrating docs

* add back default export to report clients require for React.lazy load
@twelch twelch changed the title Migrate to ES Module structure, Node 20, Vitest, esbuild Migrate to ES Module structure/runtime - Node 20, Vite, Vitest, Storybook 8 May 14, 2024
@twelch twelch marked this pull request as ready for review May 14, 2024 00:35
@twelch twelch merged commit 649fb9e into dev May 14, 2024
2 checks passed
@twelch twelch deleted the esm branch May 14, 2024 18:22
twelch added a commit that referenced this pull request Jun 25, 2024
…book 8 (#272)

* - update lerna and node, add eslint esm preset
- update typescript and ts-node config
- file path and extension changes
- migrate away from styled-icons/boxicons due to typing issue with esm
- update flatgeobuf, remove old workarounds
- add skipLibCheck to resolve flatgeobuf internal import error
- update turf to v7 prerelease, migrate to exporting latest geojson types, tighten up geometry generics
- ignore rbush type import errors
- switch from ts-node to tsx for prepare script
- migrating use of __dirname to new method

* switch CI to use node 20

* Migrate base-project and template packages to esm

* disable run of test suite on commit until resolved

* missed test

* fix fetchPolyfill import

* switch to nodenext module and moduleResolution.  Add assert to json imports

* clean up import of CJS dependencies that don't have a true default import. Migrate use of require in some places, enough to get init working.

* switch to tsx for translation extract and migrate babel config to cjs extension

* fix report name for template-blank-project

* Switch from Jest to Vitest (#273)

* start switch to vitest

* drop jest stuff, other dirname and import dep fixes, reset lerna.json

* use default-import to use and extend rbush

* add vitest dep formally, add vitest-setup loading jest-dom

* Migrate more tests, make them runnable standalone in vscode debug with direct import

* update geoblaze, update overlap output changes related to turf area. skip many remaining tests

* fix prepare

* comment out and skip tests with jest mocks

* initial migrate of scripts to vitest, skipping failing precalc datasource tests

* stub vitest project runner

* clean up vitest config type error, and punt on browser test mock error and bump vi* versions

* skip remaining failing tests, add back test commit gate

* try making vitest regular dependency

* update runner to config project vitest

* merge cli options

* fix vitest cli command

* switch to TEST_ROOT env var

* shift jsdom dep for project use

* fix TEST_ROOT

* simplify test commands, tune test config

* update vitest comments

* add data server back to test command

* vitest comments

* drop ts-node for tsx

* add testing of base and template packages, skipping most.  Migrate remaining commands to vitest. Migrate to turf 7 in projects

* fix type error, upgrade node-fetch, fix launcher paths, VectorDatasource not fetching properly?

* add new world fgb and confirm test passing with latest unstable gp docker workspace

* downgrade flatgeobuf to resolve timeout error, add back skipped tests

* migrate to new import.meta.dirname available with node 20.12

* update fetch polyfill and upgrade again to latest flatgeobuf.  subdivided datasource fetch also resolved.

* unskip remaining tests utilizing flatgeobuf, update sideEffects for fetch polyfill

* Switch to esbuild (#278)

* convert function builder to esbuild

* fix other function output dir, cleanup negative zero in cleanCoords

* add buildClient using esbuild with html plugin, refactor App to receive lazy loaded report clients, fix loose client-ui exports

* add esbuild polyfill, address esbuild strict module import - pare down projectClient import paths, switch client away from using top-level gp lib import, shift top-level indexes to prevent pollution of node code in ui code, guard use of process env var

* Switch to generating ReportApp.tsx in .build-web with lazy load of report clients

* on build copy language assets to .build-web

* bring back template smoke tests

* AWS - deploy pure ESM code for Lambda and client, on node 20 (#279)

* upgrade aws-cdk-lib and aws-sdk, add package.json to build to get esm enabled in lambda, move esbuild dep to top of monorepo, add node 20 runtime

* update data scripts to work in monorepo

* fix precalc script env

* build - create isolated mini package per function with own entry point. Add type assertion to language json import

* upgrade zx, refactor translation install and extract to use scripts clarify build script

* add missing esbuild plugins add missing babel plugin for translation extract, add client polyfill globals process true

* cleanup isHostedOnLambda

* process empty

* bump

* directly set client lambda env vars

* add esbuild metafile output

* add ANALYZE  env vars

* fix metafile output path

* Switch to gp core imports in client to not pull backend modules

* Allow minify to be overridden, defaults to true

* fix minify arg passing

* add cloufront access to client bucket to be able to access client files in directories.  Fix dynamic import of translations in browser client

* update translationInstall to be more comprehensive

* update dirname in importTerms and publishTerms, loosen gitignore on i18n dir

* Make sure .build dir created, remove old createManifest

* bump

* simplify generateHandler, add guard to .build dir copy

* Cut out intermediate build in gp .build dir, go straight to project path .build

* simplify build client gp path discovery

* make build client use gp dependencies. removes need for esbuild deps in project

* replace use of esbuild-plugin-html which doesn't support entry point outside of current directory

* upgrade to typescript 5.4.5

* swtich from assert json to with json

* fully switch from assert to with import syntax, esbuild still converts to older assert syntax but should change at some point

* rebuild package lockfile, major non-breaking npm upgrades

* additional upgrades:
- polygon-clipping upgrade resulted in additional vertice in some clip results slightly changing value (<.04 sq meters).
- Update base translations. Fix translations extract with babel import assertion plugin.

* upgrade babel proposal plugins to official

* update husky prepare command, husky install is deprecated

* migrate to new styled-components .attrs() to define caller-defined attributes to pass to components

* switch to transient prop for duration ProgressBar

* Vite for client dev, build, storybook (#282)

* switch to transient prop for duration ProgressBar

* start conversion to Vite for client build, still need to fix import attributes and add dev server

* Migrate start:client to Vite, language dynamic import not working or keeping with:json import assertion

* Migrate gp lib to storybook 8 using Vite, missing project storybook still

* drop babel from gp vite config, causing require error

* add back storybook to build docs

* migrate from start-storybook to just storybook

* refactor report stories to use code generator with story-cache

* add story-cache to base project gitignore

* change report-stories to example-stories. Migrate templates to use them.  reports required to use named export

* migrate create:report to node 20 patterns.  Fix classColumn type error

* glob delete story caches on startup, add vite types using d.ts file for asset import

* switch to Vite glob loader for language modules

* no need to copy language modules into build-web anymore

* add vite to base-project

* export pathUtils, simplify tsconfig

* Drop unused TranslatorSync component

* shift storybook to dependencies for project user

* Change story folder structure

* Add turf/helpers to base-project since common project use

* add all to dependencies to try and solve storybook start error

* export datasource helpers, remove unused vitest workspace, update min node and npm in package.json

* Make project storybook+vite independent of gp version

* make styled-components props transient, drop unused main from base-project package.json

* vite configure web worker

* remove vite worker config since web worker not being used

* fix story sort order

* fix blank-project functions

* attempt fix to favicon install command

* add storybook install script for migraters.  Add initial Migrating docs

* add back default export to report clients require for React.lazy load

* nit Migrating
@twelch twelch mentioned this pull request Jul 31, 2024
14 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Development

Successfully merging this pull request may close these issues.

1 participant