Deploy `@deephaven/jsapi-types` during release, provide tools for nightly builds #4907

niloc132 · 2023-12-01T20:31:40Z

Like other packages, the npmjs package for jsapi types will now get its version assigned as it is built from gradle. Also reads properties from gradle to specify --preid to npm (e.g. -PnpmPreid=nightly), to be given a date stamp suffix.

Public job for CI will now deploy tagged versions to npmjs.

Fixes #4888
Fixes #5081

Note that this isn't yet part of CI, either for nightly builds or for tagged releases.

Fixes deephaven#4888

niloc132 · 2023-12-01T20:34:05Z

Marked as draft until 0.31 goes out.

Two attached example builds for manual review:

"beta" for a nightly builds:

./gradlew :web-client-api:types:typedoc -PnpmPreid=beta

deephaven-jsapi-types-1.0.0-dev0.31.0beta20231201.tgz

no "preid" for a release:

./gradlew :web-client-api:types:typedoc

deephaven-jsapi-types-1.0.0-dev0.31.0.tgz

.github/workflows/publish-ci.yml

niloc132 · 2023-12-01T20:42:30Z

.github/workflows/publish-ci.yml

@@ -98,3 +106,9 @@ jobs:
          user: __token__
          password: ${{ secrets.PYDEEPHAVEN_PYPI_TOKEN }}
          packages_dir: py/client/build/wheel/
+
+      - name: Publish @deephaven/jsapi-types to npmjs


Note that I haven't tested this, all lifted from web-client-ui

niloc132 · 2023-12-01T20:44:24Z

web/client-api/types/build.gradle

+def npmVersion = '1.0.0-dev' + project.property('deephavenBaseVersion').toString().trim()
+def preId = project.findProperty('npmPreid')// i.e. 'nightly'
+if (preId) {
+    npmVersion += preId + new Date().format("yyyyMMdd")


We might want to consider hhmmss in here too? Or make it optionally something that could be specified from command line?

This is in lieu of

ensuring we tag on main (which npm insists on for "preid" to work), and

doing a checkout with full history

copying all of .git into the docker container for npm to find how many commits since the last tag.

Yea probably want to include hhmmss as well, just to avoid any collisions

niloc132 · 2023-12-01T20:44:58Z

web/client-api/types/package.json

@@ -1,6 +1,6 @@
 {
  "name": "@deephaven/jsapi-types",
-  "version": "1.0.0-dev0.31.0",
+  "version": "0.0.0-dummy",


explicit dummy version to be clear that developers need not update this, it will automatically be updated in the gradle+docker wiring

.github/workflows/publish-ci.yml

devinrsmith · 2023-12-15T19:56:31Z

.github/workflows/publish-ci.yml

+        with:
+          node-version-file: 'web/client-api/types/.nvmrc'
+          registry-url: 'https://registry.npmjs.org'
+
      # TODO(deephaven-core#2614): Improve gradle/CI assemble and publishing of artifacts

      - name: Publish to Maven Local


I'd go ahead and update this name in the same way you've updated the latter.

.github/workflows/publish-ci.yml

stanbrub · 2023-12-15T22:21:35Z

proto/raw-js-openapi/package.json

@@ -9,10 +9,11 @@
  "devDependencies": {
    "ts-loader": "^6.0.1",
    "typescript": "3.9.7",
-    "webpack": "^5.38.1",
+    "webpack": "^5.76.0",


Forgive my dumb questions here, but why 5.76.0 and not one of the patches?

Only because my experience with these tools is limited, and the output does what we need with this version.

Looking at the package-lock.json, it installed 5.89.0 anyway (specifying the version with a leading ^ prefixed will automatically take any newer minor versions if you're >=1.x, https://semver.npmjs.com/#syntax-examples).

Ok - should I update package.json then, or is it sufficient to say "the combination of the committed package.json and package-lock.json result in a deterministic build"?

No need to update, the combination of package.json/package-lock.json results in a deterministic build.

stanbrub · 2023-12-15T22:41:44Z

web/client-api/types/package.json

@@ -10,9 +10,6 @@
    "url": "https://github.com/deephaven/deephaven-core.git",
    "directory": "web/client-api/types"
  },
-  "engines": {
-    "node": ">=16"
-  },
  "devDependencies": {
    "typedoc": "^0.24.8",
    "typescript": "^5.1.6"


Why is this version different than the one in proto/raw-js-openapi/package.json?

I'm not going to go so far as to say that I fear change, but I haven't kept up with the changes, and the "inputs" into this typescript process are generated by a tool that are no longer getting updates.

devinrsmith · 2024-01-30T18:51:19Z

.github/workflows/publish-ci.yml

+      - name: Publish @deephaven/jsapi-types to npmjs
+        if: ${{ startsWith(github.ref, 'refs/heads/release/v') }}
+        env:
+          NODE_AUTH_TOKEN: ${{ secrets.DEEPHAVENBOT_NPM_TOKEN }}


We'll (I can do this) need to provide deephaven-core repo access to this secret before this action will succeed; I don't know how the npmjs secret management works, but I'm assuming this is a "heavy hammer", but we are okay w/ it @mofojed? The alternative would be a secret that is scoped explicitly for @deephaven/jsapi-types.

Man I wish npmjs had trusted publishers like PyPi does. Way better workflow.

Yes, the DEEPHAVENBOT_NPM_TOKEN is not scoped to a particular package, it is a "heavy hammer". You can create granular tokens that are scoped to specific packages, but I think this token scoped to our organization should be fine (organization wide secret) as it would be a pain to have to create new tokens for each package. Or we can have different tokens for the web-client-ui repo, deephaven-plugins repo, and whatever is published by deephaven-core, but then we need to update those tokens anytime we want to publish a new package in that repo (which happens from time to time in both web-client-ui and deephaven-plugins).
https://docs.github.com/en/actions/publishing-packages/publishing-nodejs-packages#publishing-packages-to-the-npm-registry

Done - secret now accessible from deephaven-core.

devinrsmith · 2024-01-30T18:57:42Z

.github/workflows/publish-ci.yml

        if: ${{ !startsWith(github.ref, 'refs/heads/release/v') }}
        uses: burrunan/gradle-cache-action@v1
        with:
          job-id: publish-local
-          arguments: server-netty-app:build server-jetty-app:build py-server:build py-embedded-server:build py-client:build publishToMavenLocal
+          arguments: server-netty-app:build server-jetty-app:build py-server:build py-embedded-server:build py-client:build web-client-api:types:typedoc publishToMavenLocal


I think assemble.dependsOn(typedoc) should be added to the web-client-api:types build logic; in which case, this could be updated to web-client-api:types:build.

Okay, changed - I went a bit further and declared both the npm module tarball and the doc directory as artifacts, which automatically hooks them up as dependent on assemble, just in case we use either in the future in that way.

.github/workflows/publish-ci.yml

mattrunyon · 2024-01-30T19:43:38Z

Builds and passes tests in my web-client-ui branch that consumes these types. Output looks good from the web side

mofojed · 2024-01-31T15:51:26Z

proto/raw-js-openapi/package.json

@@ -9,10 +9,11 @@
  "devDependencies": {
    "ts-loader": "^6.0.1",
    "typescript": "3.9.7",
-    "webpack": "^5.38.1",
+    "webpack": "^5.76.0",


Looking at the package-lock.json, it installed 5.89.0 anyway (specifying the version with a leading ^ prefixed will automatically take any newer minor versions if you're >=1.x, https://semver.npmjs.com/#syntax-examples).

mofojed · 2024-01-31T16:03:17Z

web/client-api/types/build.gradle


        runCommand('''set -eux; \\
                      cd /project/; \\
                      mv dist/types.d.ts dist/index.d.ts; \\
+                      npm version $VERSION; \\


Doing this will leave an uncommitted change in package.json - do we need to revert it when we're done?

No - there is no change to the source files, this happens inside docker (and doesn't use volumes/bind mounts).

mofojed · 2024-01-31T16:07:57Z

.github/workflows/publish-ci.yml

+        if: ${{ startsWith(github.ref, 'refs/heads/release/v') }}
+        env:
+          NODE_AUTH_TOKEN: ${{ secrets.DEEPHAVENBOT_NPM_TOKEN }}
+        run: npm publish --tag latest web/client-api/types/build/deephaven-jsapi-types-*.tgz


I think you should be referencing a package.json, not a tar file directly... does this actually work?

Yes - this is the way to say "here's a finished tarball, go ahead and deploy it". The work of making the tarball is done inside of docker (so that we guarantee it builds the way we expect, that it will do on our own machines), so this npm invocation just has access to the finished tarball and the secret.

mattrunyon

Approved for consuming these types. The types built from this branch work w/ my web-client-ui branch https://github.com/mattrunyon/web-client-ui/tree/jsapi-types

niloc132 · 2024-02-09T21:29:00Z

Pushing a new merge commit to get CI to re-run, since clicking re-run wasn't doing anything...

niloc132 added 9 commits November 30, 2023 08:37

Add notes on releasing jsapi-types

ce567ff

Note that this isn't yet part of CI, either for nightly builds or for tagged releases.

Revert package-lock.json until we update our tagged node image

6e745d7

Update node image, update builds to use it and enforce npm/node vers

8e475d0

Fixes deephaven#4888

Avoid extra copy of ts/js files

2ab6475

Stubbed in vers bump

a93d93c

better exclude?

eacf5f4

forgotten out

ae30a5e

unify version format

cfba8dc

Always update version, set up publish CI for js types

59263d2

niloc132 added build jsapi NoDocumentationNeeded release NoReleaseNotesNeeded No release notes are needed. labels Dec 1, 2023

niloc132 added this to the December 2023 milestone Dec 1, 2023

niloc132 commented Dec 1, 2023

View reviewed changes

.github/workflows/publish-ci.yml Outdated Show resolved Hide resolved

niloc132 commented Dec 1, 2023

View reviewed changes

Merge branch 'main' into 4888-jsapi-types-followup

169fbce

niloc132 marked this pull request as ready for review December 15, 2023 19:48

niloc132 requested review from devinrsmith, jcferretti, rcaudy, stanbrub, JamesXNelson and mofojed as code owners December 15, 2023 19:48

Remove duplicate engines entry

b110c84

devinrsmith reviewed Dec 15, 2023

View reviewed changes

stanbrub reviewed Dec 15, 2023

View reviewed changes

niloc132 added 6 commits January 25, 2024 15:16

Merge branch 'main' into 4888-jsapi-types-followup

bb56530

Review feedback from Bender/Matt

74e95e0

LayoutHints.frontColumns must be nullable

ffd4463

Avoid covariance in generated typescript

732324e

Review feedback

1a7739e

Merge branch 'main' into 4888-jsapi-types-followup

c99c4d2

niloc132 requested review from devinrsmith, stanbrub and mattrunyon January 29, 2024 17:34

Bump ts codegen version to fix inheritance issue

d5d44fe

devinrsmith reviewed Jan 30, 2024

View reviewed changes

mofojed reviewed Jan 31, 2024

View reviewed changes

Include .d.ts and npm module tarball as artifacts

22e6dcd

mofojed assigned niloc132 and mattrunyon Feb 5, 2024

niloc132 requested review from mofojed and devinrsmith February 7, 2024 22:09

mattrunyon approved these changes Feb 7, 2024

View reviewed changes

mofojed approved these changes Feb 9, 2024

View reviewed changes

devinrsmith approved these changes Feb 9, 2024

View reviewed changes

Merge branch 'main' into 4888-jsapi-types-followup

5caa58c

niloc132 enabled auto-merge (squash) February 9, 2024 21:30

niloc132 merged commit 168ff34 into deephaven:main Feb 9, 2024
25 of 37 checks passed

github-actions bot locked and limited conversation to collaborators Feb 9, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Deploy `@deephaven/jsapi-types` during release, provide tools for nightly builds #4907

Deploy `@deephaven/jsapi-types` during release, provide tools for nightly builds #4907

niloc132 commented Dec 1, 2023 •

edited

Loading

niloc132 commented Dec 1, 2023

niloc132 Dec 1, 2023

niloc132 Dec 1, 2023

mofojed Feb 1, 2024

niloc132 Dec 1, 2023

devinrsmith Dec 15, 2023

stanbrub Dec 15, 2023

niloc132 Jan 29, 2024

mofojed Jan 31, 2024

niloc132 Feb 1, 2024

mofojed Feb 1, 2024

stanbrub Dec 15, 2023

niloc132 Jan 29, 2024

devinrsmith Jan 30, 2024

mofojed Jan 30, 2024

devinrsmith Jan 31, 2024

devinrsmith Jan 30, 2024

niloc132 Feb 1, 2024

mattrunyon commented Jan 30, 2024

mofojed Jan 31, 2024

mofojed Jan 31, 2024

niloc132 Feb 1, 2024

mofojed Jan 31, 2024

niloc132 Feb 1, 2024

mattrunyon left a comment

niloc132 commented Feb 9, 2024

Deploy @deephaven/jsapi-types during release, provide tools for nightly builds #4907

Deploy @deephaven/jsapi-types during release, provide tools for nightly builds #4907

Conversation

niloc132 commented Dec 1, 2023 • edited Loading

niloc132 commented Dec 1, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mattrunyon commented Jan 30, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mattrunyon left a comment

Choose a reason for hiding this comment

niloc132 commented Feb 9, 2024

Deploy `@deephaven/jsapi-types` during release, provide tools for nightly builds #4907

Deploy `@deephaven/jsapi-types` during release, provide tools for nightly builds #4907

niloc132 commented Dec 1, 2023 •

edited

Loading