ci: merge staging to master #4

MatrixAI-Bot · 2023-08-07T06:24:17Z

This is an automatic PR generated by the pipeline CI/CD. This will be automatically fast-forward merged if successful.

[ci skip]

These are pending updates in agent migration stage 2.

CLI extracted from Polykey and migrated to Polykey-CLI

MatrixAI-Bot · 2023-08-07T06:24:19Z

Pipeline Attempt on 957701609 for 15c755f

https://gitlab.com/MatrixAI/open-source/Polykey-CLI/-/pipelines/957701609

tegefaulkes · 2023-08-07T07:03:28Z

Looks like the mac and windows builds are failing in CI. It's been a long time since these jobs were ran in Polykey so I don't know if it's a new problem with the CLI migration, or a general problem from Polykey's network migration work.

I'll disable them for now since they're not on the critical path. The integration tests for these builds are already disabled.

Next step is to get the nix-build step working, right now it's failing to clone the uWebsockets repo. I think this sort of thing should be pre-fetched by nix? I'm not sure how to add that sort of thing in.

…2nix`

MatrixAI-Bot · 2023-08-07T07:09:45Z

Pipeline Attempt on 957744108 for 1380d13

https://gitlab.com/MatrixAI/open-source/Polykey-CLI/-/pipelines/957744108

tegefaulkes · 2023-08-07T07:54:20Z

Ok, i'm putting this on hold for now. The current problem with nix-dry and nix-build -attr application is blocked by the uWebsockets removal.

Priority now is to work on the agent migration and complete that, including removing uWebsockets in favour of ws.js.

MatrixAI-Bot · 2023-08-11T06:01:26Z

Pipeline Attempt on 962904756 for 6db2859

https://gitlab.com/MatrixAI/open-source/Polykey-CLI/-/pipelines/962904756

CMCDragonkai · 2023-08-15T06:36:28Z

@tegefaulkes is it possible to pull in the latest PK release right now?

tegefaulkes · 2023-08-15T06:38:21Z

You should be able to use 1.1.5-alpha.0.

CMCDragonkai · 2023-08-15T06:40:06Z

Looks like @matrixai/quic is currently a dependency of this. Seems wrong. It seems that it's relying on the QUICConfig type. That shouldn't be done. PK should abstract around that type, so we are only using configuration types from polykey.

»» ~/Projects/Polykey-CLI/src
 ♖ ag 'quic'                                   ⚡(staging) pts/5 16:39:08
types.ts
12:import type { QUICConfig } from '@matrixai/quic';
18:// TODO: fix this... We don't need a dependecy on `@matrixai/quic
19:type PolykeyQUICConfig = {
29:} & Partial<QUICConfig>;
64:      // Agent QUICSocket config
81:    quicServerConfig?: PolykeyQUICConfig;
82:    quicClientConfig?: PolykeyQUICConfig;

MatrixAI-Bot · 2023-10-17T07:20:27Z

Pipeline Attempt on 1039206803 for 0c29de4

https://gitlab.com/MatrixAI/open-source/Polykey-CLI/-/pipelines/1039206803

MatrixAI-Bot · 2023-10-17T07:26:02Z

Pipeline Attempt on 1039212316 for 8f80d1f

https://gitlab.com/MatrixAI/open-source/Polykey-CLI/-/pipelines/1039212316

MatrixAI-Bot · 2023-10-17T07:58:43Z

Pipeline Attempt on 1039245564 for 41273c6

https://gitlab.com/MatrixAI/open-source/Polykey-CLI/-/pipelines/1039245564

MatrixAI-Bot · 2023-10-17T18:17:05Z

Pipeline Attempt on 1040057667 for 138fcdf

https://gitlab.com/MatrixAI/open-source/Polykey-CLI/-/pipelines/1040057667

CMCDragonkai · 2023-10-17T18:29:02Z

To address 3., it's important to reactivate some integration tests. But they need to be written minimally, so that they don't rely too much on implementation details. We're all the way up to the UX now. Not in the weeds. @tegefaulkes @amydevs

CMCDragonkai · 2023-10-17T19:16:22Z

Docker image has been built and deployed, so that is all still working.

However there are integration tests failing https://gitlab.com/MatrixAI/open-source/Polykey-CLI/-/pipelines/1040057667.

This was expected, it's time to reactivate and refactor our integration tests, and maybe get PK re-enabled on windows and mac too.

CMCDragonkai · 2023-10-17T21:08:41Z

The integration:linux which uses ubuntu still has a problem:

$ for f in ./builds/*-linux-*; do "$f"; done
/snapshot/Polykey-CLI/dist/polykey.js:62491
      throw new Error(`Failed requiring possible native bindings: ${prebuildTargets.concat(npmTargets)}`);
      ^
Error: Failed requiring possible native bindings: /snapshot/prebuild/quic-linux-x64.node,@matrixai/quic-linux-x64
    at requireBinding (/snapshot/Polykey-CLI/dist/polykey.js:62491:13)
    at node_modules/@matrixai/quic/dist/native/quiche.js (/snapshot/Polykey-CLI/dist/polykey.js:625[33](https://gitlab.com/MatrixAI/open-source/Polykey-CLI/-/jobs/5313359556#L33):29)
    at __require (/snapshot/Polykey-CLI/dist/polykey.js:13:50)
    at node_modules/@matrixai/quic/dist/native/index.js (/snapshot/Polykey-CLI/dist/polykey.js:62660:20)
    at __require (/snapshot/Polykey-CLI/dist/polykey.js:13:50)
    at node_modules/@matrixai/quic/dist/QUICSocket.js (/snapshot/Polykey-CLI/dist/polykey.js:6[34](https://gitlab.com/MatrixAI/open-source/Polykey-CLI/-/jobs/5313359556#L34)63:20)
    at __require (/snapshot/Polykey-CLI/dist/polykey.js:13:50)
    at node_modules/@matrixai/quic/dist/index.js (/snapshot/Polykey-CLI/dist/polykey.js:66268:24)
    at __require (/snapshot/Polykey-CLI/dist/polykey.js:13:50)
    at node_modules/polykey/dist/nodes/utils.js (/snapshot/Polykey-CLI/dist/polykey.js:70223:18)
Node.js v18.15.0

We saw this before. I think the script used to load the quic native binding doesn't seem to work after being esbuilt or something. Not sure need to check.

MatrixAI-Bot · 2023-10-17T21:09:30Z

Pipeline Attempt on 1040205173 for de054e5

https://gitlab.com/MatrixAI/open-source/Polykey-CLI/-/pipelines/1040205173

CMCDragonkai · 2023-10-17T22:47:19Z

The error comes from:

  throw new Error(
    `Failed requiring possible native bindings: ${prebuildTargets.concat(
      npmTargets,
    )}`,
  );

Which is in src/native/quiche.ts in the requireBinding function.

/**
 * Try require on all prebuild targets first, then
 * try require on all npm targets second.
 */
function requireBinding(targets: Array<string>): Quiche {
  const prebuildTargets = targets.map((target) =>
    path.join(prebuildPath, `quic-${target}.node`),
  );
  for (const prebuildTarget of prebuildTargets) {
    try {
      return require(prebuildTarget);
    } catch (e) {
      if (e.code !== 'MODULE_NOT_FOUND') throw e;
    }
  }
  const npmTargets = targets.map((target) => `@matrixai/quic-${target}`);
  for (const npmTarget of npmTargets) {
    try {
      return require(npmTarget);
    } catch (e) {
      if (e.code !== 'MODULE_NOT_FOUND') throw e;
    }
  }
  throw new Error(
    `Failed requiring possible native bindings: ${prebuildTargets.concat(
      npmTargets,
    )}`,
  );
}

The error message tells us that it is trying load native binding in:

/snapshot/prebuild/quic-linux-x64.node,@matrixai/quic-linux-x64

The first path is /snapshot/prebuild/quic-linux-x64.node which is a development-only path.

The second path is @matrixai/quic-linux-x64 which should be set up properly.

The reason is that:

The @matrixai/quic-linux-x64 is set as an external package for esbuild.
And also it's set as an asset for pkg in package.json, since the pkg.js script cannot auto-find it using node-gyp-build.

CMCDragonkai · 2023-10-17T22:51:16Z

I wonder if pkg when finally putting together the entire executable, does it support the ability to require @matrixai/quic-linux-x64?

It could be due to pkg not being able to maintain a VFS that enables the ability to require it.

It's possible that originally node-gyp-build does it differently, and so that's why it doesn't have this problem.

CMCDragonkai · 2023-10-17T22:53:02Z

I think I have an idea, it's possible the entire package.json is missing and that's why require('@matrixai/quic-linux/x64') does not work. Because I'm using the entire package name as the require path, and perhaps that requires the package.json too.

…ns the `package.json` is required to ensure that `main` can be resolved

MatrixAI-Bot · 2023-10-17T23:04:05Z

Pipeline Attempt on 1040374478 for 3612eb3

https://gitlab.com/MatrixAI/open-source/Polykey-CLI/-/pipelines/1040374478

CMCDragonkai · 2023-10-17T23:11:18Z

Yep that was the problem and was solved by adding package.json into the assets list too.

As a general matter it is somewhat more correct to depend on the special packages since the package.json uses main to dictate where the .node is.

However it is little inefficient to bundle the package.json.

One way is to change our custom requireBinding so it just tries to load the @matrixai/quic-linux-x64/node.napi.node instead. That would mean we wouldn't need the package.json. BTW this works because the special native packages don't use dist, otherwise @matrixai/quic-linux-x64/node.napi.node would be needed.

However with the ESM migration MatrixAI/TypeScript-Demo-Lib#32, node's import does not support native addons: https://nodejs.org/api/esm.html#no-addon-loading

The docs suggest using https://nodejs.org/api/process.html#processdlopenmodule-filename-flags instead.

So I think that would be more accurate, that means dropping the require and swapping to use dlopen instead, and therefore needing a deterministic path to look that up.

MatrixAI-Bot · 2023-10-17T23:30:23Z

Pipeline Attempt on 1040406103 for 1158a67

https://gitlab.com/MatrixAI/open-source/Polykey-CLI/-/pipelines/1040406103

CMCDragonkai · 2023-10-18T00:04:00Z

Ok remaining thing is:

CMCDragonkai · 2023-10-18T00:04:39Z

If that passes then:

Docker is working
Pkg - ubuntu is working
And it's all deployed to ECR

Then this should merge to master.

And then we can focus on the remaining tasks in testnet 6.

1. Making sure the docker integration tests pass in the relevant env parameters 2. extended timeouts for problimatic tests that do a password hash. Even though i'm sure we're passing in these options, the hashing still seems to be very slow. On the order of 10+ seconds. 3. Some tests were being skipped with `.skip` if this test was optionally disable based on the integration test it became `.skip.skip`. I've fixed this.

MatrixAI-Bot · 2023-10-18T02:31:54Z

Pipeline Attempt on 1040496342 for 5354256

https://gitlab.com/MatrixAI/open-source/Polykey-CLI/-/pipelines/1040496342

MatrixAI-Bot · 2023-10-18T02:55:19Z

Pipeline Attempt on 1040506440 for e029c8d

https://gitlab.com/MatrixAI/open-source/Polykey-CLI/-/pipelines/1040506440

MatrixAI-Bot · 2023-10-18T03:15:49Z

Pipeline Succeeded on 1040506440 for e029c8d

https://gitlab.com/MatrixAI/open-source/Polykey-CLI/-/pipelines/1040506440

tegefaulkes and others added 18 commits July 21, 2023 11:07

feat: copying Typescript-demo-lib as a template

3d3de6e

feat: copied over CLI bin code

825f7cd

[ci skip]

feat: copied over CLI bin tests

f33f8d6

[ci skip]

fix: fixing up tests

c2741dd

[ci skip]

fix: updated PolykeyClient usage

f8cbfc0

[ci skip]

tests: fixed tests

df836c7

[ci skip]

tests: disabled vaults clone test for now

dfe66c9

[ci skip]

ci: updated CI jobs

fac79cb

test: fixed broken mocking

386cfeb

refactor: moving nat and integration tests

4b84ea8

[ci skip]

build: moving some README.md information and updating package.json

a42ff4c

fix: fixing up polykey dependency

249c70c

build: copied over configs

08747dc

tests: disabled nat tests

582115d

These are pending updates in agent migration stage 2.

tests: all tests passing

43cb1e0

fix: dependency and CI fixes

71a9df8

build: getting pkg working

75717b2

Merge pull request #2 from MatrixAI/feature-CLI_migration

15c755f

CLI extracted from Polykey and migrated to Polykey-CLI

MatrixAI-Bot self-assigned this Aug 7, 2023

tegefaulkes added 2 commits August 7, 2023 17:05

ci: disabled windows and macos build job

bdb6dc7

build: regenerating package-lock.json as version 2 to support `node…

1380d13

…2nix`

fix: migrated nodemon from Polykey

6db2859

ci: npmDepsHash can only be calculated within nix-shell

8f80d1f

ci: shell.nix needs awscli2 for deployment

41273c6

ci: recovered the deploy-service.sh

138fcdf

ci: integration:nix should be exexcuting bin/polykey

de054e5

ci: new native bindings rely on requiring the package name, which mea…

3612eb3

…ns the `package.json` is required to ensure that `main` can be resolved

ci: no need to have pkg bundle tslib and tsyringe

1158a67

This was referenced Oct 17, 2023

Replace threads with internal implementation MatrixAI/js-workers#16

Open

Replace custom require loader with process.dlopen and using static path to node.napi.node MatrixAI/js-quic#71

Open

fix: missed one fix for ping.test.ts

e029c8d

MatrixAI-Bot merged commit e029c8d into master Oct 18, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ci: merge staging to master #4

ci: merge staging to master #4

MatrixAI-Bot commented Aug 7, 2023

MatrixAI-Bot commented Aug 7, 2023

tegefaulkes commented Aug 7, 2023

MatrixAI-Bot commented Aug 7, 2023

tegefaulkes commented Aug 7, 2023

MatrixAI-Bot commented Aug 11, 2023

CMCDragonkai commented Aug 15, 2023

tegefaulkes commented Aug 15, 2023

CMCDragonkai commented Aug 15, 2023

MatrixAI-Bot commented Oct 17, 2023

MatrixAI-Bot commented Oct 17, 2023

MatrixAI-Bot commented Oct 17, 2023

MatrixAI-Bot commented Oct 17, 2023

CMCDragonkai commented Oct 17, 2023

CMCDragonkai commented Oct 17, 2023

CMCDragonkai commented Oct 17, 2023

MatrixAI-Bot commented Oct 17, 2023

CMCDragonkai commented Oct 17, 2023 •

edited

Loading

CMCDragonkai commented Oct 17, 2023

CMCDragonkai commented Oct 17, 2023

MatrixAI-Bot commented Oct 17, 2023

CMCDragonkai commented Oct 17, 2023

MatrixAI-Bot commented Oct 17, 2023

CMCDragonkai commented Oct 18, 2023

CMCDragonkai commented Oct 18, 2023

MatrixAI-Bot commented Oct 18, 2023

MatrixAI-Bot commented Oct 18, 2023

MatrixAI-Bot commented Oct 18, 2023

ci: merge staging to master #4

ci: merge staging to master #4

Conversation

MatrixAI-Bot commented Aug 7, 2023

MatrixAI-Bot commented Aug 7, 2023

tegefaulkes commented Aug 7, 2023

MatrixAI-Bot commented Aug 7, 2023

tegefaulkes commented Aug 7, 2023

MatrixAI-Bot commented Aug 11, 2023

CMCDragonkai commented Aug 15, 2023

tegefaulkes commented Aug 15, 2023

CMCDragonkai commented Aug 15, 2023

MatrixAI-Bot commented Oct 17, 2023

MatrixAI-Bot commented Oct 17, 2023

MatrixAI-Bot commented Oct 17, 2023

MatrixAI-Bot commented Oct 17, 2023

CMCDragonkai commented Oct 17, 2023

CMCDragonkai commented Oct 17, 2023

CMCDragonkai commented Oct 17, 2023

MatrixAI-Bot commented Oct 17, 2023

CMCDragonkai commented Oct 17, 2023 • edited Loading

CMCDragonkai commented Oct 17, 2023

CMCDragonkai commented Oct 17, 2023

MatrixAI-Bot commented Oct 17, 2023

CMCDragonkai commented Oct 17, 2023

MatrixAI-Bot commented Oct 17, 2023

CMCDragonkai commented Oct 18, 2023

CMCDragonkai commented Oct 18, 2023

MatrixAI-Bot commented Oct 18, 2023

MatrixAI-Bot commented Oct 18, 2023

MatrixAI-Bot commented Oct 18, 2023

CMCDragonkai commented Oct 17, 2023 •

edited

Loading