Skip to content

Commit

Permalink
tools: make undici updater build wasm from src
Browse files Browse the repository at this point in the history
- modify updater to include all files required to rebuild
  wasm from what's stored in the deps/undici directory
- modify updater to build wasm from source while updating

Signed-off-by: Michael Dawson <mdawson@devrus.com>
PR-URL: #54128
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Marco Ippolito <marcoippolito54@gmail.com>
Reviewed-By: Robert Nagy <ronagy@icloud.com>
  • Loading branch information
mhdawson authored and targos committed Aug 14, 2024
1 parent 581c155 commit b93c6d9
Show file tree
Hide file tree
Showing 28 changed files with 20,603 additions and 33 deletions.
13 changes: 13 additions & 0 deletions deps/undici/src/.c8rc.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
{
"all": true,
"reporter": [
"lcov",
"text",
"html",
"text-summary"
],
"include": [
"lib/**/*.js",
"index.js"
]
}
8 changes: 8 additions & 0 deletions deps/undici/src/.dockerignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
# Ignore everything but the stuff following the `*` with the `!`
# See https://docs.docker.com/engine/reference/builder/#dockerignore-file

*
!package.json
!lib
!deps
!build
9 changes: 9 additions & 0 deletions deps/undici/src/.editorconfig
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
# https://editorconfig.org/

root = true

[*]
indent_size = 2
indent_style = space
insert_final_newline = true
trim_trailing_whitespace = true
86 changes: 86 additions & 0 deletions deps/undici/src/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,86 @@
# Logs
logs
*.log
npm-debug.log*
yarn-debug.log*
yarn-error.log*

# Runtime data
pids
*.pid
*.seed
*.pid.lock

# Directory for instrumented libs generated by jscoverage/JSCover
lib-cov

# Coverage directory used by tools like istanbul
coverage

# nyc test coverage
.nyc_output

# Grunt intermediate storage (http://gruntjs.com/creating-plugins#storing-task-files)
.grunt

# Bower dependency directory (https://bower.io/)
bower_components

# node-waf configuration
.lock-wscript

# Compiled binary addons (https://nodejs.org/api/addons.html)
build/Release

# Dependency directories
node_modules/
jspm_packages/

# TypeScript v1 declaration files
typings/

# Optional npm cache directory
.npm

# Optional eslint cache
.eslintcache

# Optional REPL history
.node_repl_history

# Output of 'npm pack'
*.tgz

# Yarn Integrity file
.yarn-integrity

# dotenv environment variables file
.env

# next.js build output
.next

# lock files
package-lock.json
yarn.lock

# IDE files
.idea
.vscode

*0x
*clinic*

# Fuzzing
corpus/
crash-*
fuzz-results-*.json

# Bundle output
undici-fetch.js
/test/imports/undici-import.js

# .npmrc has platform specific value for windows
.npmrc

.tap
13 changes: 13 additions & 0 deletions deps/undici/src/.npmignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
*
!lib/**/*
!index.js
!index-fetch.js

# The wasm files are stored as base64 strings in the corresponding .js files
lib/llhttp/llhttp_simd.wasm
lib/llhttp/llhttp.wasm

!types/**/*
!index.d.ts
!docs/docs/**/*
!scripts/strip-comments.js
6 changes: 6 additions & 0 deletions deps/undici/src/CODE_OF_CONDUCT.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
# Code of Conduct

Undici is committed to upholding the Node.js Code of Conduct.

The Node.js Code of Conduct document can be found at
https://github.com/nodejs/admin/blob/main/CODE_OF_CONDUCT.md
213 changes: 213 additions & 0 deletions deps/undici/src/CONTRIBUTING.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,213 @@
# Contributing to Undici

* [Guides](#guides)
* [Update `llhttp`](#update-llhttp)
* [Lint](#lint)
* [Test](#test)
* [Coverage](#coverage)
* [Releases](#releases)
* [Update `WPTs`](#update-wpts)
* [Building for externally shared node builtins](#external-builds)
* [Benchmarks](#benchmarks
* [Documentation](#documentation)
* [Developer's Certificate of Origin 1.1](#developers-certificate-of-origin)
* [Moderation Policy](#moderation-policy)

<a id="guides"></a>
## Guides

This is a collection of guides on how to run and update `undici`, and how to run different parts of the project.

<a id="update-llhttp"></a>
### Update `llhttp`

The HTTP parser used by `undici` is a WebAssembly build of [`llhttp`](https://github.com/nodejs/llhttp).

While the project itself provides a way to compile targeting WebAssembly, at the moment we embed the sources
directly and compile the module in `undici`.

The `deps/llhttp/include` folder contains the C header files, while the `deps/llhttp/src` folder contains
the C source files needed to compile the module.

The `lib/llhttp` folder contains the `.js` transpiled assets required to implement a parser.

The following are the steps required to perform an update.

#### Clone the [llhttp](https://github.com/nodejs/llhttp) project

```bash
git clone git@github.com:nodejs/llhttp.git

cd llhttp
```

#### Checkout a `llhttp` release

```bash
git checkout <tag>
```

#### Install the `llhttp` dependencies

```bash
npm i
```

#### Run the wasm build script

> This requires [docker](https://www.docker.com/) installed on your machine.
```bash
npm run build-wasm
```

#### Copy the sources to `undici`

```bash
cp build/wasm/*.js <your-path-to-undici>/lib/llhttp/

cp build/wasm/*.js.map <your-path-to-undici>/lib/llhttp/

cp build/wasm/*.d.ts <your-path-to-undici>/lib/llhttp/

cp src/native/api.c src/native/http.c build/c/llhttp.c <your-path-to-undici>/deps/llhttp/src/

cp src/native/api.h build/llhttp.h <your-path-to-undici>/deps/llhttp/include/
```

#### Build the WebAssembly module in `undici`

> This requires [docker](https://www.docker.com/) installed on your machine.
```bash
cd <your-path-to-undici>

npm run build:wasm
```

#### Commit the contents of lib/llhttp

Create a commit which includes all of the updated files in lib/llhttp.

<a id="update-wpts"></a>
### Update `WPTs`

`undici` runs a subset of the [`web-platform-tests`](https://github.com/web-platform-tests/wpt).

### Requirements:
- [Node core utils](https://github.com/nodejs/node-core-utils) setup with credentials.

To update every test, run the following commands. Typically you would only need to update the tests in a specific directory.

```bash
git node wpt resources
git node wpt interfaces
git node wpt common
git node wpt fetch
git node wpt FileAPI
git node wpt xhr
git node wpt websockets
git node wpt mimesniff
git node wpt storage
git node wpt service-workers
git node wpt eventsource
```

#### Run the tests

Run the tests to ensure that any new failures are marked as such.

You can mark tests as failing in their corresponding [status](./test/wpt/status) file.

```bash
npm run test:wpt
```

<a id="lint"></a>
### Lint

```bash
npm run lint
```

<a id="test"></a>
### Test

```bash
npm run test
```

<a id="coverage"></a>
### Coverage

```bash
npm run coverage
```

<a id="releases"></a>
### Issuing Releases

Release is automatic on commit to main which bumps the package.json version field.
Use the "Create release PR" github action to generate a release PR.

<a id="external-builds"></a>
### Building for externally shared node builtins

If you are packaging `undici` for a distro, this might help if you would like to use
an unbundled version instead of bundling one in `libnode.so`.

To enable this, pass `EXTERNAL_PATH=/path/to/global/node_modules/undici` to `build/wasm.js`.
Pass this path with `loader.js` appended to `--shared-builtin-undici/undici-path` in Node.js's `configure.py`.
If building on a non-Alpine Linux distribution, you may need to also set the `WASM_CC`, `WASM_CFLAGS`, `WASM_LDFLAGS` and `WASM_LDLIBS` environment variables before running `build/wasm.js`.

<a id="benchmarks"></a>
### Benchmarks

```bash
cd benchmarks && npm i && npm run bench
```

The benchmarks will be available at `http://localhost:3042`.

<a id="documentation"></a>
### Documentation

```bash
cd docs && npm i && npm run serve
```

The documentation will be available at `http://localhost:3000`.

<a id="developers-certificate-of-origin"></a>
## Developer's Certificate of Origin 1.1

By making a contribution to this project, I certify that:

* (a) The contribution was created in whole or in part by me and I
have the right to submit it under the open source license
indicated in the file; or

* (b) The contribution is based upon previous work that, to the best
of my knowledge, is covered under an appropriate open source
license and I have the right under that license to submit that
work with modifications, whether created in whole or in part
by me, under the same open source license (unless I am
permitted to submit under a different license), as indicated
in the file; or

* (c) The contribution was provided directly to me by some other
person who certified (a), (b) or (c) and I have not modified
it.

* (d) I understand and agree that this project and the contribution
are public and that a record of the contribution (including all
personal information I submit with it, including my sign-off) is
maintained indefinitely and may be redistributed consistent with
this project or the open source license(s) involved.

<a id="moderation-policy"></a>
### Moderation Policy

The [Node.js Moderation Policy] applies to this project.

[Node.js Moderation Policy]: https://github.com/nodejs/admin/blob/main/Moderation-Policy.md
Loading

0 comments on commit b93c6d9

Please sign in to comment.