Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

improve: wasm build #3074

Merged
merged 3 commits into from
Apr 22, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 10 additions & 3 deletions build/Dockerfile
Original file line number Diff line number Diff line change
@@ -1,18 +1,25 @@
FROM node:21-alpine@sha256:6d0f18a1c67dc218c4af50c21256616286a53c09e500fadf025b6d342e1c90ae
FROM node:21-alpine3.19@sha256:ad255c65652e8e99ce0b9d9fc52eee3eae85f445b192f6f9e49a1305c77b2ba6

ARG UID=1000
ARG GID=1000
ARG BINARYEN_VERSION=116

RUN apk add -U clang lld wasi-sdk
RUN mkdir /home/node/undici

WORKDIR /home/node/undici

RUN wget https://github.com/WebAssembly/binaryen/releases/download/version_$BINARYEN_VERSION/binaryen-version_$BINARYEN_VERSION-x86_64-linux.tar.gz && \
tar -zxvf binaryen-version_$BINARYEN_VERSION-x86_64-linux.tar.gz binaryen-version_$BINARYEN_VERSION/bin/wasm-opt && \
mv binaryen-version_$BINARYEN_VERSION/bin/wasm-opt ./ && \
rm binaryen-version_$BINARYEN_VERSION-x86_64-linux.tar.gz && \
rm -rf binaryen-version_$BINARYEN_VERSION && \
chmod +x ./wasm-opt

COPY package.json .

COPY build build
COPY deps deps
COPY lib lib

RUN npm i

USER node
94 changes: 45 additions & 49 deletions build/wasm.js
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

const { execSync } = require('node:child_process')
const { writeFileSync, readFileSync } = require('node:fs')
const { join, resolve, basename } = require('node:path')
const { join, resolve } = require('node:path')

const ROOT = resolve(__dirname, '../')
const WASM_SRC = resolve(__dirname, '../deps/llhttp')
Expand All @@ -15,19 +15,40 @@ let WASM_CFLAGS = process.env.WASM_CFLAGS || '--sysroot=/usr/share/wasi-sysroot
let WASM_LDFLAGS = process.env.WASM_LDFLAGS || ''
const WASM_LDLIBS = process.env.WASM_LDLIBS || ''

const EXTERNAL_PATH = process.env.EXTERNAL_PATH

// These are relevant for undici and should not be overridden
WASM_CFLAGS += ' -Ofast -fno-exceptions -fvisibility=hidden -mexec-model=reactor'
WASM_LDFLAGS += ' -Wl,-error-limit=0 -Wl,-O3 -Wl,--lto-O3 -Wl,--strip-all'
WASM_LDFLAGS += ' -Wl,--allow-undefined -Wl,--export-dynamic -Wl,--export-table'
WASM_LDFLAGS += ' -Wl,--export=malloc -Wl,--export=free -Wl,--no-entry'

const WASM_OPT_FLAGS = '-O4 --converge --strip-debug --strip-dwarf --strip-producers'
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mcollina

This is where we define O4 for optimizing the wasm file.


const writeWasmChunk = (path, dest) => {
const base64 = readFileSync(join(WASM_OUT, path)).toString('base64')
writeFileSync(join(WASM_OUT, dest), `'use strict'

const { Buffer } = require('node:buffer')

module.exports = Buffer.from('${base64}', 'base64')
`)
}

let platform = process.env.WASM_PLATFORM
if (!platform && process.argv[2]) {
platform = execSync('docker info -f "{{.OSType}}/{{.Architecture}}"').toString().trim()
}

if (process.argv[2] === '--rm') {
const cmd = 'docker image rm llhttp_wasm_builder'

console.log(`> ${cmd}\n\n`)
try {
execSync(cmd, { stdio: 'inherit' })
} catch (e) {}

process.exit(0)
}

if (process.argv[2] === '--prebuild') {
const cmd = `docker build --platform=${platform.toString().trim()} -t llhttp_wasm_builder -f ${DOCKERFILE} ${ROOT}`

Expand Down Expand Up @@ -59,50 +80,25 @@ if (hasApk) {
console.log('Failed to generate build environment information')
process.exit(-1)
}
writeFileSync(join(WASM_OUT, 'wasm_build_env.txt'), buildInfo)
}

const writeWasmChunk = EXTERNAL_PATH
? (path, dest) => {
const base64 = readFileSync(join(WASM_OUT, path)).toString('base64')
writeFileSync(join(WASM_OUT, dest), `
const { Buffer } = require('node:buffer')

module.exports = Buffer.from('${base64}', 'base64')
`)
}
: (path, dest) => {
writeFileSync(join(WASM_OUT, dest), `
const { fs } = require('node:fs')

module.exports = fs.readFileSync(require.resolve('./${basename(path)}'))
`)
}

// Build wasm binary
execSync(`${WASM_CC} ${WASM_CFLAGS} ${WASM_LDFLAGS} \
${join(WASM_SRC, 'src')}/*.c \
-I${join(WASM_SRC, 'include')} \
-o ${join(WASM_OUT, 'llhttp.wasm')} \
${WASM_LDLIBS}`, { stdio: 'inherit' })

writeWasmChunk('llhttp.wasm', 'llhttp-wasm.js')

// Build wasm simd binary
execSync(`${WASM_CC} ${WASM_CFLAGS} -msimd128 ${WASM_LDFLAGS} \
${join(WASM_SRC, 'src')}/*.c \
-I${join(WASM_SRC, 'include')} \
-o ${join(WASM_OUT, 'llhttp_simd.wasm')} \
${WASM_LDLIBS}`, { stdio: 'inherit' })

writeWasmChunk('llhttp_simd.wasm', 'llhttp_simd-wasm.js')

if (EXTERNAL_PATH) {
writeFileSync(join(ROOT, 'loader.js'), `
'use strict'

globalThis.__UNDICI_IS_NODE__ = true
module.exports = require('node:module').createRequire('${EXTERNAL_PATH}/loader.js')('./index-fetch.js')
delete globalThis.__UNDICI_IS_NODE__
`)
Comment on lines -62 to -107
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

^ I meant this.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, then it has to be added back + a big fat comment to not remove it because of fedora

console.log(buildInfo)

// Build wasm binary
execSync(`${WASM_CC} ${WASM_CFLAGS} ${WASM_LDFLAGS} \
${join(WASM_SRC, 'src')}/*.c \
-I${join(WASM_SRC, 'include')} \
-o ${join(WASM_OUT, 'llhttp.wasm')} \
${WASM_LDLIBS}`, { stdio: 'inherit' })
Dismissed Show dismissed Hide dismissed

execSync(`./wasm-opt ${WASM_OPT_FLAGS} -o ${join(WASM_OUT, 'llhttp.wasm')} ${join(WASM_OUT, 'llhttp.wasm')}`, { stdio: 'inherit' })
Dismissed Show dismissed Hide dismissed
writeWasmChunk('llhttp.wasm', 'llhttp-wasm.js')

// Build wasm simd binary
execSync(`${WASM_CC} ${WASM_CFLAGS} -msimd128 ${WASM_LDFLAGS} \
${join(WASM_SRC, 'src')}/*.c \
-I${join(WASM_SRC, 'include')} \
-o ${join(WASM_OUT, 'llhttp_simd.wasm')} \
${WASM_LDLIBS}`, { stdio: 'inherit' })
Dismissed Show dismissed Hide dismissed

execSync(`./wasm-opt ${WASM_OPT_FLAGS} --enable-simd -o ${join(WASM_OUT, 'llhttp_simd.wasm')} ${join(WASM_OUT, 'llhttp_simd.wasm')}`, { stdio: 'inherit' })
Dismissed Show dismissed Hide dismissed
writeWasmChunk('llhttp_simd.wasm', 'llhttp_simd-wasm.js')
}
Empty file added lib/llhttp/.gitkeep
Empty file.
199 changes: 0 additions & 199 deletions lib/llhttp/constants.d.ts

This file was deleted.

1 change: 0 additions & 1 deletion lib/llhttp/constants.js.map

This file was deleted.

4 changes: 3 additions & 1 deletion lib/llhttp/llhttp-wasm.js

Large diffs are not rendered by default.

Binary file modified lib/llhttp/llhttp.wasm
Binary file not shown.
4 changes: 3 additions & 1 deletion lib/llhttp/llhttp_simd-wasm.js

Large diffs are not rendered by default.

Binary file modified lib/llhttp/llhttp_simd.wasm
Binary file not shown.
4 changes: 0 additions & 4 deletions lib/llhttp/utils.d.ts

This file was deleted.

1 change: 0 additions & 1 deletion lib/llhttp/utils.js.map

This file was deleted.

32 changes: 0 additions & 32 deletions lib/llhttp/wasm_build_env.txt

This file was deleted.

Loading