Skip to content

Commit

Permalink
[wasm] Emscripten 3.1.1 bump (dotnet#63894)
Browse files Browse the repository at this point in the history
* Initial changes for emscripten 2.0.34

* Use emcc-link.rsp in build targets

* Use updated docker images

* Fix compiler warnings

Co-authored-by: Zoltan Varga <vargaz@gmail.com>

* Put `--profiling-funcs` to `_EmccLinkFlags`

* Fix build

    src/mono/mono/mini/mini-runtime.c:3407:25: error: ‘invoke’ undeclared (first use in this function); did you mean ‘revoke’?
       3407 |                         invoke = mono_marshal_get_runtime_invoke_dynamic ();

* Add shell to the environment

Environment setting https://github.com/emscripten-core/emscripten/blob/2.0.34/src/settings.js#L616-L641

From emscripten 2.0.25 release notes

    - Support for the 'shell' environment is now disabled by default.  Running under
      `d8`, `js`, or `jsc` is not something that most emscripten users ever want to
      do, so including the support code is, more often than not, unnecessary.  Users
      who want shell support can enable it by including 'shell' in `-s ENVIRONMENT`
      (dotnet#14535).

Example of the the size increase for bench sample:

    -a---          12/10/2021  3:35 PM         382113 dotnet.js
    -a---          12/13/2021 10:37 AM         383589 dotnet.js

* Add emcc-link.rsp to PlatformManifestFileEntry

* Feedback

https://github.com/emscripten-core/emscripten/blob/2fda25eea756c78c8cb024aa5b6c2b188bf7990f/src/settings.js#L1173-L1176

-s EXPORT_ES6 is link option

* Bump emscripten version

* Bump llvm package version and use its libclang

* Use newer docker images with emscripten 3.1.1

* Remove unused variable

* Add note about icu repo

* Add comment to improve code readability

* Bump the ICU and emsdk versions

* disable non-wasm builds

* disable wasm jobs in runtime, and runtime-staging, as we'll be running runtime-wasm manually

* [wasm] Re-enable tests that were disabled due to the earlier emsdk bump

System.Linq.Tests.csproj: dotnet#64774

System.Text.Json.Tests.csproj: dotnet#64775

System.Text.RegularExpressions.Tests.csproj: dotnet#64769

System.Threading.Channels.Tests.csproj: dotnet#65012

System.Collections.Immutable.Tests.csproj: dotnet#65335

* [wasm] Disable nodejs tests

They are still unstable, and fail with:

```
[01:27:29] fail: /datadisks/disk1/work/A3DA08F7/w/B1B30976/e/test-main.js:97
[01:27:29] fail: if (typeof globalThis.crypto === 'undefined') {
[01:27:29] fail: ^
[01:27:29] fail:
[01:27:29] fail: ReferenceError: globalThis is not defined
[01:27:29] fail:     at Object.<anonymous> (/datadisks/disk1/work/A3DA08F7/w/B1B30976/e/test-main.js:97:1)
[01:27:29] fail:     at Module._compile (internal/modules/cjs/loader.js:778:30)
[01:27:29] fail:     at Object.Module._extensions..js (internal/modules/cjs/loader.js:789:10)
[01:27:29] fail:     at Module.load (internal/modules/cjs/loader.js:653:32)
[01:27:29] fail:     at tryModuleLoad (internal/modules/cjs/loader.js:593:12)
[01:27:29] fail:     at Function.Module._load (internal/modules/cjs/loader.js:585:3)
[01:27:29] fail:     at Function.Module.runMain (internal/modules/cjs/loader.js:831:12)
[01:27:29] fail:     at startup (internal/bootstrap/node.js:283:19)
[01:27:29] fail:     at bootstrapNodeJSCore (internal/bootstrap/node.js:623:3)
```

Issue: dotnet#65485

* fix wbt build

* Update path to node in helix commands.

* disable non-wasm builds

* [wasm] Make emscripten version mismatch an error on CI

* Bump emscripten packages used to preview3, which has emsdk 3.1.1

* Update emscripten package version in Versions.props tooo

* Re-enable wasm jobs

* enable non-wasm jobs

* Add more missing exports

* __dl_seterr shouldn't be needed

* Fix MicrosoftNETCoreRuntimeICUTransportVersion

* Put back the 3.1.1 docker images after merge

* Revert "fix wbt build"

This reverts commit f7c77d9.

* Revert "[wasm] Disable nodejs tests"

This reverts commit 7b90965.

* Use newer ICU package, last that still had 3.1.1

* Try different helix queue parameter

* Revert "Try different helix queue parameter"

This reverts commit a34230f.

* Fix helix queue with docker tag

* Try to set PYTHONPATH for emscripten

* [wasm] Do not set _WasmDevel=true for blazor projects

* Increase timeout

The AOT tests are canceled after timeout

* Fix WBT tests

Co-authored-by: Ankit Jain <radical@gmail.com>

* [wasm] Export more functions

`htons,ntohs,_get_daylight,_get_timezone,_get_tzname`

.. to fix:

```
EXEC : warning : undefined symbol: htons (referenced by $inetPton6__deps: ['htons','$jstoi_q'], referenced by $writeSockaddr__deps: ['$Sockets','$inetPton4','$inetPton6','$zeroMemory'], referenced by __syscall_accept4__deps: ['$getSocketFromFD','$writeSockaddr','$DNS','$SYSCALLS'], referenced by top-level compiled C/C++ code) [C:\helix\work\workitem\e\publish\ProxyProjectForAOTOnHelix.proj]
EXEC : warning : undefined symbol: ntohs (referenced by $inetNtop6__deps: ['$inetNtop4','ntohs'], referenced by $readSockaddr__deps: ['$Sockets','$inetNtop4','$inetNtop6','ntohs'], referenced by $getSocketAddress__deps: ['$readSockaddr','$FS','$DNS'], referenced by __syscall_bind__deps: ['$getSocketFromFD','$getSocketAddress','$SYSCALLS'], referenced by top-level compiled C/C++ code) [C:\helix\work\workitem\e\publish\ProxyProjectForAOTOnHelix.proj]
EXEC : warning : undefined symbol: _get_daylight (referenced by tzset_impl__deps: ['_get_daylight','_get_timezone','_get_tzname'], referenced by tzset__deps: ['tzset_impl'], referenced by localtime_r__deps: ['tzset'], referenced by top-level compiled C/C++ code) [C:\helix\work\workitem\e\publish\ProxyProjectForAOTOnHelix.proj]
EXEC : warning : undefined symbol: _get_timezone (referenced by tzset_impl__deps: ['_get_daylight','_get_timezone','_get_tzname'], referenced by tzset__deps: ['tzset_impl'], referenced by localtime_r__deps: ['tzset'], referenced by top-level compiled C/C++ code) [C:\helix\work\workitem\e\publish\ProxyProjectForAOTOnHelix.proj]
EXEC : warning : undefined symbol: _get_tzname (referenced by tzset_impl__deps: ['_get_daylight','_get_timezone','_get_tzname'], referenced by tzset__deps: ['tzset_impl'], referenced by localtime_r__deps: ['tzset'], referenced by top-level compiled C/C++ code) [C:\helix\work\workitem\e\publish\ProxyProjectForAOTOnHelix.proj]
```

```
Starting:    Microsoft.Extensions.Logging.Tests.dll
Message1
:
testKey3:testValue, testKey2:, testKey1:
missing function: ntohs
Error
    at Object.onAbort (test-main.js:176:60)
    at abort (./dotnet.js:1316:24)
    at _ntohs (./dotnet.js:4990:35)
    at readSockaddr (./dotnet.js:5091:18)
    ...
```

* fix exported function names

Co-authored-by: Radek Doulik <radekdoulik@google.com>
Co-authored-by: Zoltan Varga <vargaz@gmail.com>
Co-authored-by: Larry Ewing <lewing@microsoft.com>
Co-authored-by: Ankit Jain <radical@gmail.com>
Co-authored-by: Marek Fišera <mara@neptuo.com>
  • Loading branch information
6 people committed Mar 30, 2022
1 parent 560546b commit de0676c
Show file tree
Hide file tree
Showing 11 changed files with 15 additions and 12 deletions.
8 changes: 4 additions & 4 deletions eng/Version.Details.xml
Original file line number Diff line number Diff line change
@@ -1,16 +1,16 @@
<Dependencies>
<ProductDependencies>
<Dependency Name="Microsoft.NETCore.Runtime.ICU.Transport" Version="7.0.0-preview.3.22157.1">
<Dependency Name="Microsoft.NETCore.Runtime.ICU.Transport" Version="7.0.0-preview.3.22121.1">
<Uri>https://github.com/dotnet/icu</Uri>
<Sha>5416401e2e54d6ce9bef7f0c4a5fe25088c42652</Sha>
<Sha>d3c9d26f8e787253536d38504d3ac9e307573bfb</Sha>
</Dependency>
<Dependency Name="System.Net.MsQuic.Transport" Version="7.0.0-alpha.1.22160.2">
<Uri>https://github.com/dotnet/msquic</Uri>
<Sha>4fe73fcad4b0af91cc49fd1f5576f0111f15f6fd</Sha>
</Dependency>
<Dependency Name="Microsoft.NET.Workload.Emscripten.Manifest-7.0.100" Version="7.0.0-preview.3.22167.1">
<Dependency Name="Microsoft.NET.Workload.Emscripten.Manifest-7.0.100" Version="7.0.0-preview.3.22121.1">
<Uri>https://github.com/dotnet/emsdk</Uri>
<Sha>d2be7d0bf644d9608bf8180d706483238d85f13a</Sha>
<Sha>b8e71431b3eaa024afad7886699f8ac6bca2ba01</Sha>
</Dependency>
<Dependency Name="System.ServiceModel.Primitives" Version="4.9.0-rc2.21473.1">
<Uri>https://github.com/dotnet/wcf</Uri>
Expand Down
4 changes: 2 additions & 2 deletions eng/Versions.props
Original file line number Diff line number Diff line change
Expand Up @@ -171,7 +171,7 @@
<MicrosoftNETILLinkTasksVersion>7.0.100-1.22168.6</MicrosoftNETILLinkTasksVersion>
<MicrosoftNETILLinkAnalyzerPackageVersion>$(MicrosoftNETILLinkTasksVersion)</MicrosoftNETILLinkAnalyzerPackageVersion>
<!-- ICU -->
<MicrosoftNETCoreRuntimeICUTransportVersion>7.0.0-preview.3.22157.1</MicrosoftNETCoreRuntimeICUTransportVersion>
<MicrosoftNETCoreRuntimeICUTransportVersion>7.0.0-preview.3.22121.1</MicrosoftNETCoreRuntimeICUTransportVersion>
<!-- MsQuic -->
<SystemNetMsQuicTransportVersion>7.0.0-alpha.1.22160.2</SystemNetMsQuicTransportVersion>
<!-- Mono LLVM -->
Expand All @@ -184,7 +184,7 @@
<runtimeosx1012x64MicrosoftNETCoreRuntimeMonoLLVMSdkVersion>11.1.0-alpha.1.22164.2</runtimeosx1012x64MicrosoftNETCoreRuntimeMonoLLVMSdkVersion>
<runtimeosx1012x64MicrosoftNETCoreRuntimeMonoLLVMToolsVersion>11.1.0-alpha.1.22164.2</runtimeosx1012x64MicrosoftNETCoreRuntimeMonoLLVMToolsVersion>
<!-- emscripten / Node -->
<MicrosoftNETWorkloadEmscriptenManifest70100Version>7.0.0-preview.3.22167.1</MicrosoftNETWorkloadEmscriptenManifest70100Version>
<MicrosoftNETWorkloadEmscriptenManifest70100Version>7.0.0-preview.3.22121.1</MicrosoftNETWorkloadEmscriptenManifest70100Version>
<MicrosoftNETRuntimeEmscriptenVersion>$(MicrosoftNETWorkloadEmscriptenManifest70100Version)</MicrosoftNETRuntimeEmscriptenVersion>
<!-- workloads -->
<SwixPackageVersion>1.1.87-gba258badda</SwixPackageVersion>
Expand Down
2 changes: 1 addition & 1 deletion eng/pipelines/common/platform-matrix.yml
Original file line number Diff line number Diff line change
Expand Up @@ -303,7 +303,7 @@ jobs:
targetRid: browser-wasm
platform: Browser_wasm
container:
image: ubuntu-18.04-webassembly-20210707133424-12f133e
image: ubuntu-18.04-webassembly-20220118141522-180197c
registry: mcr
jobParameters:
hostedOs: Linux
Expand Down
2 changes: 1 addition & 1 deletion eng/pipelines/common/templates/wasm-library-tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ jobs:
testGroup: innerloop
nameSuffix: LibraryTests${{ parameters.nameSuffix }}
buildArgs: -s mono+libs+host+packs+libs.tests -c $(_BuildConfig) /p:ArchiveTests=true /p:BrowserHost=$(_hostedOs) ${{ parameters.runSmokeOnlyArg }} ${{ parameters.extraBuildArgs }}
timeoutInMinutes: 180
timeoutInMinutes: 240
# always run for runtime-wasm builds (triggered manually)
# Always run for rolling builds
# Else run on path changes
Expand Down
2 changes: 1 addition & 1 deletion eng/pipelines/libraries/helix-queues-setup.yml
Original file line number Diff line number Diff line change
Expand Up @@ -182,6 +182,6 @@ jobs:

# WebAssembly windows
- ${{ if eq(parameters.platform, 'Browser_wasm_win') }}:
- (Windows.Server.Core.1909.Amd64.Open)windows.10.amd64.server20h2.open@mcr.microsoft.com/dotnet-buildtools/prereqs:windowsservercore-2004-helix-webassembly-amd64-20210702131541-6837048
- (Windows.Amd64.Server2022.Open)windows.amd64.server2022.open@mcr.microsoft.com/dotnet-buildtools/prereqs:windowsservercore-ltsc2022-helix-webassembly-20220118164817-dc6e6f1

${{ insert }}: ${{ parameters.jobParameters }}
1 change: 1 addition & 0 deletions src/mono/mono/component/debugger-agent.c
Original file line number Diff line number Diff line change
Expand Up @@ -414,6 +414,7 @@ static gboolean buffer_replies;
DebuggerTlsData *tls; \
tls = (DebuggerTlsData *)mono_native_tls_get_value (debugger_tls_id);
#else
/* the thread argument is omitted on wasm, to avoid compiler warning */
#define GET_TLS_DATA_FROM_THREAD(...) \
DebuggerTlsData *tls; \
tls = &debugger_wasm_thread;
Expand Down
1 change: 1 addition & 0 deletions src/mono/wasm/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -188,6 +188,7 @@ Bumping Emscripten version involves these steps:
* update https://github.com/dotnet/runtime/blob/main/src/mono/wasm/emscripten-version.txt
* bump emscripten versions in docker images in https://github.com/dotnet/dotnet-buildtools-prereqs-docker
* bump emscripten in https://github.com/dotnet/emsdk
* bump docker images in https://github.com/dotnet/icu, update emscripten files in eng/patches/
* update version number in docs
* update `Microsoft.NET.Runtime.Emscripten.<emscripten version>.Node.win-x64` package name, version and sha hash in https://github.com/dotnet/runtime/blob/main/eng/Version.Details.xml and in https://github.com/dotnet/runtime/blob/main/eng/Versions.props. the sha is the commit hash in https://github.com/dotnet/emsdk and the package version can be found at https://dev.azure.com/dnceng/public/_packaging?_a=feed&feed=dotnet6
* update packages in the workload manifest https://github.com/dotnet/runtime/blob/main/src/mono/nuget/Microsoft.NET.Workload.Mono.Toolchain.Manifest/WorkloadManifest.json.in
Expand Down
2 changes: 1 addition & 1 deletion src/mono/wasm/build/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -137,7 +137,7 @@ them for the new task assembly.

If encountering build performance issues, you can use the rollup `--perf` option and the typescript compiler `--generateCpuProfile` option to get build profile data, like so:

```../emsdk/node/14.15.5_64bit/bin/npm run rollup --perf -- --perf --environment Configuration:Release,NativeBinDir:./rollup-test-data,ProductVersion:12.3.4```
```../emsdk/node/14.18.2_64bit/bin/npm run rollup --perf -- --perf --environment Configuration:Release,NativeBinDir:./rollup-test-data,ProductVersion:12.3.4```

```node node_modules/typescript/lib/tsc.js --generateCpuProfile dotnet-tsc.cpuprofile -p tsconfig.json ```

Expand Down
1 change: 1 addition & 0 deletions src/mono/wasm/build/WasmApp.Native.targets
Original file line number Diff line number Diff line change
Expand Up @@ -256,6 +256,7 @@
<_WasmNativeFileForLinking Include="@(NativeFileReference)" />

<EmscriptenEnvVars Include="EMSDK_PYTHON=$(EmscriptenPythonToolsPath)python.exe" Condition="'$(OS)' == 'Windows_NT'" />
<EmscriptenEnvVars Include="PYTHONPATH=$(EmscriptenPythonToolsPath)" Condition="'$(OS)' == 'Windows_NT'" />
<EmscriptenEnvVars Include="EM_CACHE=$(WasmCachePath)" Condition="'$(WasmCachePath)' != ''" />
</ItemGroup>

Expand Down
2 changes: 1 addition & 1 deletion src/mono/wasm/emscripten-version.txt
Original file line number Diff line number Diff line change
@@ -1 +1 @@
2.0.23
3.1.1
2 changes: 1 addition & 1 deletion src/mono/wasm/wasm.proj
Original file line number Diff line number Diff line change
Expand Up @@ -120,7 +120,7 @@
<_EmccLinkFlags Include="-s FORCE_FILESYSTEM=1" />
<_EmccLinkFlags Include="-s EXPORTED_RUNTIME_METHODS=&quot;['FS','print','ccall','cwrap','setValue','getValue','UTF8ToString','UTF8ArrayToString','FS_createPath','FS_createDataFile','removeRunDependency','addRunDependency', 'FS_readFile']&quot;" />
<!-- _htons,_ntohs,__get_daylight,__get_timezone,__get_tzname are exported temporarily, until the issue is fixed in emscripten, https://github.com/dotnet/runtime/issues/64724 -->
<_EmccLinkFlags Include="-s EXPORTED_FUNCTIONS=_free,_malloc,_memalign,_memset" Condition="$([MSBuild]::VersionGreaterThan('$(_EmccVersion)', '3.0'))" />
<_EmccLinkFlags Include="-s EXPORTED_FUNCTIONS=_free,_malloc,_memalign,_memset,_htons,_ntohs,__get_daylight,__get_timezone,__get_tzname" Condition="$([MSBuild]::VersionGreaterThan('$(_EmccVersion)', '3.0'))" />
<_EmccLinkFlags Include="-s EXPORTED_FUNCTIONS=_free,_malloc,_htons,_ntohs,__get_daylight,__get_timezone,__get_tzname,_memalign" Condition="$([MSBuild]::VersionLessThan('$(_EmccVersion)', '3.0'))" />
<_EmccLinkFlags Include="--source-map-base http://example.com" />
<_EmccLinkFlags Include="-s STRICT_JS=1" />
Expand Down

0 comments on commit de0676c

Please sign in to comment.