Enable startup events over EventPipe #36720

josalem · 2020-05-19T19:37:00Z

These changes add the ability for EventPipe to report startup events. This is accomplished by stopping the runtime during EEStartupHelper after the Diagnostics Server and EventPipe have been turned on. This allows a monitoring agent to connect to either the reverse or traditional Diagnostics IPC transport to create and EventPipe session.

Runtime startup is resumed using the new Diagnostics IPC Command on the EventPipe CommandSet, ResumeStartup.

The pause during startup is on by default if the DOTNET_DiagnosticsMonitorAddress is set in the environment. It can be opted out of by setting DOTNET_DiagnosticsMonitorStopOnStart=0. The pausing code is specifically placed inside of a function marked as __attribute__((noinline)) so that it will always show up on top of the stack during the pause.

CC - @tommcdon @karpinsn

ghost · 2020-05-19T19:37:02Z

Tagging subscribers to this area: @tommcdon
Notify danmosemsft if you want to be subscribed.

jkotas · 2020-05-19T21:08:42Z

__attribute__((noinline)) so that it will always show up on top of the stack during the pause.

This does not guarantee that the method is going to be on the stack. It does not prevent the compiler from performing tail-call optimizations (that it will likely) on the method that will make the method disappear from the stack as well.

src/coreclr/src/vm/eventpipe.cpp

src/coreclr/tests/src/tracing/eventpipe/common/Reverse.cs

src/coreclr/tests/src/tracing/eventpipe/stoponstart/stoponstart.cs

src/coreclr/src/vm/eventpipe.cpp

sywhang · 2020-05-20T06:49:01Z

src/coreclr/tests/src/tracing/eventpipe/stoponstart/stoponstart.csproj

+    <AllowUnsafeBlocks>true</AllowUnsafeBlocks>
+    <CLRTestPriority>0</CLRTestPriority>
+    <UnloadabilityIncompatible>true</UnloadabilityIncompatible>
+    <JitOptimizationSensitive>true</JitOptimizationSensitive>


Does this test work with GCStress? (I've seen a lot of eventpipe tests fail with GCStress on mainly because of timing issues)

Haven't tried. I'll run the new tests with gcstress locally.

src/coreclr/src/vm/eventpipe.cpp

noahfalk

Its on the right track but I suspect we've got a number of issues to resolve and some may be non-trivial.

src/coreclr/src/inc/clrconfigvalues.h

src/coreclr/src/vm/ceemain.cpp

src/coreclr/src/vm/eventpipe.cpp

noahfalk · 2020-05-20T11:29:15Z

src/coreclr/tests/src/tracing/eventpipe/stoponstart/stoponstart.cs

+            return true;
+        }
+
+        public static async Task<bool> TEST_TracesHaveRelevantEvents()


You may also want to test a variation: if you don't resume the runtime then we should not have events.

src/coreclr/tests/src/tracing/eventpipe/common/IpcUtils.cs

* only init event if we use it * read only necessary config values * don't new in static constructor

* move all command ids to diagnosticsipc.h * add check for command enabled status based on fEEStartup

josalem · 2020-06-01T23:21:24Z

Perf pipeline failure seems to be an infra issue about the razor build server:

[2020/05/30 02:23:09][INFO] // ***** BenchmarkRunner: End *****
[2020/05/30 02:23:09][INFO] // ** Remained 0 benchmark(s) to run **
[2020/05/30 02:23:09][INFO] Run time: 00:00:00 (0.34 sec), executed benchmarks: 1
[2020/05/30 02:23:09][INFO] 
[2020/05/30 02:23:09][INFO] Global total time: 00:04:51 (291.64 sec), executed benchmarks: 580
[2020/05/30 02:23:09][INFO] // * Artifacts cleanup *
[2020/05/30 02:23:09][INFO] $ popd
[2020/05/30 02:23:09][INFO] $ pushd "/home/helixbot/work/B3F40969/p/performance"
[2020/05/30 02:23:09][INFO] $ dotnet build-server shutdown
[2020/05/30 02:23:10][INFO] Shutting down MSBuild server...
[2020/05/30 02:23:10][INFO] Shutting down VB/C# compiler server...
[2020/05/30 02:23:10][INFO] Shutting down Razor build server (process 9354)...
[2020/05/30 02:23:10][INFO] VB/C# compiler server shut down successfully.
[2020/05/30 02:23:10][INFO] Razor build server (process 9354) failed to shut down: The shutdown command failed: The application to execute does not exist: '/home/helixbot/work/A2B10920/w/AE05090D/e/.dotnet4587/sdk/5.0.100-preview.6.20266.3/Sdks/Microsoft.NET.Sdk.Razor/tools/netcoreapp3.0/rzc.dll'
[2020/05/30 02:23:10][INFO] 
[2020/05/30 02:23:10][INFO] MSBuild server shut down successfully.
[2020/05/30 02:23:10][INFO] $ popd
[2020/05/30 02:23:10][ERROR] Process exited with status 1
Traceback (most recent call last):
  File "/home/helixbot/work/B3F40969/p/performance/scripts/benchmarks_ci.py", line 264, in <module>
    __main(sys.argv[1:])
  File "/home/helixbot/work/B3F40969/p/performance/scripts/benchmarks_ci.py", line 250, in __main
    dotnet.shutdown_server(verbose)
  File "/home/helixbot/work/B3F40969/p/performance/scripts/dotnet.py", line 656, in shutdown_server
    get_repo_root_path())
  File "/home/helixbot/work/B3F40969/p/performance/scripts/performance/common.py", line 200, in run
    returncode, quoted_cmdline)
subprocess.CalledProcessError: Command '$ dotnet build-server shutdown' returned non-zero exit status 1.

I'm not sure this is related to my changes, but I don't see any recurrences of this issue in the history.

The other failure seems to be happening on other PRs as well; issues created at #37277

Both failures passed on a re-run of those legs.

noahfalk

Sorry for delay on this John, looking much better : ) A few more things I'd recommend dealing with commented inline.

src/coreclr/src/vm/ceemain.cpp

src/coreclr/src/vm/diagnosticserver.cpp

src/coreclr/src/vm/diagnosticsprotocol.h

src/coreclr/src/vm/eventpipe.cpp

src/coreclr/src/vm/diagnosticserver.cpp

src/coreclr/src/vm/diagnosticserver.h

…up-events-over-eventpipe

Mono port of CoreCLR PR: dotnet#36720.

Port changes done in the following CoreCLR PR's affecting corresponding Mono runtime EventPipe library sources: dotnet/runtime#36720 dotnet/runtime#37002 dotnet/runtime#38967 dotnet/runtime#40191 dotnet/runtime#40332 dotnet/runtime#40499 EventPipe native code changes track by dotnet/runtime#36820.

* [MonoVM] Enable startup events over EventPipe. Mono port of CoreCLR PR: #36720. * [MonoVM] Add the ability for profilers to listen to EventPipe events. Mono port of CoreCLR PR: #37002. * [MonoVM] Add ProcessInfo command to Diagnostics Server. Mono port of CoreCLR PR: #38967. * [MonoVM] Unregister callback when setting a provider for deferred delete. Mono port of CoreCLR PR: #40191. * [MonoVM] Generational aware analysis based on environment variable. Mono port of CoreCLR PR: #40332. NOTE, the ability to pause a session was not ported since it currently have no use in Mono and looks like a temporary solution in CoreCLR. * [MonoVM] Rebrand Diagnostics Server to DiagnosticPort. Mono port of CoreCLR PR: #40499.

Port changes done in the following CoreCLR PR's affecting corresponding Mono runtime EventPipe library sources: dotnet/runtime#36720 dotnet/runtime#37002 dotnet/runtime#38967 dotnet/runtime#40191 dotnet/runtime#40332 dotnet/runtime#40499 EventPipe native code changes track by dotnet/runtime#36820. Co-authored-by: lateralusX <lateralusX@users.noreply.github.com>

* Applied BinaryPrimitives throughout CoreLib (#37582) * Flaky multiple HTTP/2 connection tests read full request body (#40844) There are 2 flaky HTTP/2 tests verifying multiple connections feature which are randomly failing on CI, but not locally. - Http2_MultipleConnectionsEnabled_ConnectionLimitNotReached_ConcurrentRequestsSuccessfullyHandled - Http2_MultipleConnectionsEnabled_IdleConnectionTimeoutExpired_ConnectionRemovedAndNewCreated It seems the failure is caused by not reading the request body. In current implementation, `Http2LoopbackServer` read only HEADERS frame via `ReadRequestHeaderAsync` and then immediately sends response. However, the client firstly completely sends headers and body and only then starting reading a response. Thus, it seems to get blocked sometimes if the server didn't read the full body. This PR fixes this by calling `ReadAndParseRequestHeaderAsync` instead of `ReadRequestHeaderAsync`. Fixes #40436 Fixes #40115 * Fix timing issues with Http2_PingKeepAlive test. (#40788) * Ensure FileStatus and FileSystemEntry Hidden and ReadOnly attributes are retrieved the same way (#40641) * Ensure FileStatus and FileSystemEntry IsHidden attribute is retrieved the same way * Add missing check in public property FileSystemEntry.IsHidden. Address PR suggestions. * Add tests for Hidden and ReadOnly attribute check for each platform. Split existing Skip attribute test into different platforms. * Use _initialAttributes instead of Attributes for IsHidden. Add comment on top. Co-authored-by: carlossanlop <carlossanlop@users.noreply.github.com> * Move content type verification to a later stage (#40594) * Move content type verification to a later stage Fix #38713 * Rename ValidateContent to ValidateContentType Fix #38713 * Get rid of content type check * Add newline at end of the file * Rename TestValidMediaTypes to TestVariousMediaTypes * Set MaxConcurrentStreams = 100 on new Http2Connection until first SETTINGS frame (#40779) * Set MaxConcurrentStreams = 100 on new Http2Connection until first SETTINGS frame * Update src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http2Connection.cs Co-authored-by: Chris Ross <Tratcher@Outlook.com> * - Setting infinite MaxConcurrentStreams fixed - Test verifying new connection request queueing added * More tests * - const case fixed - Outerloop attributed added * ExtraStreams case fixed Co-authored-by: Chris Ross <Tratcher@Outlook.com> * Update analyzers and various fixes to be able to build with latest RC1 SDK (#40796) * Disable net analyzers from the SDK and use them from package reference instead * Fix build with new SDK and bump net analyzer version * PR Feedback * Fix ilasm, ildasm, crossgen and others launch from PATH (#40791) * Fix ilasm, ildasm and crossgen launch from PATH When trying to launch ilasm, ildasm, crossgen or any tool using PAL_Initialize that are on the current PATH just by their file names, they were crashing with SIGSEGV. The problem was that the current exe path was extracted incorrectly in that case. This change fixes the problem using the same methods that corerun uses to figure out the current exe path. * Add missing FreeBSD headers * Reflect PR feedback * Update src/coreclr/src/pal/src/init/pal.cpp * Fix typo in Solaris version of the code Co-authored-by: Jan Kotas <jkotas@microsoft.com> * Fix a deadlock in EventSource and CounterGroup (#40259) * Fix a deadlock in CounterGroup and EventSource * add more comments * Add some more comments * PR feedback * Add comments * Fixing the allconfig build. (#40812) * fixing all config build * address feedback * Adding support for constrained open generics to DI (#39540) * Porting changes from https://github.com/dotnet/extensions/pull/536 * Correcting reference * Avoiding wrapping exception as there may be other reasons why closing the generic type fails other than constraint failure * Throw when non-nullable struct converters indicate that they handle nullable structs (#40824) * Wrap non-nullable struct converters in NullableOfT converter when they explicitly handle nullable structs * Throw exception instead of being nice * Improve exception message * Change null handling in custom converters for backwards compat (#40859) * Fix TailCallStress mode. (#40698) Improve validation of tail calls that are not tail-prefixed in the IL but are marked as such because of TailCallStress. We now do the same correctness validation in morph for such tail calls as we do for implicit tail calls. That blocks tail calls when we have address-taken locals, struct promoted params, and pinned vars. Fixes #39398. Fixes #39309. Fixes #38892. Fixes #38889. Fixes #38887. Fixes #37117. Fixes #8017. * Ignore the executable file be deleted in Process Start (#40748) * Fix stack overflow createdump triggering on Windows (#40869) * Move Font*, Image* and Margins converters to System.Drawing.Common and apply Unit and Name converter (#40825) * Move Font, Image, and Margins converters to System.Drawing.Common * Add FontUnit converter and FontName converter to Font properties * Fix Unix tests * Sort Compile items * [mono] Make bundle filename changes netcore-only (#40845) * Generational aware analysis based on environment variable. (#40332) * Nullable: System.Xml, part 6 (XSLT minus XSLT.Runtime) (#40368) * Nullable: System.Xml, part 6 (XSLT minus XSLT.Runtime) * change Evaluate to take non-nullable * apply pr feedback * fix nullability merge conflicts * apply feedback * Fix annotation in DataTableReaderListnener (#40855) When running the new nullable constructor analysis on runtime we discovered what appears to be a missing annotation in `DataTableReaderListener`. Note: the second `if` check in the constructor seems like dead code. It's checking for the field to be non-null before ever setting the field to a value. Probably could be deleted. * Reverts https://github.com/dotnet/corefx/pull/37583 ("Use clonefile for CopyFile, if available #37583") (#40753) * Revert " Use clonefile for CopyFile, if available (dotnet/corefx#37583)" This reverts commit 02ba75ad0fdda2e5f5c93a8e4e3ab6600d87fb99. * Add unit test * Disable tests failing in browser-wasm * Fix sync ConnectHelper.Connect (#40880) * Change Assembly.GetFile(s) to throw when in-memory (#40593) * Change Assembly.GetFile(s) to throw when in-memory This is technically a breaking change, although a minor one, as the previous behavior also threw an exception. The previous exception was an IOException and this is an InvalidOperationException, so if users were catching the previous exception they would not succeed now. Fixes #40154 * Don't specify exception for GetFile when passed null * Skip ImplementedInterfaces test on Mono * Fix tests for running in memory Co-authored-by: Dan Moseley <danmose@microsoft.com> Co-authored-by: Jan Kotas <jkotas@microsoft.com> * [wasm] Add Timezone data generator as build task (#39913) Add two build tasks: 1. DownloadTimeZoneData 2. CreateWasmBundle (given directory bundle files into a data archive) * Update PGO/IBC Data. (#40850) * Delete unused files (#40889) * Limit Debugger.Log() calls in EventSource (#40823) * Limit Debugger.Log() calls in EventSource Fixing #40799 Previously we would send a message via Debugger.Log() for event call to ReportOutOfBandMessage(). Now we only do it the first 16 times, then all errors after that are silent. This should alleviate performance overhead from a scenarios with a large number of errors. I made this change quickly via github to help out Bing. I have never compiled or tested it prior to the PR but fingers crossed it appears simple enough that it has a decent chance to work : ) * Remove trailing whitespace Co-authored-by: David Mason <davmason@microsoft.com> * Workaround for WSL1 process IO redirection bug (#40851) * Workaround for WSL1 process IO redirection bug * Update src/libraries/System.Diagnostics.Process/src/System/Diagnostics/Process.Unix.cs Co-authored-by: Adeel Mujahid <adeelbm@outlook.com> Co-authored-by: Adeel Mujahid <adeelbm@outlook.com> * Make suspend the default port configuration (#40888) * Make suspend the default port configuration * ignore empty port addresses * Appease MSVC * Remove the coreClrRepoRoot YAML variable (#40893) This variable was useful for a transitional period during the consolidation of dotnet runtime repos; it has no value anymore after the consolidation and its presence makes the YAML scripts more complex and harder to reason about. I propose removing it and replacing its uses with the open-coded relative path to the coreclr repo which also makes parts of the script logic more obvious. This was motivated by some of Nathan's struggles when standing up WebAssembly tests in the runtime repo. Thanks Tomas * implement ConnectAsync overloads with CancellationToken on Socket and TcpClient (#40750) * implement ConnectAsync overloads with CancellationToken on Socket and TcpClient * Fix runtime crash on Linux ARM64 running in WSL (#40597) getauxval(AT_HWCAP) returns zero on Linux ARM64 running in WSL1. Fall back to reading capabilities from /proc/cpuinfo in that case. * Disable HttpTelemetry for HTTP 3 requests (#40839) * Disable HttpTelemetry for HTTP 3 requests * Link to tracking issue for HTTP/3 Telemetry * Add remaining System.Net.Http Telemetry (#40838) * Add remaining System.Net.Http Telemetry * Remove http20-streams-current-total * Use consts instead of magic values * Move ResponseHeadersBegin event closer to actual timestamp for HTTP2 * Ensure value-type handling converter created by factory can be used to (#40879) handle corresponding nullable value types * Implement concept of unstable ABI into crossgen2 (#40820) * Implement concept of unstable ABI into crossgen2 - Use for the Vector types based on current defined stability rules - Will prevent compilation of methods with not yet stable abis, or calls to methods with said unstable apis * Bugs * Renamed EventSources "Microsoft-"->"Private.InternalDiagnostics.". * Add NetworkError.TimedOut, rename Unknown to Other (#40841) Adds NetworkError.TimedOut, Renames Unknown to Other assuming my proposal gets accepted, will revert if not, Improves exception message * Implement ReadOnlySequence<T>.GetPosition(SequencePosition) (#771), fixes #27158 * Include <sys/sysctl.h> only for FreeBSD in pal.cpp (#40923) * Make Enumerable.Cast<T> and friends return IEnumerable<T?> (#40929) * Make IEnumerable.Cast<TResult>() annotation admit nullable elements * Make ImmutableArray cast method annotations report nullable elements * fix System.Data nullability warnings * Update linker warning numbers (#40865) * Update linker warning numbers In response to https://github.com/mono/linker/pull/1385. This replaces the global suppressions for IL2006 with the warnings that it was split into. * Add suppressions for linker tests * Fix IndexOutOfRangeException in CallSite.MoveRule. (#39159) Because synchronization of AddRule is omitted for performance, concurrent invocations of AddRule may cause Rules to revert back to an older (smaller) version, making i out of bounds. Add a guard against that. The reasoning behind no synchronization is that Rules is just a cache after all. As long as the cache is in a consistent state, wrong updates will only cause more cache misses or longer lookup time. This performance loss is fine if we get more performance gain by omitting synchronization. Fix #36983 * Block out the Runtime_40444 test failing in PR's + alpha-ordering (#40907) * Make the buffer size configurable for generational aware analysis (#40918) * Fix GC liveness check in ByRefLocals test (#40857) Fix #40751 * Reenable crossgen2smoke under GC stress (#40849) This change removes the open-coded Crossgen2 execution from the crossgen2smoke test project and switches it over to use the normal CLRTest.Crossgen.targets logic for Crossgen2 invocation. As the CLRTest.Crossgen.targets script already contains my fix for the underlying issue - we were inadvertently running the Crossgen2 compiler itself in GC stress mode and timing out due to the long compilation times - we can now reenable this test in GC stress mode. Thanks Tomas * Intrinsicify SpanHelpers.IndexOfAny(char,...) (#40729) * Intrinsicify IndexOfAny(char,char) * Intrinsicify IndexOfAny(char,char,char) * Intrinsicify IndexOfAny(char,char,char,char) * Intrinsicify IndexOfAny(char,char,char,char,char) * Avoid movsx * Feedback * Super nit * Normalize directory separator of paths in FileSpec (#40699) * Create dump in utility thread after m_LastThrownObjectHandle has been set (#40915) The previous Windows stack overflow createdump trigger fix generated the dump before the m_LastThrownObjectHandle in the thread was set. This breaks SOS's !pe command. * Not all object converters return a JsonElement (#40921) Co-authored-by: devsko <stefan.kohlmeister@outlook.de> * Test concurrent calls to CallSite.AddRule() and MoveRule() (#40858) * Do not build Targeting pack in source-build (#40870) * Do not build Targeting pack in source-build * Update src/installer/pkg/projects/Directory.Build.targets Co-authored-by: Davis Goodin <dagood@users.noreply.github.com> Co-authored-by: Davis Goodin <dagood@users.noreply.github.com> * Pass length to PathCharString.Append on SunOS (#40928) * [aot] Avoid emitting generic instances with vtype arguments on wasm to save space. (#40854) Co-authored-by: vargaz <vargaz@users.noreply.github.com> * use port in SPN calculation if not default (#40860) * use port in SPN calculation if not default * remove extra line * fix internal error test failure * Fix patch skip + emulation on ARM64 (#40941) * Fix patch skip + emulation on ARM64 * Remove ObsoletedInOSPlatformAttribute (#40945) * Throw when reference cannot be assigned to converter.TypeToConvert (#39015) * Throw when reference cannot be assigned to converter.TypeToConvert * Throw InvalidOperationException instead of JsonException since the cause for this error would be an error in the user code rather than in the JSON payload * Use try-catch instead of IsAssignableFrom * Alternatively validate reference can be assigned as T at the time is fetch * Add suggested comment * Do not include extensions sequence if there are no extensions. When encoding an X509 certificate, omit the extensions sequence if the effective number of extensions is zero. Per RFC 5280, this sequence is "one or more certificate extensions". * Extend timeouts in revocation timeout tests. Extends the amount of time that the delay has to give the build servers more wiggle room to fetch and process revocation. Improve assertion output. * Don't load assemblies inside bundle when using LoadFrom (#40826) * Don't use bundler's probe in LoadFromPath * Add support for the InnerException in Json formatter using ToString() (#40852) * Support InnerException(s) on JsonConsoleFormatter Fix #40507 Co-authored-by: Werner Mairl <werner.mairl@gmail.com> * [mono] Add `X86Base` support. (#40861) Implement `BitScanForward` and `BitScanReverse` as intrinsics. Implement `__cpuidex` as a netcore-only icall. Rename `cpuid` (in mono-hwcap-x86.c) to `mono_cpuidex`, give it extern linkage, and add support for extended CPUID. Related: https://github.com/dotnet/runtime/pull/40167 Co-authored-by: imhameed <imhameed@users.noreply.github.com> * Make RegexParseException and RegexParseError public, rename enum fields (#40902) * RegexParseException, RegexParseError public, rename enum fields * Add Unknown field to RegexParseError * Add definitions to ref file * Satisfy NET48 tests * Some nits to resolve review comments * Add default message and Unknown message * Added doc comments * Fix nit in auto-layout change of csproj file * Simplify message for RegexParseException ctor overload * Remove PlatformDetection.IsNetFramework in RegexParserTests.cs * Rename resource names to match the enum names * Use partial class and methods to separate netfx and core * Add offset tests * Apply suggestions from code review Nits * Add remaining test offsets. Verify all offsets. * Nit and TFM * Update src/libraries/System.Text.RegularExpressions/tests/RegexParserTests.netcoreapp.cs Co-authored-by: Stephen Toub <stoub@microsoft.com> * Update src/libraries/System.Text.RegularExpressions/tests/RegexParserTests.netfx.cs Co-authored-by: Stephen Toub <stoub@microsoft.com> * Update src/libraries/System.Text.RegularExpressions/tests/RegexParserTests.netfx.cs Co-authored-by: Stephen Toub <stoub@microsoft.com> * Code review: properly test for deserialized type of exn * Code review: fix doc comments * Code review: fix doc comments * Code review: offset gt zero * Code review: add some tests, remove redundant whiteline * Code review: add doc comment as suggested * Code review: fix doc comments * ref edits with tool * Code review: some nits * Remove using * Rename unknown * Incorrect edit in AssertExtensions * Cherry-pick #d4090296 * Fix naming, ref file, resource strings Co-authored-by: Dan Moseley <danmose@microsoft.com> Co-authored-by: Prashanth Govindarajan <prgovi@microsoft.com> Co-authored-by: Stephen Toub <stoub@microsoft.com> * retire TLS1.0/1.1 ciphers from default list (#40746) * retire TLS1.0/1.1 ciphers from default list * fix build with 1.0 * feedback from review * add compat bio_st * don't enforce cipher default on openssl 1.0 * fix ServerAsyncAuthenticate_MismatchProtocols_Fails * call ERR_clear_error after protocol det:w!ection * feedback from review * add missing Using * [mono] Loader improvements: bundle optimization and logging (#40819) * Delay bundle loading until after checking the ALC cache This avoids having to create and destroy MonoImage objects over and over when looking for already-loaded assemblies, which should cut down on overall load time a bit. * Improve loader algorithm logging * Deduplicate Helix environment setup in send-to-helix-step.yml (#40909) * Ordinal Ignore Case Optimization (#40910) * Use Roslyn apis to generate Platform Not Supported Assemblies (#40296) * fixes on the runtime side * address feedback * update arcade version * merge conflicts * update the version * [wasm][debugger] Support deep member accesses for EvaluteOnCallFrame (#40836) * [wasm][debugger] Support deep member accesses for EvaluteOnCallFrame Eg. `obj.Property.X`, or `obj.Y + obj.Z.p` Each of the member access expressions (like `a.b.c`) must be of only primitive types. Though if the expression is a single member access (and nothing else, like `"a.b.c"`), then that can be a non-primitive type. This works by sending the expression to the browser, where it gets resolved by `library_mono.js`. And that takes an easy route for doing this, essentially just fetching the list of locals/properties, and using that. There are better ways to do this, that will be explored later. * [wasm][debugger][tests] Remove some debug spew * [wasm][debugger] fix formatting with dotnet-format * Fix EventActivityIdControl test on MonoVM. (#39178) Make EventSource call static version of EventActivityIdControl in EventPipeEventProvider instead of direct calling low level EventPipeInternal (now only called from EventPipe* sources). Doing it that way will also keep EventPipeEventProvider.EventActivityIdControl in S.P.C meaning that tests like eventactivityidcontrol won't risk of using methods that has been trimmed away by linker. Added missing case into Mono's EventPipeInternal::EventActivityIdControl implementations. Enabled eventactivityidcontrol test on Mono runtime. Adjusted test to use static EventPipeEventProvider.EventActivityIdControl version. * Fix Runtime_40444 test exclusion in issues.targets (#40973) After I noticed that the test Runtime_40444 is still failing in PR runs after I disabled it in issues.targets, I found out that I had missed the extra folder level in the exclusion spec. Thanks Tomas * Fix duplicate --runtime-path in generated helix command (#40975) While looking at some helix debug logs I noticed we pass the `--runtime-path` option twice: ``` Executed on dci-mac-build-083.local + ./RunTests.sh --runtime-path /tmp/helix/working/A68F0925/p --runtime-path /tmp/helix/working/A68F0925/p ``` Looks like this was an oversight from https://github.com/dotnet/runtime/pull/35606, it's harmless but I thought I'd clean it up :) * Increase timeout for yaml as well (#40987) * Fix index adjustments for Get/Set/Address called on 1 dim arrays. (#35008) Enable tests failing due to missing array ctor and lbound index checks. Fix interpreter Get/Set/Address using lbound. Add missing OP_FCONV_TO_I on Arm64, causing test errors. Fix interpreter arrray index and lower bound when using negative values. Add branch checking array->bounds for NULL. * Disable System.Xml.Tests.SameInstanceXslTransformWriter.Variations under JitStress (#40964) * [aot] Fix the lookup of jit info for WASM. On wasm, methods have no size, so only the method address can be looked up, jinfo->code_size is set to 1. (#40969) Fixes https://github.com/mono/mono/issues/20256. Co-authored-by: vargaz <vargaz@users.noreply.github.com> * Add parsing of zero port for unknown Uri scheme (#40986) Fixes #37865 * Fix packaging for the browser (#40814) * Fix packaging for the browser * fix wasm leg * remove unused resource * Fix bad quoting (#41002) * Add GitHub Action for backporting PRs to branches (#40955) This adds an action that can be triggered with a special comment on a PR: `/backport to <some branch>`. The action then creates a new PR to the target branch with the commits from the source PR applied. * Disable default non-portable build for armel (#41031) * Configuration Json provider: adds empty objects to configuration (#40829) * Add Designer attributes to types that had it in full framework (part 2) (#40966) * Move DesignerSerializerAttribute to S.CM.Primitives and add to StringDictionary * Add TolboxItemAttribute to types that had it in Desktop * Remove TypeDescriptionProviderAttribute from DefaultGenApiDocIds * PR Feedback, fix build * Remove semicolon from WASM config (#41017) * Fix Memory Leak in System.DirectoryServices.Protocols caused by not freeing native memory (#41053) * Disable System.IO.Tests.FileStream_SafeFileHandle.ThrowWhenHandlePositionIsChanged_async test on Browser wasm (#40920) * update product version to 6.0.0 (#40950) * update to net6 * adding comment * Fix trailing whitespaces (#40891) * Trim trailing whitespaces * Match raw with rendered * Delete extra asterisks and | * Update ELT Hooks - tail calls.md Co-authored-by: Jan Kotas <jkotas@microsoft.com> * Delete redundant or no longer necessary build properties (#41051) * Cleaning up unused TargetsBrowser conditions (#41062) * minor cleanup * remove unused string * [Pipelines] Noop Writes when Reader completed (#40905) * remove dead code (#41064) * Enable nullability on System.Xml.XPath (#41060) * Enable nullability on System.Xml.XPath * Switch XPathException ctor's message parameter to string? * Fix Mono LLVMJIT official build to produce the right package (#41049) * Fix NativeBinDir for individual projects when BuildTargetFramework is not set (#41074) * CoreRT support for ARM64&Unix (#41023) * Update codegenarm64.cpp * ARM64 CFI unwind Data * Add nullability annotations to System.Private.Xml.Linq project (#40744) * Add nullability annotations to Xml.Linq project * Fix misplaced assertion * Address feedback * Add missing NotNullIfNotNull attributes to operators * Mark some System.Net.* APIs as unsupported on Browser WASM (#40924) * Change nullability of public IXmlNamespaceResolver parameters where appropriate (#41079) * [mono] Add MethodImplOptions.NoOptimization support (#40990) * [wasm] Skip Assert.InRange in Copy.cs for Browser (#41000) * Revert "[Wasm] Enable CopyWithData FileSystem test (#40663)" This reverts commit 1821567ab60e156101f2a5bdcd5dab6d96dc3c71. * Provide context for skipping Assert in Copy.cs Co-authored-by: Mitchell Hwang <mitchell.hwang@microsoft.com> * Fix assembly name due to rename of assembly. (#41037) * Remove Makefile from gitignore (#41075) * Address warnings for possibly null array elements (#41046) * Detect read-only Oid at runtime. The compile-time approach doesn't work because Pkcs is shipped as a standalone package, so if the new asset runs on .NET Core 3.1 or 2.1 it would assume immutable Oids when they were still mutable. * Fix OS version check and testhost copying in stress tests (#40980) SslStress also fails due to OperatingSystem.IsWindows() call which is not available on the public SDK yet. * [wasm] Skip EnsureThrowWhenCopyToNonSharedFile on Browser (#41048) * [wasm] Skip EnsureThrowWhenCopyToNonSharedFile on Browser * Provide context for skipping Copy.cs test on Browser Co-authored-by: Mitchell Hwang <mitchell.hwang@microsoft.com> * [Browser] Update ICU (#40981) * Fix propagation of target parameters in helixpublishwitharcade (#40959) Nathan discovered this inconsistency in his work on standing up web assembly CI testing; the top three target parameters don't match the equivalent 'env' properties in send-to-helix-step.yml. To achieve consistency I have removed the target properties from the 'env' set and instead I added logic to pass these properties explicitly to msbuild via command line options. Thanks Tomas * Update dotnet_support.js (#41058) * Update Mono building docs with wasm-specific advice (#41122) * Fix an over-constrained use of a byte reg (#41004) * Fix an over-constrained use of a byte reg Fix #40963 * PR Feedback * Modify the test case Runtime_40444.cs (#40951) * Modify the test case Runtime_40444.cs 1) Increase the maximum loop count for failure to 1000 million 2) Added additional computation inside of loop 3) Add addition message when the loop executes to the limit and the test case fails Updated test case and insured that it fails on JITS without the fix Test now checks it the other thread set t2_finished * Re-enable the Runtime_40444 test * Fix typos * Add dependencies for BrowserDebugHost into Microsoft.NETCore.BrowserDebugHost.Transport nupkg (#41111) Before we were only packaging the host and proxy assemblies, but we need their dependencies as well. Moved the `CopyLocalLockFileAssemblies=true` to BrowserDebugHost.csproj to make sure we get all dependencies there. * Annotate DbDataReader.GetSchemaTable as nullable (#41082) Reverts decision made in #509 * Fix memory ordering issue in StressHeap StressHeap contains a hack where we chop off the end of a string allocated for that purpose and make a new free object out of the chopped of piece. We also reduce the size of the string, so the GC sees the free object when walking the heap - the idea is to force compaction. On processors with weak memory ordering semantics, we need a memory barrier to make sure the write to the new free object is seen before the write to the string length, otherwise the heap won't be walkable, leading to an AV in background_sweep. It looks like this will fix most of the ARM64 stress issues we've seen (notably the AV in gc_heap::background_sweep), we'll see what remains. * [wasm] Mark System.ComponentModel APIs as unsupported on Browser (#41094) * [wasm] System.ComponentModel enable platform attributes * [wasm] Mark ExtendedProtectionPolicyTypeConverter.ConvertTo as unsupported * [wasm] Mark System.ComponentModel.TypeDescriptor.CreateInstance as unsupported * [wasm] Mark System.ComponentModel.TypeDescriptionProvider.CreateInstance as unsupported * [wasm] Mark System.ComponentModel.LicenseManager.CreateWithContext as unsupported * [wasm] Mark System.ComponentModel.MaskedTextProvider.Clone as unsupported Co-authored-by: Mitchell Hwang <mitchell.hwang@microsoft.com> * Update to 6.0.0 for mono and reset timeouts (#41132) * don't run "NoMono" benchmarks for Mono runtime: (#41148) * Mono does not implement Utf8String ctor yet * we want to run Utf8String benchmarks for CoreCLR * Re-enable ImplementedInterfaces tests on Mono (#41136) * Allow Windows dbgshim to support Linux targets (#41116) * Add missing exception checkpoint (#41098) * Add missing exception checkpoint * Enable failed test * Improve Windows error handling in Diagnostics IPC (#41008) * Add a method GetRequiredSection on Configuration (#40976) * Add a method GetRequiredSection * Adapt to use GetRequiredSection as Extension method to avoid External Implmenter breaking change * clean csproj remove local path * clean not used using * remove empty line * remove unused designer of resources file * add test on sub section (GetRequiredSection) * Update src/libraries/Microsoft.Extensions.Configuration.Abstractions/src/ConfigurationExtensions.cs Co-authored-by: Eric Erhardt <eric.erhardt@microsoft.com> * add unit test case if confiuration root is null * Use command to auto generate ref Co-authored-by: cmichel <cmichel@interparking.com> Co-authored-by: Eric Erhardt <eric.erhardt@microsoft.com> * Handle windows shutdown event (#41101) * Handle windows logoff and shutdown events By doing a clean EEShutdown, including calling ProcessExit handlers. This allows ProcessExit to respond to 'docker stop', which sends CTRLSHUTDOWN_EVENT to windows containers. Fixes https://github.com/dotnet/runtime/issues/36089 * Don't handle logoff event * Return host's file path when using Environment.GetCommandLineArgs()[0] in single-file apps (#41019) * Return host's path when using GetCommandLineArgs from within a single-file app * Add blazor job in runtime (#40863) * add blazor job in runtime * nit fixes * turn on PR runs for testing;PR fix * update wasm configurations arg * unpack wasm to correct location * disable PR trigger * [coop] Add handles for some SRE scenarios (#40925) * [coop] Pin fields of structure Ideally ReflectionMethodBuilder should contain coop handles fields. That approach is less practical since it requires a lot of code change. * [coop] Pin object stored on stack and passed over calls * Improve condition for generating test execution scripts (#41151) I have received heads-up from two sources - @naricc's work on Mono webassembly testing and @gbalykov's recent PR #40979 that there's something weird going on in the execution script creation logic. I have investigated the exact purpose of the condition referring to _CopyNativeProjectBinaries. I believe the intent of the condition was to suppress the creation of execution scripts during managed test build once we switched over to building managed test components only once. The problem is that the common build of managed test components reused by runs on all OS'es and architectures uses the allTargets specification to build all test projects including those that only work on some platforms. This is one of the reasons why we cannot emit the execution scripts at managed test component build time - we would emit them even for tests that later on in the pipeline turn out to be disabled for a particular platform or build configuration. The other reason is practical - today the CLRTest.*.targets scripts only generate one version of the script (cmd vs. sh) - but that would be easy to fix. Based on these findings I propose modifying the condition to check the 'allTargets' flag instead. In local builds (where we typically build managed test components for just a single target platform using the src/coreclr/build-test script) the option allTargets is not set and so the execution script can be easily created as part of the managed build. In the runtime pipeline case, the execution scripts end up getting emitted during the copynativeonly step before sending the tests to Helix as by this time the targeting platform and configuration for the run is completely resolved. Thanks Tomas * Fix pythonpath and internal parameter (#41190) * update helixprecommands order * update internal param * Fix category exclusion (#41192) * Update Mono runtime NativeLibrary test exclusions (#41181) * Cast cache Add should handle odd entry versions as inconsistent/taken. (#41109) * Cast cache Add should handle odd entry versions as inconsistent/taken * PR feedback * Switch over managed test component build to run on Linux x64 (#41169) * Rollback noopt change (llvm-backend) (#41186) * [runtime] Ongoing work on mono cmake build (#41055) * [runtime] Ongoing work on cmake build. * Install libraries * Add more configure options * Add wasm support * Fix the build. * Remove unused defines, reformat. * Install headers. * Remove memset in EventPipeBuffer allocation (#41194) * [runtime] Make the code which handles pinvoke lookup retries netcore only. (#41201) Co-authored-by: vargaz <vargaz@users.noreply.github.com> * Ensure non-shipping symbol packages get indexed (#41142) * Add g_ptr_array_find to eglib (#41157) * [mono] Disable XML file loading entirely on netcore (#41158) * [Browser] Bump ICU (#41166) * Remove issues.targets exclusion for fixed issue 34316 (#41208) * [mono] Don't leak coop handles, fix type in managed ALC struct (#41159) * Don't leak coop handle * Fix type in MonoManagedAssemblyLoadContext * [wasm][debugger] Fix expression evaluation when it is a reference (#41135) * [wasm][debugger] Fix expression evaluation when it is a reference .. preceded by spaces. Eg: `" foo.dateTime", or `" foo.count"` * Explanation of the fix: - these particular expressions end up in the code path where we get a SimpleMemberAccessExpression, and only one member access (like `a.b.c`) was found. - that code path had `return memberAccessValues[0]?["value"]?.Value<JObject>();` - which is incorrect, and we need to return `memberAccessValues[0]`, which is the value itself. * Add DO-NOT-SIGN comment for files that should not be signed (#41185) * Add regression test for GitHub issue 13394 (#41210) * Avoid having two copies of System.Private.CoreLib.ni.pdb in the same NuGet package directory (#41057) * Move Editor Attribute to S.CM.Primitives and apply to types that had it in netfx (#41145) * Move EditorAttribute to System.ComponentModel.Primitives * Add EditorAttribute to types that had it in netfx and fix some DesignerSerializableAttributes * PR Feedback * PR Feedback * Revert "Improve Windows error handling in Diagnostics IPC (#41008)" (#41220) This reverts commit 5cabbabe3a3bdeaaf8f0b9dc34cf401ee88e86ff. * Fix linux_musl runs (#41224) This bug was caused by slight semantic change I inadvertently introduced in my PR https://github.com/dotnet/runtime/pull/40959 According to today logic in the relevant scripts, TargetOS shouldn't include the OS subgroup spec. I have modified send-to-helix-step to pass it in a separate property parameter (TargetOSSubgroup) so that it can be used for construction of the HelixRuntimeRid. Thanks Tomas * Use nint instead of IntPtr in generic SpanHelpers (#41199) * [master] Update dependencies from dotnet/runtime-assets dotnet/llvm-project dotnet/icu mono/linker dotnet/arcade Microsoft/vstest dotnet/xharness (#41025) * Update dependencies from https://github.com/dotnet/arcade build 20200817.6 Microsoft.DotNet.XUnitExtensions , Microsoft.DotNet.GenFacades , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Build.Tasks.Packaging , Microsoft.DotNet.Build.Tasks.SharedFramework.Sdk , Microsoft.DotNet.Build.Tasks.TargetFramework.Sdk , Microsoft.DotNet.CodeAnalysis , Microsoft.DotNet.GenAPI , Microsoft.DotNet.XUnitConsoleRunner , Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.RemoteExecutor , Microsoft.DotNet.VersionTools.Tasks , Microsoft.DotNet.ApiCompat From Version 5.0.0-beta.20407.3 -> To Version 5.0.0-beta.20417.6 * Update dependencies from https://github.com/mono/linker build 20200818.3 Microsoft.NET.ILLink.Tasks From Version 5.0.0-preview.3.20413.1 -> To Version 5.0.0-preview.3.20418.3 * Update dependencies from https://github.com/dotnet/llvm-project build 20200818.1 runtime.linux-x64.Microsoft.NETCore.Runtime.Mono.LLVM.Tools , runtime.win-x64.Microsoft.NETCore.Runtime.Mono.LLVM.Tools , runtime.win-x64.Microsoft.NETCore.Runtime.Mono.LLVM.Sdk , runtime.osx.10.12-x64.Microsoft.NETCore.Runtime.Mono.LLVM.Tools , runtime.osx.10.12-x64.Microsoft.NETCore.Runtime.Mono.LLVM.Sdk , runtime.linux-arm64.Microsoft.NETCore.Runtime.Mono.LLVM.Tools , runtime.linux-arm64.Microsoft.NETCore.Runtime.Mono.LLVM.Sdk , runtime.linux-x64.Microsoft.NETCore.Runtime.Mono.LLVM.Sdk From Version 9.0.1-alpha.1.20410.1 -> To Version 9.0.1-alpha.1.20418.1 * Update dependencies from https://github.com/dotnet/xharness build 20200819.1 Microsoft.DotNet.XHarness.CLI , Microsoft.DotNet.XHarness.TestRunners.Xunit From Version 1.0.0-prerelease.20411.1 -> To Version 1.0.0-prerelease.20419.1 * Update dependencies from https://github.com/dotnet/icu build 20200818.2 Microsoft.NETCore.Runtime.ICU.Transport From Version 5.0.0-preview.8.20410.1 -> To Version 5.0.0-preview.8.20418.2 * Update dependencies from https://github.com/dotnet/xharness build 20200819.2 Microsoft.DotNet.XHarness.CLI , Microsoft.DotNet.XHarness.TestRunners.Xunit From Version 1.0.0-prerelease.20411.1 -> To Version 1.0.0-prerelease.20419.2 * Update dependencies from https://github.com/dotnet/runtime-assets build 20200820.1 System.ComponentModel.TypeConverter.TestData , System.Drawing.Common.TestData , System.IO.Compression.TestData , System.IO.Packaging.TestData , System.Net.TestData , System.Private.Runtime.UnicodeData , System.Security.Cryptography.X509Certificates.TestData , System.Windows.Extensions.TestData From Version 5.0.0-beta.20377.1 -> To Version 5.0.0-beta.20420.1 * Update dependencies from https://github.com/mono/linker build 20200820.2 Microsoft.NET.ILLink.Tasks From Version 5.0.0-preview.3.20413.1 -> To Version 6.0.0-alpha.1.20420.2 * Update dependencies from https://github.com/dotnet/arcade build 20200819.21 Microsoft.DotNet.GenFacades , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Build.Tasks.Packaging , Microsoft.DotNet.Build.Tasks.SharedFramework.Sdk , Microsoft.DotNet.Build.Tasks.TargetFramework.Sdk , Microsoft.DotNet.CodeAnalysis , Microsoft.DotNet.GenAPI , Microsoft.DotNet.XUnitExtensions , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.RemoteExecutor , Microsoft.DotNet.VersionTools.Tasks , Microsoft.DotNet.XUnitConsoleRunner , Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.ApiCompat From Version 5.0.0-beta.20417.5 -> To Version 5.0.0-beta.20419.21 * Update dependencies from https://github.com/microsoft/vstest build 20200821-01 Microsoft.NET.Test.Sdk From Version 16.8.0-preview-20200730-03 -> To Version 16.8.0-preview-20200821-01 * Update dependencies from https://github.com/dotnet/xharness build 20200820.1 Microsoft.DotNet.XHarness.CLI , Microsoft.DotNet.XHarness.TestRunners.Xunit From Version 1.0.0-prerelease.20411.1 -> To Version 1.0.0-prerelease.20420.1 * Pin compilers version * Add dotnet6 feed to NuGet.config * Update dependencies from https://github.com/mono/linker build 20200822.1 Microsoft.NET.ILLink.Tasks From Version 5.0.0-preview.3.20413.1 -> To Version 6.0.0-alpha.1.20422.1 * Update dependencies from https://github.com/dotnet/arcade build 20200819.21 Microsoft.DotNet.GenFacades , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Build.Tasks.Packaging , Microsoft.DotNet.Build.Tasks.SharedFramework.Sdk , Microsoft.DotNet.Build.Tasks.TargetFramework.Sdk , Microsoft.DotNet.CodeAnalysis , Microsoft.DotNet.GenAPI , Microsoft.DotNet.XUnitExtensions , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.RemoteExecutor , Microsoft.DotNet.VersionTools.Tasks , Microsoft.DotNet.XUnitConsoleRunner , Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.ApiCompat From Version 5.0.0-beta.20417.5 -> To Version 5.0.0-beta.20419.21 * Update dependencies from https://github.com/microsoft/vstest build 20200821-04 Microsoft.NET.Test.Sdk From Version 16.8.0-preview-20200730-03 -> To Version 16.8.0-release-20200821-04 Co-authored-by: dotnet-maestro[bot] <dotnet-maestro[bot]@users.noreply.github.com> Co-authored-by: Matt Mitchell <mmitche@microsoft.com> Co-authored-by: Santiago Fernandez Madero <safern@microsoft.com> * Remove unused local (#40967) issue dotnet#30457 remove unused local variable dwErrorCode at line 248 * Fix Array.Sort in the presence of nulls (#41234) * Add failing test case * Fix Array.Sort in the presence of nulls * Fix wrong copy prop after `ASG(promoted LCL_VAR with 1 field, call)`. (#41197) * Add Andy's repro test. * Andy's fix. * fix other places * Improve dump. * [mono] Add more detailed logs for the PInvoke resolution process (#41228) Similar logging was added to the managed loading algorithm when it proved extremely useful for debugging a Blazor crash, so the equivalent is probably useful for the unmanaged library algorithm as well * remove unused local (#41226) * remove unused local issue dotnet#30457 remove unused local variable fullPath at line 415 * Update src/libraries/System.Security.Cryptography.X509Certificates/src/System/Security/Cryptography/X509Certificates/X509Certificate2.cs Co-authored-by: Kevin Gosse <krix33@gmail.com> Co-authored-by: Kevin Gosse <krix33@gmail.com> * Fix perf regression in IntPtr operators on 32-bit platforms (#41198) Switching to C# built-in uint/nuint types caused these operators to use long/ulong IntPtr/UIntPtr constructors instead of int/uint IntPtr constructors that it were used before. The fix is to avoid going through the IntPtr/UIntPtr constructors and just depend on the built-in uint/nuint implicit conversions. Fixes #41167 * more managed pointer rooting (#40934) Update more bindings and library code to store managed pointers in roots so that objects can't be collected or moved. Fixes some bugs in the wasm root APIs. * [mono] Thread DllImportSearchPath flags down to LoadLibraryEx (#41229) This mostly completes support for the attribute. Our algorithm differs a bit from CoreCLR right now, so we don't support the legacy behavior flag since it makes less since in the context of our algorithm, but this should be good enough for most cases. * [Wasm] Reduce max length on browser for ToArrayShouldWorkWithSpecialLengthLazyEnumerables Linq test (#41241) Addresses the occasional timeouts in System.Linq tests. The last 3-4 iterations of ToArrayShouldWorkWithSpecialLengthLazyEnumerables take between 20 - 56 seconds each and likely trigger longer GC runs. Fixes https://github.com/dotnet/runtime/issues/41114 Co-authored-by: Steve Pfister <steve.pfister@microsoft.com> Co-authored-by: Alexander Köplinger <alex.koeplinger@outlook.com> * Fix MethodInfo Emit for generic interfaces (#40587) * update helixprecommands (#41205) * Re-enable testing on OSX 10.15 (#40524) * Re-enable testing on OSX 10.15 * ActiveIssue for failing font issue * Fix Char.GetUnicodeCategory to returns correct results (#41200) * Fix Char.GetUnicodeCategory to returns correct results * Delete dead code and obsolete comments (#41211) * Fix wasm sample after that was broken after this PR: https://github.com/dotnet/runtime/pull/40478 (#41277) * [coop] Add debug option to disable stack scan (#40985) When this is enabled, the GC doesn't scan stack or registers. This mode is meant to disable scanning of native frames. This implementation only makes sense in full interpreter mode, where the interp stack is scanned separately from the thread stack. Longer term we could use this mode to scan only the managed stack, also in jit mode. The point of this mode is to be able to stress test the GC in a wasm like environment, where stack can't be scanned. We can more easily reproduce and investigate GC crashes in this configuration, that would also happen on wasm. * map macOS compat 10.16 version to 11.0 (#41176) * map macOS compat 10.16 version to 11.0 * feedback from review * fix GetAsync_SupportedSSLVersion_Succeeds test (#41056) * Correctly enumerate files in Microsoft.NETCore.BrowserDebugHost.Transport.pkgproj (#41290) On CI when the repo is built from a clean checkout in one go we were evaluating the wildcard in the ItemGroup before the files were actually built which meant they weren't put into the nuget package. We need to list each file explicitly to fix that. * Fix half equality method (#41259) * Align Half.Equals implementation with Double.Equals. * Add tests for new Half.Equals behaviour. * Fix GetAppdomainStaticAddress for non-collectible assemblies (#41076) * Improve tailcallstress testing (#41059) 1. Dispatch all tail calls under TailCallStress via helpers. That increases our coverage since valid non-tail-prefixed calls are dispatched as fast calls in normal non-stress mode. 2. Don't attempt to tail call from methods that have a localloc unless there is an explicit tail prefix. Note that we already disallowed fast tail calls from such methods so this change only affects tailcallstress mode. 3. Fix a bug in TestInvokeDOPAndCancel. As the test was written this assert was firing under tailcallstress: https://github.com/dotnet/runtime/blob/480c49b2419ab4a0b34bfd86754abc2f17079c77/src/libraries/System.Threading.Tasks.Parallel/tests/ParallelForTests.cs#L1074 When this call to `cts.Cancel`: https://github.com/dotnet/runtime/blob/480c49b2419ab4a0b34bfd86754abc2f17079c77/src/libraries/System.Threading.Tasks.Parallel/tests/ParallelForTests.cs#L1065 is dispatched via helpers, it takes much longer than when it's dispatched via a fast tail call (we have to jit a couple of IL stubs). Because of that, it's possible that all other actions complete on the other thread before cancellation is completed. * Avoid following invalid pointers in mono_w32process_get_modules on Darwin. (#41115) `_dyld_get_image_name` and `_dyld_get_image_header` can return pointers to data that will be freed or unmapped by concurrent calls to `dlclose`. Avoid this by using `_dyld_register_func_for_add_image` and `_dyld_register_func_for_remove_image` to install callbacks that maintain a runtime-internal set of loaded dynamic libraries. Some notes: - dyld doesn't provide a public "app store friendly" API (like `dl_iterate_phdr`) to traverse a snapshot of all loaded dynamic libraries in a process. There are two alternative private APIs that do this: one uses `_dyld_process_info_create` and friends and is present in macOS 10.12 and above, and the other depends on `_dyld_get_all_image_infos`, which is documented by Apple (... in a WWDC video) to "go away" at some point in dyld3. - `dladdr` linearly scans the dyld loaded image array (and each image's segment set) for a matching address, so `image_added` will get slower as more dynamic libraries are loaded in the process. But dyld3's implementation of dlopen already scales linearly in time with the number of loaded libraries. The use of `dladdr` can be avoided by using `_dyld_register_for_image_loads` or `_dyld_register_for_bulk_image_loads`, but those are both private APIs. - The callbacks registered with `_dyld_register_func_for_add_image` and `_dyld_register_func_for_remove_image` can never be unregistered, so this is yet another thing that makes it impossible to safely unload an embedded mono. - `MonoW32ProcessModule::inode` only seems to be used to avoid dupicates in other implementations of w32process; sorting the returned module list in `mono_w32process_get_modules` wouldn't be necessary if the `mono_dyld_image_info`s were also kept in an intrusive circular doubly linked list. - It looks like `mono_w32process_get_modules` in `w32process-unix-bsd.c` has a similar race condition. Also see: https://github.com/mono/mono/issues/20107 Co-authored-by: imhameed <imhameed@users.noreply.github.com> * Correct a few mistakes in the ComInterop code (#40234) * Enable Crossgen2 and Blazor Perf Tests on Linux (#41270) * copy changes from blazor PR * add crossgen2 job on linux * fix helixprecommands issue * fix internal param * will revert: add back PR trigger * fix category exclusion * reset PR trigger * Modified #41008 without GetOverlappedResultEx (#41236) * Revert "Revert "Improve Windows error handling in Diagnostics IPC (#41008)" (#41220)" This reverts commit 15839c0294b51d8bb9b7f689669da86ebe64546d. * don't use win8+ API * Port EventPipe CoreCLR changes to Mono. (#41152) * [MonoVM] Enable startup events over EventPipe. Mono port of CoreCLR PR: https://github.com/dotnet/runtime/pull/36720. * [MonoVM] Add the ability for profilers to listen to EventPipe events. Mono port of CoreCLR PR: https://github.com/dotnet/runtime/pull/37002. * [MonoVM] Add ProcessInfo command to Diagnostics Server. Mono port of CoreCLR PR: https://github.com/dotnet/runtime/pull/38967. * [MonoVM] Unregister callback when setting a provider for deferred delete. Mono port of CoreCLR PR: https://github.com/dotnet/runtime/pull/40191. * [MonoVM] Generational aware analysis based on environment variable. Mono port of CoreCLR PR: https://github.com/dotnet/runtime/pull/40332. NOTE, the ability to pause a session was not ported since it currently have no use in Mono and looks like a temporary solution in CoreCLR. * [MonoVM] Rebrand Diagnostics Server to DiagnosticPort. Mono port of CoreCLR PR: https://github.com/dotnet/runtime/pull/40499. * Missed cases not discovered during first phase (#40382) Co-authored-by: Jan Jahoda <jajahoda@.microsoft.com> * [master] Update dependencies from dotnet/xharness mono/linker (#41257) [master] Update dependencies from dotnet/xharness mono/linker - Updates: - Microsoft.DotNet.XHarness.CLI: from 1.0.0-prerelease.20420.1 to 1.0.0-prerelease.20424.1 - Microsoft.DotNet.XHarness.TestRunners.Xunit: from 1.0.0-prerelease.20420.1 to 1.0.0-prerelease.20424.1 - Updates: - Microsoft.NET.ILLink.Tasks: from 6.0.0-alpha.1.20422.1 to 6.0.0-alpha.1.20424.3 * Mark System.Diagnostics.FileVersionInfo as unsupported on Browser (#41271) Co-authored-by: Mitchell Hwang <mitchell.hwang@microsoft.com> * Exclude Mono.Cecil assemblies from signing (#41325) We're just shipping them in the Microsoft.NETCore.BrowserDebugHost.Transport package. * Fix tailcall regression with compiled F# (#41206) * Fix tailcall regression with compiled F# This change skips instantiating stubs for direct tailcalls and instead passes the inst argument directly to the target method. Fixes #40864 * Add regression test Co-authored-by: Jakob Botsch Nielsen <Jakob.botsch.nielsen@gmail.com> * Prevent invoking property 'init' method (#41293) * HTTP/3 interop fixes (#41298) Support DNS-based quic connections for SNI. Support multiple ALPN values. Update H3 ALPN from "h3" to "h3-29". * Fix NRE introduced in FileSystemWatcher during nullability annotations (#41315) * Fix NRE introduced during nullability annotations * Apply Jeff's suggestions from code review Co-authored-by: Jeff Handley <jeffhandley@users.noreply.github.com> Co-authored-by: Jeff Handley <jeffhandley@users.noreply.github.com> * [runtime] Catch attempts to create enum types with an underlying type that is itself an incomplete enum type. (#41165) * Catch attempts to create enum types with an underlying type that is itself an incomplete enum type. Previously, only non-enum types containing fields with incomplete enum types were forbidden. * Permit static constant fields with incomplete types if the incomplete type is the type currently being defined Co-authored-by: Alexis Christoforides <alexis@thenull.net> Co-authored-by: imhameed <imhameed@users.noreply.github.com> * Fix signing in official build with 3rd party libraries (#41346) * Fix signing in official build with 3rd party libraries * Update eng/Signing.props Co-authored-by: Alexander Köplinger <alex.koeplinger@outlook.com> * Add tests mapping invalid JSON to objects with parameterized ctors (#41121) * Move invalid JSON tests to dedicated file * Add tests mapping invalid JSON to objects with parameterized ctors * [mono] Pass -Wc++-compat on all platforms, -Werror on OSX, fix errors (#41233) This PR: * Enables `-Wc++-compat` on all platforms: we support building the runtime as C++, so this helps catch errors before they're found in CI and will help keep the netcore build C++-compatible * Sets `-Werror` on netcore CI for OSX It also fixes various warnings currently present in the build. `-Werror` should probably be extended to cover all CI platforms eventually. Adding a new platform is as simple as adding to the line in mono.proj whitelisting platforms and fixing the warnings. Not sure if there's a better way to handle the cpuidex function, but if we're using that from mini it should definitely be in a header somewhere. * Fix \u00A7 char Unicode category (#41343) * Add sleep in hijacking test (#41340) Calling GC.Collect in a tight loop is unnecessary for the test and causing timeouts in CI. Fixes #40916 * Throw from CodeBase if assembly is in the bundle (#40974) Also adds test for Module.FullyQualifiedName * Support polymorphic value-type converters and add validation on values returned by custom converters (#40914) * IL emit Box/Unbox in member access * Test case for object converter * Test case for class with primitive members and object converter * Address feedback * Fixed issues with JsonConverter<object>and nullable value-types * Address feedback * Fix test for .NETStandard version * Fix #41146 * Accidentally commited the local test hack * Throw JsonException when a deserialized value cannot be assigned to the property/field * Fix merge * Fix merge again * Undo Fix #41146 * Consolidate nullable annotations * Addressed feedback * Addressed feedback * Webassembly Runtime Tests (#38556) Enable runtime tests for the Wasm Architecture on Mono runtime. Co-authored-by: Jan Kotas <jkotas@microsoft.com> Co-authored-by: Fan Yang <yangfan@microsoft.com> Co-authored-by: jashook <jashoo@microsoft.com> Co-authored-by: Tomas Rylek <trylek@microsoft.com> * set TLS versions explicitly to prevent SNI test failure on Deb10 (#41014) * add extra padding for icmp6stat to support macOS 11 (#41179) * add extra padding for icmp6stat to support macOS 11 * update SystemNative_GetIcmpv6GlobalStatistics to use variable size * add cast * avoid realloc * Restrict generational aware analysis to processes that matches the command line configuration (#41052) * Disable the Platform Compat Analyzer until #41354 is fixed (#41367) * update macOS 11 rid (#41044) * update macOS 11 rid * add compat 10.16 as well * [HeapVerify=2] Fix Assert failure "Pointer updated without using write barrier" (#41375) * [HeapVerify=2] Fix Assert failure "Pointer updated without using write barrier" * Change Environment.CurrentManagedThreadId back to FCall (#41360) * Reset all Uri fields when reusing an instance (#41324) * Reset all Uri fields when reusing an instance * Test combined Uri's AbsoluteUri * [mono] Add mono_assembly_decref, return value on both decref and addref (#41342) * [wasm][browser] Fix System.IO.FileSystem failing tests requiring nanosecond/millisecond granularity (#41288) * Emscripten lacks ns/ms granularity which I included a switch for and added a browser platform check * Emscripten takes maximum of utime and atime so I set utime and atime together for Browser * Add netcoreapp build of System.Formats.Asn1 This avoids the library causing a cyclic dependency in generating netstandard.dll. The netcoreapp version is not included in the NuGet package, following the pattern of * System.Text.Json * System.Collections.Immutable * System.Reflection.Metadata * Retry up to three times for X509Chain timeout tests * HTTP stress docker fix (#41397) Fixed docker files not using currently built runtime for http stress test. * Nullability annotations for HttpListener (#41189) * src files annotated * ref file * Address feedback * Exclude Fannkuch_9 test on ARM (#41380) * select Debian10 over Debian9 for PR validation (#41015) * Added VersionConverter (#41384) * Added VersionConverter * Nullable annotation * [VisualBasic] Fix tests for Android (#41378) Remove some empty tests and add data for others. Resolves #37066 * Three fixes for R2RDump bugs I discovered while investigating #38290 (#41303) 1) As ReadyToRunMethod newly looks at method bodies, in the composite case we need to pass around per-module IL info, not only the metadata reader. I have replaced such usages of MetadataReader with the new interface IAssemblyMetadata and I implemented the standalone variant StandaloneAssemblyMetadata. 2) I hit an UnwindInfo decoding issue and I found out that the UnwindCode decoder is weirdly split between the UnwindCode ctor and a separate method ParseUnwindCode that weren't completely consistent in whether the UnwindInfoArray refers to the raw 16-bit entries in the UnwindInfo or to the parsed UnwindCode instances. I have basically migrated ParseUnwindCode into the UnwindCode ctor and I simplified the related logic. 3) We shouldn't block method dump on header dump, otherwise there's no way to dump both headers and methods in one R2RDump execution. Thanks Tomas * Nullable: System.Xml, part 7 (Serialization) (#41261) * Nullable: System.Xml, part 7 (Serialization) * Fix build errors on Release (ifdef-ed code) * fix incidental product bug * address PR feedback * Make XmlSerializer._rootType nullable * BrowserDebugProxy: bump package versions to latest stable (#41436) Fixes https://github.com/dotnet/runtime/issues/41356 * Globally enable nullability in XML (#41438) * Globally enable nullability in XML * remove double #nullable enable in the file * Add support for using cmake in the mono runtime build. (#41316) * Add support for using cmake in the mono runtime build. Not enabled by default. Ninja is used for building if available, else we fall back to make. * Build and install libmonosgen-2.0.dylib. * Address review comments. * Add a few more MONO_EMPTY_SOURCE_FILE defines to silence ranlib warnings on osx. * Pass -mmacosx-version-min=10.12 on osx. * Add eventpipe support. * Quiet the OSX build a bit. * Address review comments. * [mono] ALC function restructuring for unloadability (#41344) We previously had a weird split between the domain and loader headers/files. This makes fewer things public, moves the useful ALC functions to loader-internals.h with the implementation in assembly-load-context.c, and cleans up the flow for some unloading behavior. * Issue18826 fix (#35137)  Fixes mono/mono#18826 Co-authored-by: HinTak <HinTak@users.noreply.github.com> * Allow return null from ConnectionListener.AcceptAsync (#41442) Changed API as it was proposed and approved in #41304 Closes #41304 * Fix ArgumentNullException when using minimum json file (#41428) * Console.Unix: fix window size not always being invalidated (#41317) Ensure signal handlers are setup to invalidate cached values for window size. * Fix null check in S.Diagnostics.Contract regressed by nullability annotations (#41382) * Fix null check in S.Diagnostics.Contract regressed by nullability annotations * fix product per Jan's recommendation * Make minimal OSX version consistent between build-native.sh and configurecompiler.cmake (#41435) The latter still specified 10.12 but .NET Core 3+ actually only supports macOS 10.13 or later. * Fix SPMI to handle replays of BBINSTR jit method contexts (#41386) Move the handling of allocMethodBlockCounts to the method context. While this data is both an input and an output, we're more interested in the input side. We could also treat it as an output, if we say wanted to verify that the replay jit wrote the same sequence of IL offsets into the BlockCount records, but that doesn't seem crucial as changes here would also lead to code diffs. Fix the code that checks the AddressMap for reloc hints so it doesn't fail if the exact lookup fails, but instead falls back to a search. This is needed because the base of the replay count buffer is recorded as the map key, but the instrumentation probes in jitted code refer to offsets from this value. Fixes #37270. * Port nullability annotations to System.Xml.ReaderWriter contract (#41083) * Port nullability annotations to System.Xml.ReaderWriter contract assembly * Port annotations of recently updated members in Xml/Serialization * Fix anontations on XmlElement discovered while updating System.Data.Common Xml related annotations * Fix Xml related annotations on System.Data.Common * address feedback Co-authored-by: Krzysztof Wicher <kwicher@microsoft.com> * add tracing to Decryp to complement tracing in Encrypt (#40998) * add tracing to Decryp to complement tracing in Encrypt * feedback from review * revert encrypt * Update SYSLIB0012 obsoletion messages to be clear and consistent (#41467) * Clean up changes in #40914 (#41427) * Fixup JSON equality checks in polymorphic converter tests * Limit deserialization value validation to custom _polymorphic_ converters * Switch text param to nullable in WriteProcessingInstructionAsync (#41481) * Fix perf token (#41345) * remove semicolon from upload token * will revert - test private commands * remove condition * add debug code * add dump task * comment out internal param * rename helixprecommands to sharedhelixprecommands * update and revert for check-in * improve InitializeCurrentThread (#41440) * Remove if-def from JSON ref file previously used to support linker attributes (#41484) * Remove if-def from JSON ref file previously used to support linker attributes * Run GenerateReferenceSource target on ref project * Switch Markup property to XmlNode?[]? to reflect it can contain null elements (#41488) * Switch Markup property to XmlNode?[]? to reflect it can contain null elements * Update ref * [wasm][debugger] Make sure `System.Private.CoreLib.dll` is in sync with (#41469) .. it's pdb. Running debugger-tests, I saw lot of messages like: `CWL: : /Users/radical/dev/runtime/src/mono/mono/metadata/debug-mono-ppdb.c:230 <disabled>` .. which is trying to print: `CWL: Symbol file data-0x1c38008 doesn't match image System.Private.CoreLib.dll` And indeed the dll is out of sync with it's pdb. We need to copy the pdb too. * Port nullability annotations to refs XmlDocument and XmlSerializer (#41474) * Port nullability annotations to refs XmlDocument and XmlSerializer * Fix new System.Data.Common nullability related errors * Switch nullability of parameter in SoapElementAttribute ctor * Port nullability annotations from System.Private.Xml.Linq to contracts (#41086) * Enable nullability on System.Xml.XPath.XDocument src * Port nullability annotations to System.Xml.XPath.XDocument * Port nullability annotations to System.Xml.XDocument * Add ? to XmlAttribute argument in int? explicit operator * mono.proj: properly pass -mmacosx-version-min to CFLAGS for OSX (#41433) Noticed we were missing this while looking at another PR. * Bump the local emscripten version to 2.0.1. (#41504) * Fixes Windows docker debug prints. (#41443) * Client side logging for HTTP stress. (#41444) * Bump emscripten v…

josalem added the area-Diagnostics-coreclr label May 19, 2020

josalem added this to the 5.0 milestone May 19, 2020

josalem requested review from noahfalk and sywhang May 19, 2020 19:37

josalem self-assigned this May 19, 2020

jkotas reviewed May 19, 2020

View reviewed changes

src/coreclr/src/vm/eventpipe.cpp Outdated Show resolved Hide resolved

jkotas reviewed May 19, 2020

View reviewed changes

src/coreclr/src/vm/eventpipe.cpp Outdated Show resolved Hide resolved

jkotas reviewed May 19, 2020

View reviewed changes

src/coreclr/src/vm/eventpipe.cpp Outdated Show resolved Hide resolved

sywhang reviewed May 20, 2020

View reviewed changes

noahfalk requested changes May 20, 2020

View reviewed changes

lambdageek mentioned this pull request May 21, 2020

Tracking: Diagnostic Server and EventPipe C++ to C backport #36820

Closed

leculver reviewed May 25, 2020

View reviewed changes

src/coreclr/tests/src/tracing/eventpipe/common/IpcUtils.cs Outdated Show resolved Hide resolved

John Salem added 11 commits May 29, 2020 16:53

initial hack to get startup stop working

777ac13

clean up

a0984f4

Modify reverse tests to not stop on start

e71676b

Add tests for stop on start functionality

4b7e25c

remove dead code

5acf205

PR feedback

220e09a

* only init event if we use it * read only necessary config values * don't new in static constructor

prevent dangling pointer dereference

1a79e49

Refactor PauseOnStart into DiagnosticServer

1a5581a

refactor command ids and check enabled status

7117d8f

* move all command ids to diagnosticsipc.h * add check for command enabled status based on fEEStartup

Rename stoponstart to pauseonstart

d86cdf3

Updated tests to gracefully exit

4ea11af

josalem force-pushed the dev/josalem/startup-events-over-eventpipe branch from cc74515 to 4ea11af Compare May 29, 2020 23:59

John Salem added 3 commits May 29, 2020 17:51

clean up test

34e789d

Fix Startup event on Windows

a95f6a1

polish reverse tests

1d36c78

noahfalk approved these changes Jun 2, 2020

View reviewed changes

noahfalk reviewed Jun 2, 2020

View reviewed changes

src/coreclr/src/vm/diagnosticserver.cpp Outdated Show resolved Hide resolved

sywhang reviewed Jun 3, 2020

View reviewed changes

src/coreclr/src/vm/diagnosticserver.h Show resolved Hide resolved

sywhang reviewed Jun 3, 2020

View reviewed changes

src/coreclr/src/vm/diagnosticserver.h Outdated Show resolved Hide resolved

josalem mentioned this pull request Jun 4, 2020

Refactor FEATURE flags for Diagnostics Server and EventPipe #37438

Open

John Salem added 9 commits June 4, 2020 12:08

move shutdown callback to before DiagServer init

f136acd

refactor enums

e144807

Refactor tests to share code

f78f3b6

Allow disabling sessions while runtime is paused

5ddbd7e

modify test to turn on all keywords

d789f48

Add printf to pause on start

003ba5f

Merge branch 'master' into dev/josalem/startup-events-over-eventpipe

5889f49

Merge remote-tracking branch 'upstream/master' into dev/josalem/start…

774229a

…up-events-over-eventpipe

Fixed namedpipeserver test issue

3440f44

jaredpar mentioned this pull request Jun 12, 2020

OSX machines are de-provisioned during CI / PR runs leading to failures #34472

Closed

make profiler error sensible

863c3b3

josalem merged commit 7a8da1a into dotnet:master Jun 16, 2020

josalem deleted the dev/josalem/startup-events-over-eventpipe branch June 16, 2020 21:46

lateralusX mentioned this pull request Jun 29, 2020

Port EventPipeBuffer, EventPipeBufferList, EventPipeBufferManager to Mono. #37756

Merged

lateralusX added a commit to lateralusX/runtime that referenced this pull request Aug 21, 2020

[MonoVM] Enable startup events over EventPipe.

960a175

Mono port of CoreCLR PR: dotnet#36720.

lateralusX mentioned this pull request Aug 21, 2020

Port EventPipe CoreCLR changes to Mono. #41152

Merged

monojenkins mentioned this pull request Aug 21, 2020

Port EventPipe CoreCLR changes to Mono. mono/mono#20278

Merged

ghost locked as resolved and limited conversation to collaborators Dec 9, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Enable startup events over EventPipe #36720

Enable startup events over EventPipe #36720

josalem commented May 19, 2020

ghost commented May 19, 2020

jkotas commented May 19, 2020 •

edited

Loading

sywhang May 20, 2020

josalem May 20, 2020

noahfalk left a comment

noahfalk May 20, 2020

josalem commented Jun 1, 2020

noahfalk left a comment

Enable startup events over EventPipe #36720

Enable startup events over EventPipe #36720

Conversation

josalem commented May 19, 2020

ghost commented May 19, 2020

jkotas commented May 19, 2020 • edited Loading

sywhang May 20, 2020

Choose a reason for hiding this comment

josalem May 20, 2020

Choose a reason for hiding this comment

noahfalk left a comment

Choose a reason for hiding this comment

noahfalk May 20, 2020

Choose a reason for hiding this comment

josalem commented Jun 1, 2020

noahfalk left a comment

Choose a reason for hiding this comment

jkotas commented May 19, 2020 •

edited

Loading