Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix DiagnosticSource to work with NativeAOT #76109

Merged
merged 7 commits into from
Sep 28, 2022
Merged
Show file tree
Hide file tree
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions eng/pipelines/coreclr/nativeaot-post-build-steps.yml
Original file line number Diff line number Diff line change
Expand Up @@ -29,3 +29,6 @@ steps:
- ${{ if ne(parameters.osGroup, 'windows') }}:
- script: $(Build.SourcesDirectory)/src/tests/run.sh --runnativeaottests $(buildConfigUpper) ${{ parameters.archType }}
displayName: Run tests in single file mode

- script: $(Build.SourcesDirectory)$(dir)build$(scriptExt) -ci -arch ${{ parameters.archType }} $(_osParameter) -s libs.tests -c $(_BuildConfig) /p:TestAssemblies=false /p:TestNativeAot=true $(_officialBuildParameter) $(_crossBuildPropertyArg) /bl:$(Build.SourcesDirectory)/artifacts/log/$(buildConfigUpper)/NativeAotTests.binlog ${{ parameters.extraTestArgs }}
displayName: Run NativeAot Library Tests
4 changes: 3 additions & 1 deletion eng/testing/linker/SupportFiles/Directory.Build.props
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,6 @@
<PropertyGroup>
<SkipConfigureTrimming>true</SkipConfigureTrimming>
<PublishTrimmed>true</PublishTrimmed>
<SkipImportRepoLinkerTargets>true</SkipImportRepoLinkerTargets>
<TrimMode>full</TrimMode>
<TrimmerRemoveSymbols>false</TrimmerRemoveSymbols>
<SelfContained>true</SelfContained>
Expand All @@ -13,5 +12,8 @@
<EnableTrimAnalyzer>false</EnableTrimAnalyzer>
<SuppressTrimAnalysisWarnings>true</SuppressTrimAnalysisWarnings>
<NoWarn>$(NoWarn);IL2121</NoWarn>

<!-- EventSourceSupport is disabled by default with PublishAot=true, so need to enable it here. -->
<EventSourceSupport>true</EventSourceSupport>
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@MichalStrehovsky @jkotas - with this change we now have tests ensuring EventSource works (at least in proc) with PublishAot. Do you think in .NET 8 we should re-enable EventSourceSupport=true by default? Or do you think we should keep it disabled by default going forward (for size savings?)

Copy link
Member

@jkotas jkotas Sep 26, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would wait for feedback on the current .NET 7 default before doing more changes.

I think it makes sense to leave it off by default and turn it on only for server app-models in their SDKs.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agree with Jan. I kind of like the size savings. I would like to see feedback. Re-enabling it is an instant 30% size regression from 7.0 for hello world.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just curious, how big (in bytes) is 30%? I wouldn't have guessed the savings were such a high percentage but I don't know if that is because I overestimated the size of HelloWorld or underestimated the size of EventSource.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, we have a log and I'm hoping to do some offcycle work to visualize/diff them.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've love to see the results of this. Agreed with @noahfalk that this is more than I would have expected. It feels important to describe the diagnostic expectations for native AOT in .NET 8 so that we can determine how this decision fits in.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's a similar difference for PublishTrimmed as well - the size of managed code goes from 2,718,848 bytes to 3,003,008. So the cost is about 300 kB in IL terms. (The repro steps for both is basically just run dotnet publish -r win-x64 -c Release with -p:PublishTrimmed=true for trimming or -p:PublishAot=true for NativeAOT. Use -p:EventSourceSupport=false to enable/disable EventSource.)

For the NativeAOT case a quick breakdown is following.

Size of code per namespace with EventSource enabled

Namespace Size of code in bytes
System 414,414
System.Collections.Generic 246,832
System.Globalization 128,834
Internal.Runtime.TypeLoader 128,583
System.Diagnostics.Tracing 116,975
System.Text 92,958
Internal.TypeSystem 85,341
System.Reflection 40,905
System.Reflection.Runtime.TypeInfos 36,802
System.Collections.Concurrent 34,363
System.Reflection.Runtime.General 31,092
System.Numerics 30,384
System.Threading 27,273
Internal.Reflection.Execution 27,180
System.IO 26,659
System.Resources 25,769
System.Reflection.Runtime.MethodInfos 22,513
System.Runtime 21,400
System.Runtime.CompilerServices 20,287
System.Buffers 20,055

Size of code per namespace with EventSource disabled

Namespace Size of code in bytes
System 306,356
System.Collections.Generic 156,429
Internal.Runtime.TypeLoader 127,630
System.Globalization 107,047
System.Text 86,065
Internal.TypeSystem 85,341
System.Reflection 36,931
System.Reflection.Runtime.TypeInfos 35,576
System.Collections.Concurrent 34,363
System.Numerics 30,384
System.Reflection.Runtime.General 27,776
System.IO 26,299
Internal.Reflection.Execution 25,920
System.Resources 25,445
System.Runtime 21,336

(This is just namespaces with more than 20 kB of code)

Most of the cost is in transitive dependencies. EventSource has a big transitive closure and depends on a lot of framework code.

I can file an issue if there's interest in reducing the transtive closure.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Most of the cost is in transitive dependencies. EventSource has a big transitive closure and depends on a lot of framework code.

This tells me that this is the upper bound of the size savings for having EventSource disabled. In a real world app that uses these transitive dependencies, they won't be able to be trimmed. So the size savings of disabling EventSource will be less. For example, 90 kB of size savings is coming from System.Collections.Generic which many apps will use. When the app uses those collections, they won't be able to be trimmed anymore, even with EventSource disabled.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So the size savings of disabling EventSource will be less. For example, 90 kB of size savings is coming from System.Collections.Generic which many apps will use. When the app uses those collections, they won't be able to be trimmed anymore, even with EventSource disabled

That depends - when thinking about native code, one needs to think about instantiated generic code. For example the costs associated with System.Collections.Generic.Dictionary2<System.Type,System.Diagnostics.Tracing.TraceLoggingTypeInfo>` are exclusive to EventSource not matter what the app does.

It's easy to test out theories - just pick a bigger app and see the impact of EventSource. For apps the size of current ASP.NET, an extra 600 kB overhead is negligible either way,

eerhardt marked this conversation as resolved.
Show resolved Hide resolved
</PropertyGroup>
</Project>
4 changes: 4 additions & 0 deletions eng/testing/linker/SupportFiles/Directory.Build.targets
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,10 @@
DependsOnTargets="BundleTestWasmApp"
Condition="'$(TargetArchitecture)' == 'wasm' And '$(TargetOS)' == 'browser'" />

<PropertyGroup Condition="'$(PublishAot)' == 'true'">
<ILCompilerTargetsPath>$(CoreCLRBuildIntegrationDir)Microsoft.DotNet.ILCompiler.SingleEntry.targets</ILCompilerTargetsPath>
</PropertyGroup>

<!-- Overriding these targets as these projects won't need to binplace -->
<Target Name="PublishTestAsSelfContained" />

Expand Down
8 changes: 8 additions & 0 deletions eng/testing/linker/project.csproj.template
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@
<NETCoreAppMaximumVersion>{NetCoreAppMaximumVersion}</NETCoreAppMaximumVersion>
<UseMonoRuntime>{UseMonoRuntime}</UseMonoRuntime>
<RuntimeIdentifier>{RuntimeIdentifier}</RuntimeIdentifier>
<PublishAot>{PublishAot}</PublishAot>

<!-- wasm specific -->
<MonoAOTCompilerDir>{MonoAOTCompilerDir}</MonoAOTCompilerDir>
Expand All @@ -25,6 +26,13 @@

<RepositoryEngineeringDir>{RepositoryEngineeringDir}</RepositoryEngineeringDir>
<_ExtraTrimmerArgs>{ExtraTrimmerArgs} $(_ExtraTrimmerArgs)</_ExtraTrimmerArgs>

<!-- Needed for PublishAot -->
<IlcToolsPath>{IlcToolsPath}</IlcToolsPath>
<IlcBuildTasksPath>{IlcBuildTasksPath}</IlcBuildTasksPath>
<IlcSdkPath>{IlcSdkPath}</IlcSdkPath>
<IlcFrameworkPath>{IlcFrameworkPath}</IlcFrameworkPath>
<CoreCLRBuildIntegrationDir>{CoreCLRBuildIntegrationDir}</CoreCLRBuildIntegrationDir>
</PropertyGroup>

<ItemGroup>
Expand Down
6 changes: 6 additions & 0 deletions eng/testing/linker/trimmingTests.targets
Original file line number Diff line number Diff line change
Expand Up @@ -85,8 +85,14 @@
.Replace('{NetCoreAppMaximumVersion}', '$(NetCoreAppMaximumVersion)')
.Replace('{UseMonoRuntime}','$(UseMonoRuntime)')
.Replace('{RuntimeIdentifier}','%(TestConsoleApps.TestRuntimeIdentifier)')
.Replace('{PublishAot}','$(IsNativeAotTestProject)')
.Replace('{MicrosoftNETILLinkTasksVersion}', '$(MicrosoftNETILLinkTasksVersion)')
.Replace('{ExtraTrimmerArgs}', '%(TestConsoleApps.ExtraTrimmerArgs)')
.Replace('{IlcToolsPath}', '$(CoreCLRILCompilerDir)')
.Replace('{IlcBuildTasksPath}', '$(CoreCLRILCompilerDir)netstandard/ILCompiler.Build.Tasks.dll')
.Replace('{IlcSdkPath}', '$(CoreCLRAotSdkDir)')
.Replace('{IlcFrameworkPath}', '$(MicrosoftNetCoreAppRuntimePackRidLibTfmDir)')
.Replace('{CoreCLRBuildIntegrationDir}', '$(CoreCLRBuildIntegrationDir)')
.Replace('{RuntimeHostConfigurationOptions}', '$(_runtimeHostConfigurationOptionsString)')
.Replace('{AdditionalProjectReferences}', '$(_additionalProjectReferencesString)')
.Replace('{RepositoryEngineeringDir}', '$(RepositoryEngineeringDir)')
Expand Down
10 changes: 6 additions & 4 deletions src/libraries/Directory.Build.props
Original file line number Diff line number Diff line change
Expand Up @@ -22,10 +22,12 @@
<PropertyGroup Condition="$(MSBuildProjectFullPath.Contains('$([System.IO.Path]::DirectorySeparatorChar)tests$([System.IO.Path]::DirectorySeparatorChar)'))">
<IsTestProject Condition="$(MSBuildProjectName.EndsWith('.UnitTests')) or $(MSBuildProjectName.EndsWith('.Tests'))">true</IsTestProject>
<IsTrimmingTestProject Condition="$(MSBuildProjectName.EndsWith('.TrimmingTests'))">true</IsTrimmingTestProject>
<IsTestSupportProject Condition="'$(IsTestProject)' != 'true' and '$(IsTrimmingTestProject)' != 'true'">true</IsTestSupportProject>
<IsNativeAotTestProject Condition="$(MSBuildProjectName.EndsWith('.NativeAotTests'))">true</IsNativeAotTestProject>
<IsPublishedAppTestProject Condition="'$(IsTrimmingTestProject)' == 'true' or '$(IsNativeAotTestProject)' == 'true'">true</IsPublishedAppTestProject>
<IsTestSupportProject Condition="'$(IsTestProject)' != 'true' and '$(IsPublishedAppTestProject)' != 'true'">true</IsTestSupportProject>

<!-- Treat test assemblies as non-shipping (do not publish or sign them). -->
<IsShipping Condition="'$(IsTestProject)' == 'true' or '$(IsTestSupportProject)' == 'true' or '$(IsTrimmingTestProject)' == 'true'">false</IsShipping>
<IsShipping Condition="'$(IsTestProject)' == 'true' or '$(IsTestSupportProject)' == 'true' or '$(IsPublishedAppTestProject)' == 'true'">false</IsShipping>
</PropertyGroup>

<PropertyGroup>
Expand All @@ -36,14 +38,14 @@
'$(IsReferenceAssemblyProject)' != 'true' and
'$(IsGeneratorProject)' != 'true' and
'$(IsTestProject)' != 'true' and
'$(IsTrimmingTestProject)' != 'true' and
'$(IsPublishedAppTestProject)' != 'true' and
'$(IsTestSupportProject)' != 'true' and
'$(UsingMicrosoftNoTargetsSdk)' != 'true' and
'$(UsingMicrosoftTraversalSdk)' != 'true'">true</IsSourceProject>
</PropertyGroup>

<!-- Warnings that should be disabled in our test projects. -->
<PropertyGroup Condition="'$(IsTestProject)' == 'true' or '$(IsTestSupportProject)' == 'true' or '$(IsTrimmingTestProject)' == 'true'">
<PropertyGroup Condition="'$(IsTestProject)' == 'true' or '$(IsTestSupportProject)' == 'true' or '$(IsPublishedAppTestProject)' == 'true'">
<!-- don't warn on usage of BinaryFormatter from test projects -->
<NoWarn>$(NoWarn);SYSLIB0011</NoWarn>
<!-- don't warn about unnecessary trim warning suppressions. can be removed with preview 6. -->
Expand Down
2 changes: 1 addition & 1 deletion src/libraries/Directory.Build.targets
Original file line number Diff line number Diff line change
Expand Up @@ -127,7 +127,7 @@
<Import Project="$(RepositoryEngineeringDir)references.targets" />
<Import Project="$(RepositoryEngineeringDir)resolveContract.targets" />
<Import Project="$(RepositoryEngineeringDir)testing\tests.targets" Condition="'$(EnableTestSupport)' == 'true'" />
<Import Project="$(RepositoryEngineeringDir)testing\linker\trimmingTests.targets" Condition="'$(IsTrimmingTestProject)' == 'true'" />
<Import Project="$(RepositoryEngineeringDir)testing\linker\trimmingTests.targets" Condition="'$(IsPublishedAppTestProject)' == 'true'" />
<Import Project="$(RepositoryEngineeringDir)testing\runtimeConfiguration.targets" />
<Import Project="$(RepositoryEngineeringDir)testing\runsettings.targets" Condition="'$(EnableRunSettingsSupport)' == 'true'" />
<Import Project="$(RepositoryEngineeringDir)testing\coverage.targets" Condition="'$(EnableRunSettingsSupport)' == 'true' or '$(EnableCoverageSupport)' == 'true'" />
Expand Down
2 changes: 0 additions & 2 deletions src/libraries/Microsoft.Extensions.Hosting/src/HostBuilder.cs
Original file line number Diff line number Diff line change
Expand Up @@ -188,8 +188,6 @@ internal static DiagnosticListener LogHostBuilding(HostApplicationBuilder hostAp
return diagnosticListener;
}

[UnconditionalSuppressMessage("AOT", "IL3050:RequiresDynamicCode",
Justification = "DiagnosticSource is used here to pass objects in-memory to code using HostFactoryResolver. This won't require creating new generic types.")]
[UnconditionalSuppressMessage("ReflectionAnalysis", "IL2026:UnrecognizedReflectionPattern",
Justification = "The values being passed into Write are being consumed by the application already.")]
private static void Write<T>(
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,6 @@ public virtual void Dispose() { }
public virtual System.IDisposable Subscribe(System.IObserver<System.Collections.Generic.KeyValuePair<string, object?>> observer, System.Predicate<string>? isEnabled) { throw null; }
public override string ToString() { throw null; }
[System.Diagnostics.CodeAnalysis.RequiresUnreferencedCode("The type of object being written to DiagnosticSource cannot be discovered statically.")]
[System.Diagnostics.CodeAnalysis.RequiresDynamicCode("DiagnosticSource may require creating new generic types or methods, which requires creating code at runtime. This may not work when AOT compiling.")]
public override void Write(string name, object? value) { }
}
public abstract partial class DiagnosticSource
Expand All @@ -29,7 +28,6 @@ protected DiagnosticSource() { }
public abstract bool IsEnabled(string name);
public virtual bool IsEnabled(string name, object? arg1, object? arg2 = null) { throw null; }
[System.Diagnostics.CodeAnalysis.RequiresUnreferencedCode("The type of object being written to DiagnosticSource cannot be discovered statically.")]
[System.Diagnostics.CodeAnalysis.RequiresDynamicCode("DiagnosticSource may require creating new generic types or methods, which requires creating code at runtime. This may not work when AOT compiling.")]
public abstract void Write(string name, object? value);
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -197,10 +197,8 @@ public abstract partial class DiagnosticSource
public virtual void OnActivityExport(System.Diagnostics.Activity activity, object? payload) { }
public virtual void OnActivityImport(System.Diagnostics.Activity activity, object? payload) { }
[System.Diagnostics.CodeAnalysis.RequiresUnreferencedCode("The type of object being written to DiagnosticSource cannot be discovered statically.")]
[System.Diagnostics.CodeAnalysis.RequiresDynamicCode("DiagnosticSource may require creating new generic types or methods, which requires creating code at runtime. This may not work when AOT compiling.")]
public System.Diagnostics.Activity StartActivity(System.Diagnostics.Activity activity, object? args) { throw null; }
[System.Diagnostics.CodeAnalysis.RequiresUnreferencedCode("The type of object being written to DiagnosticSource cannot be discovered statically.")]
[System.Diagnostics.CodeAnalysis.RequiresDynamicCode("DiagnosticSource may require creating new generic types or methods, which requires creating code at runtime. This may not work when AOT compiling.")]
public void StopActivity(System.Diagnostics.Activity activity, object? args) { }
}
public enum ActivitySamplingResult
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -87,6 +87,8 @@ System.Diagnostics.DiagnosticSource</PackageDescription>
<Compile Include="System\Diagnostics\System.Diagnostics.DiagnosticSource.Typeforwards.netcoreapp.cs" />
<Compile Include="$(CommonPath)System\LocalAppContextSwitches.Common.cs"
Link="Common\System\LocalAppContextSwitches.Common.cs" />
<Compile Include="$(CommonPath)System\Text\ValueStringBuilder.cs"
Link="Common\System\Text\ValueStringBuilder.cs" />
</ItemGroup>

<ItemGroup Condition="'$(TargetFrameworkIdentifier)' != '.NETCoreApp'">
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -34,8 +34,6 @@ public partial class DiagnosticListener : DiagnosticSource, IObservable<KeyValue
/// </summary>
public static IObservable<DiagnosticListener> AllListeners
{
[UnconditionalSuppressMessage("AotAnalysis", "IL3050:RequiresDynamicCode",
Justification = "ENABLE_HTTP_HANDLER is not enabled in the .NET current version")]
get
{
#if ENABLE_HTTP_HANDLER
Expand Down Expand Up @@ -255,7 +253,6 @@ public override bool IsEnabled(string name, object? arg1, object? arg2 = null)
/// Override abstract method
/// </summary>
[RequiresUnreferencedCode(WriteRequiresUnreferencedCode)]
[RequiresDynamicCode(WriteRequiresDynamicCode)]
public override void Write(string name, object? value)
{
for (DiagnosticSubscription? curSubscription = _subscriptions; curSubscription != null; curSubscription = curSubscription.Next)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,6 @@ namespace System.Diagnostics
public abstract partial class DiagnosticSource
{
internal const string WriteRequiresUnreferencedCode = "The type of object being written to DiagnosticSource cannot be discovered statically.";
internal const string WriteRequiresDynamicCode = "DiagnosticSource may require creating new generic types or methods, which requires creating code at runtime. This may not work when AOT compiling.";

/// <summary>
/// Write is a generic way of logging complex payloads. Each notification
Expand All @@ -36,7 +35,6 @@ public abstract partial class DiagnosticSource
/// <param name="value">An object that represent the value being passed as a payload for the event.
/// This is often an anonymous type which contains several sub-values.</param>
[RequiresUnreferencedCode(WriteRequiresUnreferencedCode)]
[RequiresDynamicCode(WriteRequiresDynamicCode)]
public abstract void Write(string name, object? value);

/// <summary>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,6 @@ public abstract partial class DiagnosticSource
/// <returns>Started Activity for convenient chaining</returns>
/// <seealso cref="Activity"/>
[RequiresUnreferencedCode(WriteRequiresUnreferencedCode)]
[RequiresDynamicCode(WriteRequiresDynamicCode)]
public Activity StartActivity(Activity activity, object? args)
{
activity.Start();
Expand All @@ -45,7 +44,6 @@ public Activity StartActivity(Activity activity, object? args)
/// <param name="args">An object that represent the value being passed as a payload for the event.</param>
/// <seealso cref="Activity"/>
[RequiresUnreferencedCode(WriteRequiresUnreferencedCode)]
[RequiresDynamicCode(WriteRequiresDynamicCode)]
public void StopActivity(Activity activity, object? args)
{
// Stop sets the end time if it was unset, but we want it set before we issue the write
Expand Down
Loading