.NET 9 Preview 3 occassionally fails to build in arm64 with `Object reference not set to an instance of an object` errors #7349

omajid · 2024-05-14T18:41:33Z

I am trying to build .NET 9 Preview 3 on arm64 in an offline environment.

It works on some platforms, but sometimes fails on others.

Full log: https://download.copr.fedorainfracloud.org/results/@dotnet-sig/dotnet-preview/centos-stream-9-aarch64/07444252-dotnet9.0/builder-live.log.gz

Describe the Bug

                         "/builddir/build/BUILD/dotnet-v9.0.0-preview.3.24172.9-x64-bootstrap/.packages/ArcadeBootstrapPackage/microsoft.dotnet.arcade.sdk/9.0.0-beta.24162.2/tools/Build.proj" (default target) (1) ->
                         (TriggerBuildDependencyPackageProjects target) -> 
                           CSC : error AD0001: Analyzer 'Microsoft.NetCore.CSharp.Analyzers.Runtime.CSharpDetectPreviewFeatureAnalyzer' threw an exception of type 'System.NullReferenceException' with message 'Object reference not set to an instance of an object.'. [/builddir/build/BUILD/dotnet-v9.0.0-preview.3.24172.9-x64-bootstrap/src/source-build-reference-packages/src/referencePackages/src/system.composition.hosting/8.0.0/System.Composition.Hosting.8.0.0.csproj::TargetFramework=net6.0] [/builddir/build/BUILD/dotnet-v9.0.0-preview.3.24172.9-x64-bootstrap/.packages/ArcadeBootstrapPackage/microsoft.dotnet.arcade.sdk/9.0.0-beta.24162.2/tools/Build.proj]
                           CSC : error AD0001: Analyzer 'Microsoft.NetCore.CSharp.Analyzers.Runtime.CSharpDetectPreviewFeatureAnalyzer' threw an exception of type 'System.NullReferenceException' with message 'Object reference not set to an instance of an object.'. [/builddir/build/BUILD/dotnet-v9.0.0-preview.3.24172.9-x64-bootstrap/src/source-build-reference-packages/src/referencePackages/src/system.composition.hosting/8.0.0/System.Composition.Hosting.8.0.0.csproj::TargetFramework=net6.0] [/builddir/build/BUILD/dotnet-v9.0.0-preview.3.24172.9-x64-bootstrap/.packages/ArcadeBootstrapPackage/microsoft.dotnet.arcade.sdk/9.0.0-beta.24162.2/tools/Build.proj]
                           /builddir/build/BUILD/dotnet-v9.0.0-preview.3.24172.9-x64-bootstrap/src/source-build-reference-packages/eng/DotNetBuild.props(28,5): error MSB3073: The command "./build.sh --configuration Release /bl:/builddir/build/BUILD/dotnet-v9.0.0-preview.3.24172.9-x64-bootstrap/src/source-build-reference-packages/artifacts/sourcebuild-dependency-projects.binlog /p:LocalNuGetPackageCacheDirectory=/builddir/build/BUILD/dotnet-v9.0.0-preview.3.24172.9-x64-bootstrap/artifacts/obj/source-build-reference-package-cache /p:SourceBuildOutputDir=/builddir/build/BUILD/dotnet-v9.0.0-preview.3.24172.9-x64-bootstrap/src/source-build-reference-packages/artifacts/sb/ /p:BuildDependencyPackageProjects=true /p:SetUpSourceBuildIntermediateNupkgCache=true /p:DotNetBuildSourceOnly=true /p:DotNetBuildInnerRepo=true /p:MicrosoftNetCoreIlasmPackageRuntimeId=linux-arm64  /p:RestoreConfigFile=/builddir/build/BUILD/dotnet-v9.0.0-preview.3.24172.9-x64-bootstrap/artifacts/obj/source-build-reference-packages/NuGet.config" exited with code 1. [/builddir/build/BUILD/dotnet-v9.0.0-preview.3.24172.9-x64-bootstrap/.packages/ArcadeBootstrapPackage/microsoft.dotnet.arcade.sdk/9.0.0-beta.24162.2/tools/Build.proj]

Steps to Reproduce

Other Information

The text was updated successfully, but these errors were encountered:

dotnet-issue-labeler · 2024-05-14T18:41:38Z

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

ViktorHofer · 2024-05-15T13:05:54Z

I think this issue should go into roslyn-analyzers which is the home of the CSharpDetectPreviewFeatureAnalyzer. cc @buyaa-n @jeffhandley

buyaa-n · 2024-05-15T17:40:51Z

In case you are blocked by this, maybe disable CA2252 for the repo

ViktorHofer · 2024-05-15T19:44:39Z

Just happened in a VMR official build but with a different exception this time:

CSC : error AD0001: Analyzer 'Microsoft.NetCore.CSharp.Analyzers.Runtime.CSharpDetectPreviewFeatureAnalyzer' threw an exception of type 'System.MissingFieldException' with message 'Field not found: 'Microsoft.CodeAnalysis.CSharp.BoundTypeOrValueExpression.k__BackingField'.'. [/Users/runner/work/1/s/eng/tools/tasks/Microsoft.DotNet.UnifiedBuild.Tasks/Microsoft.DotNet.UnifiedBuild.Tasks.csproj]

https://dev.azure.com/dnceng/internal/_build/results?buildId=2451952&view=logs&j=772df19d-2788-50a3-ff88-b0cb2cc9f1cf&t=1dd247f5-2cbf-5a62-8b1c-2bc204313a12

This is on OSX x64

ViktorHofer · 2024-05-21T09:46:49Z

Interestingly, this seems to affect more than just this one analyzer, see dotnet/sdk#41030

cc @jaredpar

jaredpar · 2024-05-22T01:49:33Z

That error looks like a weird tearing problem.

That failing leg looks like it's generating a binary log but I can't find it in the published artifacts.

sharwell · 2024-05-22T13:24:41Z

Here's the full error:

        CSC error AD0001: Analyzer 'Microsoft.NetCore.CSharp.Analyzers.Runtime.CSharpDetectPreviewFeatureAnalyzer' threw an exception of type 'System.MissingFieldException' with message 'Field not found: 'Microsoft.CodeAnalysis.CSharp.BoundTypeOrValueExpression.<Data>k__BackingField'.'. [/Users/runner/work/1/s/eng/tools/tasks/Microsoft.DotNet.UnifiedBuild.Tasks/Microsoft.DotNet.UnifiedBuild.Tasks.csproj]
    Exception occurred with following context:
    Compilation: Microsoft.DotNet.UnifiedBuild.Tasks
    System.MissingFieldException: Field not found: 'Microsoft.CodeAnalysis.CSharp.BoundTypeOrValueExpression.<Data>k__BackingField'.
    at Microsoft.CodeAnalysis.CSharp.Binder.CheckValue(BoundExpression expr, BindValueKind valueKind, BindingDiagnosticBag diagnostics, Boolean dynamificationOfAssignmentResultIsHandled)
    at Microsoft.CodeAnalysis.CSharp.Binder.BindArgumentExpression(BindingDiagnosticBag diagnostics, ExpressionSyntax argumentExpression, RefKind refKind, Boolean allowArglist)
    at Microsoft.CodeAnalysis.CSharp.Binder.BindAttributeArguments(AttributeArgumentListSyntax attributeArgumentList, NamedTypeSymbol attributeType, BindingDiagnosticBag diagnostics)
    at Microsoft.CodeAnalysis.CSharp.Binder.BindAttributeCore(AttributeSyntax node, NamedTypeSymbol attributeType, Symbol attributedMember, BindingDiagnosticBag diagnostics)
    at Microsoft.CodeAnalysis.CSharp.Binder.GetAttribute(AttributeSyntax node, NamedTypeSymbol boundAttributeType, Action`1 beforeAttributePartBound, Action`1 afterAttributePartBound, BindingDiagnosticBag diagnostics)
    at Microsoft.CodeAnalysis.CSharp.Binder.GetAttributes(ImmutableArray`1 binders, ImmutableArray`1 attributesToBind, ImmutableArray`1 boundAttributeTypes, CSharpAttributeData[] attributeDataArray, BoundAttribute[] boundAttributeArray, Action`1 beforeAttributePartBound, Action`1 afterAttributePartBound, BindingDiagnosticBag diagnostics)
    at Microsoft.CodeAnalysis.CSharp.Symbol.LoadAndValidateAttributes(OneOrMany`1 attributesSyntaxLists, CustomAttributesBag`1& lazyCustomAttributesBag, AttributeLocation symbolPart, Boolean earlyDecodingOnly, Binder binderOpt, Func`2 attributeMatchesOpt, Action`1 beforeAttributePartBound, Action`1 afterAttributePartBound)
    at Microsoft.CodeAnalysis.CSharp.Symbols.SourceAssemblySymbol.EnsureAttributesAreBound()
    at Microsoft.CodeAnalysis.CSharp.Symbols.SourceAssemblySymbol.GetAttributes()
    at Microsoft.CodeAnalysis.CSharp.Symbols.PublicModel.Symbol.Microsoft.CodeAnalysis.ISymbol.GetAttributes()
    at Analyzer.Utilities.Extensions.ISymbolExtensions.HasAnyAttribute(ISymbol symbol, INamedTypeSymbol attribute)
    at Microsoft.NetCore.Analyzers.Runtime.DetectPreviewFeatureAnalyzer.<Initialize>b__33_0(CompilationStartAnalysisContext context)
    at Microsoft.CodeAnalysis.Diagnostics.AnalyzerExecutor.ExecuteAndCatchIfThrows_NoLock[TArg](DiagnosticAnalyzer analyzer, Action`1 analyze, TArg argument, Nullable`1 info, CancellationToken cancellationToken)
    -----
    Suppress the following diagnostics to disable this analyzer: CA2252

sharwell · 2024-05-22T13:26:52Z

@jaredpar The log is under OSX_Mono_x64_BuildLogs_Attempt1

ViktorHofer · 2024-05-22T13:30:47Z

Correct. Just to be more precise, the binlog is here:

jaredpar · 2024-05-22T14:06:56Z

@333fred PTAL

jaredpar · 2024-05-23T18:56:08Z

@jjonescz PTAL

jjonescz · 2024-05-24T12:16:19Z

I'm unable to reproduce this, but that's not a surprise given this is a flaky issue.

I think I would need at least a memory dump or something like that to be able to investigate this.

Also it seems there might be two separate issues:

Field not found: 'Microsoft.CodeAnalysis.CSharp.BoundTypeOrValueExpression.k__BackingField' - happens inside Roslyn which is called from within an analyzer (stack trace). Looks like some assembly loading issue which makes runtime fail to resolve the field.
NullReferenceException - in build logs from the linked issue I can see the stack trace is:
```
System.NullReferenceException: Object reference not set to an instance of an object.
at Analyzer.Utilities.BoundedCacheWithFactory`2.GetOrCreateValue(TKey key, Func`2 valueFactory)
at Microsoft.CodeAnalysis.PublicApiAnalyzers.DeclarePublicApiAnalyzer.Impl.OnSymbolAction(SymbolAnalysisContext symbolContext)
at Microsoft.CodeAnalysis.Diagnostics.AnalyzerExecutor.ExecuteAndCatchIfThrows_NoLock[TArg](DiagnosticAnalyzer analyzer, Action`1 analyze, TArg argument, Nullable`1 info, CancellationToken cancellationToken)
```
This might be some actual bug in the implementation (although that file wasn't changed in several years). https://github.com/dotnet/roslyn-analyzers/blob/main/src/Utilities/Compiler/BoundedCacheWithFactory.cs
Another possibility: a null key is being provided and the NRE is triggered inside the valueFactory. The key for caches here seems to be the Compilation, so SymbolAnalysisContext.Compilation would have to be null, I'm not sure if that can happen.

MichaelSimons · 2024-06-06T15:47:48Z

We had an occurrence of this in source-build CI - dotnet/source-build#4451

ViktorHofer · 2024-06-28T14:01:45Z

Another hit: https://dev.azure.com/dnceng/internal/_build/results?buildId=2484343&view=logs&j=27bc86d6-5b04-5e7e-1952-551ad1371b06&t=7de518f5-71d4-5df0-99bc-efa6e473f337

NullReferenceException thrown by DoNotHideBaseClassMethodsAnalyzer. Given that this happens with all different kinds of analyzers, might be a bug in the abstraction layer / the compiler API.

ViktorHofer · 2024-07-08T20:20:34Z

Another hit in https://dev.azure.com/dnceng-public/public/_build/results?buildId=733454&view=logs&jobId=f0f4e1eb-2872-579b-17f7-75dc26fbb285&j=f0f4e1eb-2872-579b-17f7-75dc26fbb285&t=e5ca527a-fe4f-597c-4330-d9b726220892

CSC : error AD0001: Analyzer 'Microsoft.NetCore.CSharp.Analyzers.Runtime.CSharpDetectPreviewFeatureAnalyzer' threw an exception of type 'System.NullReferenceException' with message 'Object reference not set to an instance of an object.'. [/vmr/src/efcore/src/EFCore/EFCore.csproj]

Affects non-source-build (msft build) as well.

jaredpar · 2024-07-09T05:14:58Z

@arkalyanms

ViktorHofer · 2024-07-09T12:41:20Z

Also see dotnet/dnceng#3305 which tracks all the hits. According to Alexander/Kusto, this happened 133 times in the last 60 days.

buyaa-n · 2024-07-10T16:21:19Z

This looks more like roslyn issue to me, as several different analyzers affected, I think we should move this to roslyn

jaredpar · 2024-07-10T19:06:34Z

@buyaa-n please do not transfer this issue. It is mostly likely a bug in roslyn-analyzers.

jjonescz · 2024-07-11T17:15:46Z

I wonder if we can collect a crash dump in some of the pipelines where this is happening. That could allow us to determine the cause.

jaredpar · 2024-07-19T20:07:30Z

This is the stack of the NullReferenceException in at least one case:

System.NullReferenceException: Object reference not set to an instance of an object.
at System.Collections.Concurrent.ConcurrentDictionary`2.TryRemoveInternal(TKey key, TValue& value, Boolean matchValue, TValue oldValue)
at Microsoft.CodeQuality.Analyzers.Maintainability.AvoidUnusedPrivateFieldsAnalyzer.<>c__DisplayClass5_0.<Initialize>b__2(OperationAnalysisContext operationContext)
at Microsoft.CodeAnalysis.Diagnostics.AnalyzerExecutor.ExecuteAndCatchIfThrows_NoLock[TArg](DiagnosticAnalyzer analyzer, Action`1 analyze, TArg argument, Nullable`1 info, CancellationToken cancellationToken)

That almost certainly represents this line in code:

IFieldSymbol field = ((IFieldReferenceOperation)operationContext.Operation).Field;
if (field.DeclaredAccessibility == Accessibility.Private)
{
    referencedPrivateFields.TryAdd(field, default);
    // Error is here. 
    maybeUnreferencedPrivateFields.TryRemove(field, out _);
}

Both values here are non-null:

maybeUnreferencedPrivateFields: is single assign and initialized to non-null at declaration
field: is used above this line several times without null reffing.

That seems like a runtime bug.

ellahathaway · 2024-07-22T17:24:45Z

Just encountered this issue while source-building the VMR at dotnet/dotnet@c58501e

/vmr/repo-projects/Directory.Build.targets(428,5): error MSB3073: The command "/vmr/src/symreader/eng/common/build.sh --restore --build --pack --publish --ci --configuration Release -bl /p:DotNetBuildRepo=true /p:DotNetBuildOrchestrator=true /p:RestoreConfigFile=/vmr/artifacts/obj/symreader/NuGet.config /p:SourceBuildUseMonoRuntime=true /p:SourceBuiltAssetsDir=/vmr/artifacts/assets/Release/ /p:SourceBuiltAssetManifestsDir=/vmr/artifacts/obj/AssetManifests/symreader/ /p:SourceBuiltSymbolsDir=/vmr/artifacts/obj/Symbols/symreader/ /p:ArcadeBuildFromSource=true /p:DotNetBuildSourceOnly=true /p:PreviouslySourceBuiltNupkgCacheDir="/vmr/prereqs/packages/previously-source-built/" /p:ReferencePackageNupkgCacheDir="/vmr/prereqs/packages/reference/" /p:SourceBuiltShippingPackagesDir=/vmr/artifacts/packages/Release/Shipping/symreader/ /p:SourceBuiltNonShippingPackagesDir=/vmr/artifacts/packages/Release/NonShipping/symreader/ /v:minimal > /vmr/artifacts/log/Release/symreader.log 2>&1" exited with code 1. [/vmr/repo-projects/symreader.proj]
  
    
      Determining projects to restore...
      Restored /vmr/artifacts/source-built-sdks/Microsoft.DotNet.Arcade.Sdk/tools/Tools.proj (in 171 ms).
      Determining projects to restore...
      Restored /vmr/src/symreader/src/Microsoft.DiaSymReader/Microsoft.DiaSymReader.csproj (in 81 ms).
      Microsoft.DiaSymReader -> /vmr/src/symreader/artifacts/bin/Microsoft.DiaSymReader/Release/netstandard2.0/Microsoft.DiaSymReader.dll
    CSC : error AD0001: Analyzer 'Microsoft.NetCore.CSharp.Analyzers.Runtime.CSharpDetectPreviewFeatureAnalyzer' threw an exception of type 'System.NullReferenceException' with message 'Object reference not set to an instance of an object.'. [/vmr/src/symreader/src/Microsoft.DiaSymReader/Microsoft.DiaSymReader.csproj::TargetFramework=net9.0]
##[error]CSC(0,0): error AD0001: (NETCORE_ENGINEERING_TELEMETRY=Build) Analyzer 'Microsoft.NetCore.CSharp.Analyzers.Runtime.CSharpDetectPreviewFeatureAnalyzer' threw an exception of type 'System.NullReferenceException' with message 'Object reference not set to an instance of an object.'.

Link to build (internal Microsoft link)

jaredpar · 2024-07-26T16:23:11Z

Resolving as a dupe of runtime issue 104123. The fix is in PR and will be back ported to 9.0 P7.

dotnet-issue-labeler bot added the untriaged label May 14, 2024

buyaa-n transferred this issue from dotnet/source-build May 15, 2024

buyaa-n added help wanted The issue is up-for-grabs, and can be claimed by commenting and removed untriaged labels May 15, 2024

ViktorHofer mentioned this issue May 21, 2024

VMR Unified Linux Build Failing dotnet/sdk#41030

Closed

MichaelSimons mentioned this issue Jun 6, 2024

Analyzers throwing all kinds of exceptions during build in VMR dotnet/source-build#4451

Closed

buyaa-n removed the help wanted The issue is up-for-grabs, and can be claimed by commenting label Jul 10, 2024

buyaa-n transferred this issue from dotnet/roslyn-analyzers Jul 10, 2024

jaredpar transferred this issue from dotnet/roslyn Jul 10, 2024

MichaelSimons moved this from Backlog to Blocked in .NET Source Build Jul 24, 2024

jaredpar closed this as completed Jul 26, 2024

github-project-automation bot moved this from Blocked to Done in .NET Source Build Jul 26, 2024

hoyosjs mentioned this issue Jul 26, 2024

[release/9.0-preview7] JIT: Fix placement of GT_START_NOGC for tailcalls in face of bulk copy with write barrier calls dotnet/runtime#105572

Merged

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

.NET 9 Preview 3 occassionally fails to build in arm64 with `Object reference not set to an instance of an object` errors #7349

.NET 9 Preview 3 occassionally fails to build in arm64 with `Object reference not set to an instance of an object` errors #7349

omajid commented May 14, 2024

dotnet-issue-labeler bot commented May 14, 2024

ViktorHofer commented May 15, 2024

buyaa-n commented May 15, 2024

ViktorHofer commented May 15, 2024 •

edited

Loading

ViktorHofer commented May 21, 2024

jaredpar commented May 22, 2024

sharwell commented May 22, 2024

sharwell commented May 22, 2024

ViktorHofer commented May 22, 2024

jaredpar commented May 22, 2024

jaredpar commented May 23, 2024

jjonescz commented May 24, 2024 •

edited

Loading

MichaelSimons commented Jun 6, 2024

ViktorHofer commented Jun 28, 2024

ViktorHofer commented Jul 8, 2024

jaredpar commented Jul 9, 2024

ViktorHofer commented Jul 9, 2024 •

edited

Loading

buyaa-n commented Jul 10, 2024

jaredpar commented Jul 10, 2024 •

edited

Loading

jjonescz commented Jul 11, 2024

jaredpar commented Jul 19, 2024

ellahathaway commented Jul 22, 2024

jaredpar commented Jul 26, 2024

.NET 9 Preview 3 occassionally fails to build in arm64 with Object reference not set to an instance of an object errors #7349

.NET 9 Preview 3 occassionally fails to build in arm64 with Object reference not set to an instance of an object errors #7349

Comments

omajid commented May 14, 2024

Describe the Bug

Steps to Reproduce

Other Information

dotnet-issue-labeler bot commented May 14, 2024

ViktorHofer commented May 15, 2024

buyaa-n commented May 15, 2024

ViktorHofer commented May 15, 2024 • edited Loading

ViktorHofer commented May 21, 2024

jaredpar commented May 22, 2024

sharwell commented May 22, 2024

sharwell commented May 22, 2024

ViktorHofer commented May 22, 2024

jaredpar commented May 22, 2024

jaredpar commented May 23, 2024

jjonescz commented May 24, 2024 • edited Loading

MichaelSimons commented Jun 6, 2024

ViktorHofer commented Jun 28, 2024

ViktorHofer commented Jul 8, 2024

jaredpar commented Jul 9, 2024

ViktorHofer commented Jul 9, 2024 • edited Loading

buyaa-n commented Jul 10, 2024

jaredpar commented Jul 10, 2024 • edited Loading

jjonescz commented Jul 11, 2024

jaredpar commented Jul 19, 2024

ellahathaway commented Jul 22, 2024

jaredpar commented Jul 26, 2024

.NET 9 Preview 3 occassionally fails to build in arm64 with `Object reference not set to an instance of an object` errors #7349

.NET 9 Preview 3 occassionally fails to build in arm64 with `Object reference not set to an instance of an object` errors #7349

ViktorHofer commented May 15, 2024 •

edited

Loading

jjonescz commented May 24, 2024 •

edited

Loading

ViktorHofer commented Jul 9, 2024 •

edited

Loading

jaredpar commented Jul 10, 2024 •

edited

Loading