Unable to build: Unable to load Analyzer assembly #77723

a74nh · 2022-11-01T15:38:14Z

Description

With the latest HEAD of runtime I'm unable to build on Arm64 and X64 Linux.

Lots of errors that look like this:
CSC : error CS8034: Unable to load Analyzer assembly /home/alahay01/.nuget/packages/microsoft.codeanalysis.csharp.codestyle/4.4.0-2.22423.18/analyzers/dotnet/cs/Microsoft.CodeAnalysis.CSharp.CodeStyle.dll : Could not load file or assembly 'Microsoft.CodeAnalysis.CSharp.CodeStyle, Version=4.4.6.42318, Culture=neutral, PublicKeyToken=31bf3856ad364e35'. Access is denied. [/home/alahay01/dotnet/runtime_csel_else/src/tasks/AndroidAppBuilder/AndroidAppBuilder.csproj]

Reproduction Steps

git reset --hard 1f0a0960029
rm -fr artifacts
rm -fr .dotnet
./build.sh -rc checked -lc release

Expected behavior

A clean build :)

Actual behavior

Full output: log.txt

Regression?

Used to work. Was working at 17d613e

Known Workarounds

None :(

Configuration

Arm64 Linux Ubuntu 18.04

Other information

No response

The text was updated successfully, but these errors were encountered:

ghost · 2022-11-01T15:38:22Z

Tagging subscribers to this area: @dotnet/area-infrastructure-libraries
See info in area-owners.md if you want to be subscribed.

Issue Details

Description

With the latest HEAD of runtime I'm unable to build on Arm64 Linux.

Lots of errors that look like this:
CSC : error CS8034: Unable to load Analyzer assembly /home/alahay01/.nuget/packages/microsoft.codeanalysis.csharp.codestyle/4.4.0-2.22423.18/analyzers/dotnet/cs/Microsoft.CodeAnalysis.CSharp.CodeStyle.dll : Could not load file or assembly 'Microsoft.CodeAnalysis.CSharp.CodeStyle, Version=4.4.6.42318, Culture=neutral, PublicKeyToken=31bf3856ad364e35'. Access is denied. [/home/alahay01/dotnet/runtime_csel_else/src/tasks/AndroidAppBuilder/AndroidAppBuilder.csproj]

Reproduction Steps

git reset --hard 1f0a0960029
rm -fr artifacts
rm -fr .dotnet
./build.sh -rc checked -lc release

Expected behavior

A clean build :)

Actual behavior

Full output: log.txt

Regression?

Used to work. Was working at 17d613e

Known Workarounds

None :(

Configuration

Arm64 Linux Ubuntu 18.04

Other information

No response

Author:	a74nh
Assignees:	-
Labels:	`area-Infrastructure-libraries`, `untriaged`
Milestone:	-

a74nh · 2022-11-01T15:38:31Z

@kunalspathak

a74nh · 2022-11-01T15:41:44Z

I'm seeing the same on X64 too.

a74nh · 2022-11-02T11:17:42Z

Looks like this is due to:
e459c4e 2022-10-27.. dotnet-maestr.. [main] Update dependencies from 7 repositories (#77213)

@akoeplinger and @stephentoub

kunalspathak · 2022-11-02T13:30:59Z

Thanks for reporting @a74nh . Does it work if you revert the change from #77213?

a74nh · 2022-11-02T14:08:25Z

Thanks for reporting @a74nh . Does it work if you revert the change from #77213?

It does.

Suspect this is the issue.

file  /home/alahay01/.nuget/packages/microsoft.dotnet.codeanalysis/8.0.0-beta.22528.1/analyzers/Microsoft.DotNet.CodeAnalysis.dll
/home/alahay01/.nuget/packages/microsoft.dotnet.codeanalysis/8.0.0-beta.22528.1/analyzers/Microsoft.DotNet.CodeAnalysis.dll: PE32 executable (DLL) (console) Intel 80386 Mono/.Net assembly, for MS Windows

$ file /home/alahay01/.nuget/packages/microsoft.codeanalysis.csharp.codestyle/4.4.0-2.22423.18/analyzers/dotnet/cs/Microsoft.CodeAnalysis.CSharp.CodeStyle.dll
/home/alahay01/.nuget/packages/microsoft.codeanalysis.csharp.codestyle/4.4.0-2.22423.18/analyzers/dotnet/cs/Microsoft.CodeAnalysis.CSharp.CodeStyle.dll: PE32 executable (DLL) (console) Intel 80386 Mono/.Net assembly, for MS Window

I'm on Arm64 Linux, but these are for X86 Windows.

kunalspathak · 2022-11-02T14:36:17Z

I am able to repro this on Linux/arm64 machine, but I don't think it is related to x86 windows.

I see the same printed for preview5

~/git/runtime/.dotnet/sdk$ file 7.0.100-rc.1.22431.12/Sdks/Microsoft.NET.Sdk/analyzers/ILLink.RoslynAnalyzer.dll 
7.0.100-rc.1.22431.12/Sdks/Microsoft.NET.Sdk/analyzers/ILLink.RoslynAnalyzer.dll: PE32 executable (DLL) (console) Intel 80386 Mono/.Net assembly, for MS Windows

~/git/runtime/.dotnet/sdk$ file 7.0.100-preview.7.22377.5/Sdks/Microsoft.NET.Sdk/analyzers/ILLink.RoslynAnalyzer.dll 
7.0.100-preview.7.22377.5/Sdks/Microsoft.NET.Sdk/analyzers/ILLink.RoslynAnalyzer.dll: PE32 executable (DLL) (console) Intel 80386 Mono/.Net assembly, for MS Windows

Even the previous version of the binaries was labeled that way (and I believe it just means that they were compiled on Windows machine) but are x-platform.

~/git/runtime$ file /home/kpathak/.nuget/packages/microsoft.codeanalysis.csharp.codestyle/4.4.0-2.22423.18/analyzers/dotnet/cs/Microsoft.CodeAnalysis.CSharp.CodeStyle.dll
/home/kpathak/.nuget/packages/microsoft.codeanalysis.csharp.codestyle/4.4.0-2.22423.18/analyzers/dotnet/cs/Microsoft.CodeAnalysis.CSharp.CodeStyle.dll: PE32 executable (DLL) (console) Intel 80386 Mono/.Net assembly, for MS Windows

~/git/runtime$ file /home/kpathak/.nuget/packages/microsoft.codeanalysis.csharp.codestyle/4.3.0-1.22206.2/analyzers/dotnet/cs/Microsoft.CodeAnalysis.CodeStyle.dll 
analyzers/dotnet/cs/Microsoft.CodeAnalysis.CodeStyle.dll: PE32 executable (DLL) (console) Intel 80386 Mono/.Net assembly, for MS Windows
kpathak@b99csiampere05:~/.nuget/packages/microsoft.codeanalysis.csharp.codestyle/4.3.0-1.22206.2$

The error is saying "Access is denied." and I am not sure why that would be coming.

CSC : error CS8034: Unable to load Analyzer assembly /home/kpathak/.nuget/packages/microsoft.codeanalysis.csharp.codestyle/4.4.0-2.22423.18/analyzers/dotnet/cs/Microsoft.CodeAnalysis.CSharp.CodeStyle.dll : Could not load file or assembly 'Microsoft.CodeAnalysis.CSharp.CodeStyle, Version=4.4.6.42318, Culture=neutral, PublicKeyToken=31bf3856ad364e35'. Access is denied. [/home/kpathak/git/runtime/src/tasks/AndroidAppBuilder/AndroidAppBuilder.csproj]

hoyosjs · 2022-11-03T20:42:04Z

FYI @stephentoub @ViktorHofer @jkoritzinsky looks like we are seeing breaks over updated versions of Roslyn/analyzers. This is odd... since it's an access denied error code I'd expect permission issues or locks, but this is Linux and the access list looks sane (764). I can't repro on x64 and don't have arm64 hardware...

a74nh · 2022-11-04T07:59:37Z

I can't repro on x64

It's probably not much use, but here's the log of it failing on X86 Linux.
x86log.txt

It's a dell r540. The test is running inside a clean ubuntu 18.04 docker image.

kunalspathak · 2022-11-07T14:58:27Z

@ViktorHofer - did you get a chance to look at this?

ViktorHofer · 2022-11-07T15:10:19Z

@kunalspathak I wasn't investing this issue since it's affecting the repo tasks that are owned by the Mono infrastructure team.

kunalspathak · 2022-11-07T15:20:44Z

@steveisok - any idea?

kunalspathak · 2022-11-07T15:22:11Z

Also +@jkoritzinsky

steveisok · 2022-11-07T16:26:58Z

@kunalspathak I wasn't investing this issue since it's affecting the repo tasks that are owned by the Mono infrastructure team.

This isn't impacting just mono tasks. CrossGen2Tasks is also showing the same problem.

kunalspathak · 2022-11-09T19:31:27Z

Spoke with @jkoritzinsky and we have a suspicion that it could be because of dotnet/roslyn#64831.

~/git/runtime/src/tasks/AndroidAppBuilder$ ~/git/runtime/.dotnet/dotnet build -c Release AndroidAppBuilder.csproj 

CSC : error CS8034: Unable to load Analyzer assembly /home/kpathak/.nuget/packages/microsoft.codeanalysis.csharp.codestyle/4.4.0-2.22423.18/analyzers/dotnet/cs/Microsoft.CodeAnalysis.CSharp.CodeStyle.dll : Could not load file or assembly 'Microsoft.CodeAnalysis.CSharp.CodeStyle, Version=4.4.6.42318, Culture=neutral, PublicKeyToken=31bf3856ad364e35'. Access is denied. [/home/kpathak/git/runtime/src/tasks/AndroidAppBuilder/AndroidAppBuilder.csproj]
CSC : error CS8034: Unable to load Analyzer assembly /home/kpathak/.nuget/packages/microsoft.codeanalysis.csharp.codestyle/4.4.0-2.22423.18/analyzers/dotnet/cs/Microsoft.CodeAnalysis.CodeStyle.dll : Could not load file or assembly 'Microsoft.CodeAnalysis.CodeStyle, Version=4.4.6.42318, Culture=neutral, PublicKeyToken=31bf3856ad364e35'. Access is denied. [/home/kpathak/git/runtime/src/tasks/AndroidAppBuilder/AndroidAppBuilder.csproj]
CSC : error CS8034: Unable to load Analyzer assembly /home/kpathak/.nuget/packages/microsoft.codeanalysis.netanalyzers/7.0.0-preview1.22553.2/analyzers/dotnet/cs/Microsoft.CodeAnalysis.CSharp.NetAnalyzers.dll : Could not load file or assembly 'Microsoft.CodeAnalysis.CSharp.NetAnalyzers, Version=7.0.7.5302, Culture=neutral, PublicKeyToken=31bf3856ad364e35'. Access is denied. [/home/kpathak/git/runtime/src/tasks/AndroidAppBuilder/AndroidAppBuilder.csproj]

But if I pass -p:UseSharedCompilation=false, it works as expected.

@jaredpar - any idea on how to debug this?

jaredpar · 2022-11-09T19:34:15Z

Spoke with @jkoritzinsky and we have a suspicion that it could be because of dotnet/roslyn#64831.

Based on the evidence here that does seem like a good candidate but I'm struggling to see how that could result in an Access Denied error. This didn't change how we load assemblies, just where we load them ....

Let me see if I can repro this.

kunalspathak · 2022-11-09T19:36:41Z

Let me see if I can repro this.

Surprisingly this doesn't hit in our CI, but I can consistently repro in on Arm64 device.

jaredpar · 2022-11-09T19:36:46Z

As a quick option can someone who can repro do the following:

Kill all dotnet processes
export RoslynCommandLineLogFile=/some/path/log.txt
dotnet build -bl -c Release AndroidAppBuilder.csproj

Share out both log.txt and msbuild.binlog

jaredpar · 2022-11-09T19:37:47Z

Surprisingly this doesn't hit in our CI, but I can consistently repro in on Arm64 device.

Do we have any known issues around ALC and ARM64? The functional change in behavior here is that we are creating, using and disposing more ALC in the same process. Before the change, assuming it's the culprit, we were more likely to drop do this in separate processes.

kunalspathak · 2022-11-09T19:42:43Z

Share out both log.txt and msbuild.binlog

Sent you offline.

jaredpar · 2022-11-09T21:44:30Z

Sent you offline.

Thanks. Looking through that I suspect the cause is not a result of dotnet/roslyn#64831. Or at least that didn't introduce any bugs that result in an access denied error. That change most likely made this build occur in the compiler server vs. the command line compiler. I think that exposed an existing bug in this scenario.

The primary difference between using the compiler server and not using it for analyzers / generators is that in the compiler server case we first copy all of the files to a temp directory. The compiler server is a long lived process, we don't want to keep handles to files that could be in the user repository so we copy them first.

You can imagine the process as essentially:

Create a directory under temp
Copy all analyzers to that directory (matching existing sub-directory structure)
Load analyzers from there

The access denied is most likely coming from either the creation of the temp directory or tyring to read the analyzer files. Given we can successfully compile without the server that means we can read the analyzers. So I suspect there is an issue in ARM64 with creating the temp directory. That isn't really something the compilerc can fix though.

@RikkiGibson

kunalspathak · 2022-11-09T23:00:50Z

Given we can successfully compile without the server

You mean -p:UseSharedCompilation=false compiles without the server, yes?

So I suspect there is an issue in ARM64 with creating the temp directory

That is odd because it is a common scenario that would have surfaced if it was problem on Arm64 device. May be the circumstances under which we create the temp directory is problematic. Is there a code base that you can point at which does the creation of temp directory? Is there a simpler repro that I can debug locally to confirm if creation of temp directory indeed is problematic?

The access denied is most likely coming from either the creation of the temp directory or tyring to read the analyzer files.

Is there a way to print the temp directory path on console?

a74nh · 2022-11-10T14:26:26Z

Is there a way to print the temp directory path on console?

I was stracing the build:

strace --string-limit 4096 -f -o out.txt ~/dotnet/runtime/.dotnet/dotnet build -c release AndroidAppBuilder.csproj

The error message is coming from a socket (recvFrom). I'm assuming this is the client getting it from the build server.
What I don't see in the logs is anything from the server. I'm not sure how the server is started up - if the server was being started from the dotnet process then I'd expect to see something in the log.

Full log file: out.txt
Location of the error is annotated with "FAIL IS HERE"

jaredpar · 2022-11-10T17:07:20Z

You mean -p:UseSharedCompilation=false compiles without the server, yes?

Correct

The error message is coming from a socket (recvFrom).

That doesn't line up with the logs that @kunalspathak sent me though. There the client / server shows that the sockets are completing successfully. If the socket fails the build will assume the server panic'd and it will silently fall back to the command line. That would also be visible in the logs.

if the server was being started from the dotnet process then I'd expect to see something in the log.

The server stays alive between builds so it's possible when you did the strace the build hooked into an existning server vs. created one during build. If you want to be sure that the server starts during the build kill all dotnet processes before starting the bulid. You can also just start the server manually and the build will connect to it dotnet exec VBCSCompiler.dll

a74nh · 2022-11-14T12:01:29Z

Got it!

From the server:

3802448 mkdirat(AT_FDCWD, "/tmp/VBCSCompiler/AnalyzerAssemblyLoader/a174c142e8594bd388d4a56c8b6d0921", 0777) = -1 EACCES (Permission denied)

Looking at those files...

❯ ls -l /tmp/VBCSCompiler/
total 4
drwxr-xr-x 3 swagai01 list 4096 Aug 10 13:40 AnalyzerAssemblyLoader

Deleted that, and the build now works for me.

❯ ls -l /tmp/VBCSCompiler/AnalyzerAssemblyLoader
total 4
drwxrwxr-x 14 alahay01 alahay01 4096 Nov 14 11:57 1bf00b62f4b347339d5425a9e5a9c416

Looks like both /tmp/VBCSCompiler and /tmp/VBCSCompiler/AnalyzerAssemblyLoader need write access by everyone.

a74nh · 2022-11-14T12:15:34Z

This issue will only occur when there are multiple usernames running builds on the same machine without using docker and without clearing out /tmp/ between builds.

Suspect this is the roughly the location of the issue:
https://github.com/dotnet/roslyn/blob/158ba3aaff230b1b89284406323fcc79fa2d04e2/src/Compilers/Server/VBCSCompiler/CompilerRequestHandler.cs#L44
public IAnalyzerAssemblyLoader AnalyzerAssemblyLoader { get; } = new ShadowCopyAnalyzerAssemblyLoader(Path.Combine(Path.GetTempPath(), "VBCSCompiler", "AnalyzerAssemblyLoader"));

Something just needs to ensure that any tmp directories that get created are set to be writable by anyone.

a74nh · 2022-11-14T14:14:13Z

Alternatively, if these files are unique per user, the build server could use /run/user/$uid/AnalyzerAssemblyLoader instead of /tmp

jaredpar · 2022-11-14T17:37:40Z

Looks like both /tmp/VBCSCompiler and /tmp/VBCSCompiler/AnalyzerAssemblyLoader need write access by everyone.

That shouldn't be done though as it would allow developers to interfere with each others builds. I think rather we need to have a directory specific to each user.

akoeplinger · 2022-11-22T15:07:03Z

Closing this since it's not a dotnet/runtime issue. If you hit this deleting /tmp/VBCSCompiler/AnalyzerAssemblyLoader works around the issue until a roslyn fix is implemented.

dotnet-issue-labeler bot added the area-Infrastructure-libraries label Nov 1, 2022

ghost added the untriaged New issue has not been triaged by the area owner label Nov 1, 2022

a74nh mentioned this issue Nov 1, 2022

Add simple Else conditions to If Conversion #77728

Merged

ViktorHofer added area-Infrastructure-mono and removed area-Infrastructure-libraries labels Nov 2, 2022

jaredpar mentioned this issue Nov 14, 2022

Analyzer / generator shadow load directory needs to be per user on Unix dotnet/roslyn#65415

Closed

akoeplinger closed this as completed Nov 22, 2022

ghost removed the untriaged New issue has not been triaged by the area owner label Nov 22, 2022

ghost locked as resolved and limited conversation to collaborators Dec 22, 2022

Unable to build: Unable to load Analyzer assembly #77723

Unable to build: Unable to load Analyzer assembly #77723

Comments

a74nh commented Nov 1, 2022 • edited Loading

Description

Reproduction Steps

Expected behavior

Actual behavior

Regression?

Known Workarounds

Configuration

Other information

ghost commented Nov 1, 2022

Description

Reproduction Steps

Expected behavior

Actual behavior

Regression?

Known Workarounds

Configuration

Other information

a74nh commented Nov 1, 2022

a74nh commented Nov 1, 2022

a74nh commented Nov 2, 2022

kunalspathak commented Nov 2, 2022

a74nh commented Nov 2, 2022

kunalspathak commented Nov 2, 2022

hoyosjs commented Nov 3, 2022

a74nh commented Nov 4, 2022

kunalspathak commented Nov 7, 2022

ViktorHofer commented Nov 7, 2022

kunalspathak commented Nov 7, 2022

kunalspathak commented Nov 7, 2022

steveisok commented Nov 7, 2022 • edited Loading

kunalspathak commented Nov 9, 2022

jaredpar commented Nov 9, 2022

kunalspathak commented Nov 9, 2022

jaredpar commented Nov 9, 2022

jaredpar commented Nov 9, 2022

kunalspathak commented Nov 9, 2022

jaredpar commented Nov 9, 2022 • edited Loading

kunalspathak commented Nov 9, 2022

a74nh commented Nov 10, 2022

jaredpar commented Nov 10, 2022

a74nh commented Nov 14, 2022

a74nh commented Nov 14, 2022

a74nh commented Nov 14, 2022

jaredpar commented Nov 14, 2022

akoeplinger commented Nov 22, 2022

a74nh commented Nov 1, 2022 •

edited

Loading

steveisok commented Nov 7, 2022 •

edited

Loading

jaredpar commented Nov 9, 2022 •

edited

Loading