PortableRuntimeIdentifierGraph.json not found during runtime repo build

[mthalman]

The following error occurs when attempting to build the VMR (during the runtime repo build) with a source-built version of the SDK (stage 2 bootstrapping scenario):

/vmr/.dotnet/sdk/8.0.100-rc.1.23415.1/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.FrameworkReferenceResolution.targets(90,5): error MSB4018: The "ProcessFrameworkReferences" task failed unexpectedly.
System.IO.FileNotFoundException: Could not find file '/vmr/src/runtime/artifacts/source-build/self/src/src/libraries/Microsoft.NETCore.Platforms/src/PortableRuntimeIdentifierGraph.json'.
File name: '/vmr/src/runtime/artifacts/source-build/self/src/src/libraries/Microsoft.NETCore.Platforms/src/PortableRuntimeIdentifierGraph.json'
   at Interop.ThrowExceptionForIoErrno(ErrorInfo errorInfo, String path, Boolean isDirError)
   at Microsoft.Win32.SafeHandles.SafeFileHandle.Open(String path, OpenFlags flags, Int32 mode, Boolean failForSymlink, Boolean& wasSymlink, Func`4 createOpenException)
   at Microsoft.Win32.SafeHandles.SafeFileHandle.Open(String fullPath, FileMode mode, FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize, UnixFileMode openPermissions, Int64& fileLength, UnixFileMode& filePermissions, Boolean failForSymlink, Boolean& wasSymlink, Func`4 createOpenException)
   at System.IO.Strategies.OSFileStreamStrategy..ctor(String path, FileMode mode, FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize, Nullable`1 unixCreateMode)
   at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options, Int64 preallocationSize)
   at System.IO.File.OpenRead(String path)
   at NuGet.RuntimeModel.JsonRuntimeFormat.ReadRuntimeGraph(String filePath)
   at Microsoft.NET.Build.Tasks.RuntimeGraphCache.GetRuntimeGraph(String runtimeJsonPath)
   at Microsoft.NET.Build.Tasks.ProcessFrameworkReferences.ProcessRuntimeIdentifier(String runtimeIdentifier, KnownRuntimePack selectedRuntimePack, String runtimePackVersion, List`1 additionalFrameworkReferencesForRuntimePack, HashSet`1 unrecognizedRuntimeIdentifiers, List`1 unavailableRuntimePacks, List`1 runtimePacks, List`1 packagesToDownload, String isTrimmable, Boolean addRuntimePackAndDownloadIfNecessary, Boolean wasReferencedDirectly)
   at Microsoft.NET.Build.Tasks.ProcessFrameworkReferences.AddPacksForFrameworkReferences(List`1 packagesToDownload, List`1 runtimeFrameworks, List`1 targetingPacks, List`1 runtimePacks, List`1 unavailableRuntimePacks, List`1& knownRuntimePacksForTargetFramework)
   at Microsoft.NET.Build.Tasks.ProcessFrameworkReferences.ExecuteCore()
   at Microsoft.NET.Build.Tasks.TaskBase.Execute()
   at Microsoft.Build.BackEnd.TaskExecutionHost.Microsoft.Build.BackEnd.ITaskExecutionHost.Execute()
   at Microsoft.Build.BackEnd.TaskBuilder.ExecuteInstantiatedTask(ITaskExecutionHost taskExecutionHost, TaskLoggingContext taskLoggingContext, TaskHost taskHost, ItemBucket bucket, TaskExecutionMode howToExecuteTask) [/vmr/src/runtime/artifacts/source-build/self/src/Build.proj]

This occurred after the merge of dotnet/installer#17118.

Binlog: runtime.zip

@elinor-fung - Is this related to dotnet/installer#17185?

[dotnet-issue-labeler]

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

[elinor-fung]

This is not really related to dotnet/installer#17185.

With dotnet/sdk#34279, the PortableRuntimeIdentifierGraph.json is assumed to be next to BundledRuntimeIdentifierGraphFile. From the binlog:

BundledRuntimeIdentifierGraphFile = /vmr/src/runtime/artifacts/source-build/self/src/src/libraries/Microsoft.NETCore.Platforms/src/runtime.json

I believe that comes from:
https://github.com/dotnet/runtime/blob/1eb702ce1efe3581488a37704cee47603a291069/eng/liveBuilds.targets#L264-L268
to use the full RID graph that gets produced by the live build.

@ViktorHofer @dsplaisted thoughts on what the right thing to do here is? Should runtime stop setting that property? Or should sdk allow overriding portable vs non-portable RID graph separately?

[dsplaisted]

Since the full RID graph isn't supposed to be updated any more, it's probably OK for the runtime to stop setting the BundledRuntimeIdentifierGraphFile property.

You could probably also copy the portable RID graph from the SDK next to the full RID graph, or we could update the SDK to allow overriding the one path without the other. But the first thing I'd try would be to stop setting the property.

[elinor-fung]

@mthalman how do we go about testing the stage 2 bootstrapping scenario? I'm not sure how to repro the failure locally to check just doing:

Since the full RID graph isn't supposed to be updated any more, it's probably OK for the runtime to stop setting the BundledRuntimeIdentifierGraphFile property.

by removing https://github.com/dotnet/runtime/blob/1eb702ce1efe3581488a37704cee47603a291069/eng/liveBuilds.targets#L264-L268

[mthalman]

@mthalman how do we go about testing the stage 2 bootstrapping scenario? I'm not sure how to repro the failure locally to check just doing:

The steps are a bit involved to do manually. I kicked off an internal CI build to test this scenario with the removal of that property. I'll report back on the results. Takes a few hours to run.

[ViktorHofer]

Since the full RID graph isn't supposed to be updated any more, it's probably OK for the runtime to stop setting the BundledRuntimeIdentifierGraphFile property.

+1

[mthalman]

@elinor-fung - I've verified the stage 2 build passes with this removal. Go ahead with that change. We'll need it in all the relevant branches: rc1, 8.0, main.

[ViktorHofer]

Rc1 automatically flows into release/8.0 so we just need two PRs. We usually start in main and then just backport via the backport command.

[ViktorHofer]

Since the full RID graph isn't supposed to be updated any more, it's probably OK for the runtime to stop setting the BundledRuntimeIdentifierGraphFile property.

cc @tmds

[tmds]

Since the full RID graph isn't supposed to be updated any more, it's probably OK for the runtime to stop setting the BundledRuntimeIdentifierGraphFile property.

During the runtime build BundledRuntimeIdentifierGraphFile gets set to make the SDK that is used know about the rid being built.

I'd be surprised if we can just remove it, as otherwise we wouldn't have need to set it in the first place.

I've verified the stage 2 build passes with this removal. Go ahead with that change. We'll need it in all the relevant branches: rc1, 8.0, main.

This seems to suggest it is not necessary. 🤞

To verify you need to have all the recent changes both in the bootstrap SDK and the sources being built.

[mthalman]

@tmds - I have verified that the removal of this property resolves the issue we were having in source-build. It allows us to successfully run through the SDK bootstrapping scenario, and the smoke tests are passing as well.

[tmds]

@tmds - I have verified that the removal of this property resolves the issue we were having in source-build. It allows us to successfully run through the SDK bootstrapping scenario, and the smoke tests are passing as well.

I wonder how code like this can work: https://github.com/dotnet/sdk/blob/05dbd2efce67e49ca6f32224e2761c8766d8cd1e/src/Tasks/Microsoft.NET.Build.Tasks/ResolveReadyToRunCompilers.cs#L183-L189.

but anyway, if it works ... it works.

[tmds]

I wonder how code like this can work: https://github.com/dotnet/sdk/blob/05dbd2efce67e49ca6f32224e2761c8766d8cd1e/src/Tasks/Microsoft.NET.Build.Tasks/ResolveReadyToRunCompilers.cs#L183-L189.

I'm going to take a guess.

This task uses BundledRuntimeIdentifierGraphFile: https://github.com/dotnet/runtime/blob/64243bbf5e9ee53c0c4c5678f2cd8c7f1c9b4f6f/src/tasks/Crossgen2Tasks/Microsoft.NET.CrossGen.targets#L446

This is the legacy, non-trimmed runtime graph.

@mthalman did you verify it on an OS that is in this graph? Then the runtime build would not need a patched version of it.

We may be back in a situation where .NET won't source-build unless the distro is already added to this graph...

If you update the ID in /etc/os-release to something unknown, you can verify if this scenario still works.

[mthalman]

@tmds - I changed the ID to banana and ran through the bootstrapping scenario. Everything still works.

[tmds]

I'm surprised it works, but happy we didn't break it.

Thanks for checking!