Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Possible parallel workload resolver use #7946

Open
Forgind opened this issue May 7, 2024 · 4 comments
Open

Possible parallel workload resolver use #7946

Forgind opened this issue May 7, 2024 · 4 comments

Comments

@Forgind
Copy link
Member

Forgind commented May 7, 2024

Product

dotnet CLI (dotnet new)

Describe The Bug

It seems like there may be some parallel calls to the same workload resolver after it's initialized but before any (serialized) access calls (version or manifests), and it means that we follow the lazy process for computing our manifests multiple times, leading to errors. This should not happen in the first place because the workload resolver should only ever be used from one thread.

One example of this was dotnet/installer#19222

A partial solution is dotnet/sdk#40485, but that doesn't get at the root cause of this instance. We currently believe that root cause is somewhere here.

To Reproduce

See dotnet/installer#19222

dotnet Info

output

Visual Studio Version

No response

Additional context

No response

@dsplaisted
Copy link
Member

dsplaisted commented May 13, 2024

This started showing up a lot more in validation for .NET SDK 8.0.300. See dotnet/aspire#4145

I was able to debug it, and it looks like the issue is here:

Parallel.For(0, allTemplatePackages.Count, async (int index) =>
{
try
{
var scanResult = await _installScanner.ScanAsync(allTemplatePackages[index].MountPointUri, cancellationToken).ConfigureAwait(false);
scanResults[index] = scanResult;
}
catch (Exception ex)
{
_logger.LogWarning(LocalizableStrings.TemplatePackageManager_Error_FailedToScan, allTemplatePackages[index].MountPointUri, ex.Message);
_logger.LogDebug($"Stack trace: {ex.StackTrace}");
}
});

That code is scanning each template package in a separate parallel task. That leads to each template being instantiated in parallel, and when there are workload installation constraints the constraints get created in parallel, thus getting the list of installed workloads in parallel, which results to parallel calls to the workload resolver, which is not thread-safe.

The workload resolver checks to see whether the manifests have been initialized, and if not, then it gets the manifests, adds them to the internal structures that hold them, and then sets the initialized flag to true. Here's that code: https://github.com/dotnet/sdk/blob/4d3cca6fa5437395a0bc984a809c1d5c12c18a26/src/Resolvers/Microsoft.NET.Sdk.WorkloadManifestReader/WorkloadResolver.cs#L87-L95

When multiple threads run this code at once, it's possible for both of them to see that the manifests haven't been initialized, and then for both threads to try to load the manifests, resulting in an error when one thread tries to add a manifest that has already been added by the other thread.

The workload resolver was never intended to be thread-safe, and writing thread-safe code can be difficult. So probably we need to find a way for the template engine to not call the workload resolver code in parallel.

@dsplaisted
Copy link
Member

In case they help, here are some stack traces of the code that ends up accessing the workload resolver in parallel:

Microsoft.DotNet.TemplateLocator.dll!Microsoft.NET.Sdk.WorkloadManifestReader.WorkloadResolver.LoadManifestsFromProvider(Microsoft.NET.Sdk.WorkloadManifestReader.IWorkloadManifestProvider manifestProvider) Line 196	C#
Microsoft.DotNet.TemplateLocator.dll!Microsoft.NET.Sdk.WorkloadManifestReader.WorkloadResolver.InitializeManifests() Line 164	C#
Microsoft.DotNet.TemplateLocator.dll!Microsoft.NET.Sdk.WorkloadManifestReader.WorkloadResolver.GetAvailableWorkloadDefinitions() Line 568	C#
System.Linq.dll!System.Linq.Enumerable.SelectEnumerableIterator<(Microsoft.NET.Sdk.WorkloadManifestReader.WorkloadDefinition, Microsoft.NET.Sdk.WorkloadManifestReader.WorkloadManifest), string>.MoveNext()	Unknown
System.Linq.dll!System.Linq.Enumerable.Contains<string>(System.Collections.Generic.IEnumerable<string> source, string value, System.Collections.Generic.IEqualityComparer<string> comparer)	Unknown
dotnet.dll!Microsoft.DotNet.Workloads.Workload.VisualStudioWorkloads.GetInstalledWorkloads(Microsoft.NET.Sdk.WorkloadManifestReader.IWorkloadResolver workloadResolver, Microsoft.DotNet.Workloads.Workload.List.InstalledWorkloadsCollection installedWorkloads, Microsoft.NET.Sdk.WorkloadManifestReader.SdkFeatureBand? sdkFeatureBand) Line 59	C#
dotnet.dll!Microsoft.DotNet.Workloads.Workload.List.WorkloadInfoHelper.AddInstalledVsWorkloads(System.Collections.Generic.IEnumerable<Microsoft.NET.Sdk.WorkloadManifestReader.WorkloadId> sdkWorkloadIds) Line 66	C#
dotnet.dll!Microsoft.DotNet.Workloads.Workload.List.WorkloadInfoHelper.InstalledAndExtendedWorkloads.get() Line 38	C#
dotnet.dll!Microsoft.DotNet.Tools.New.WorkloadsInfoProvider.GetInstalledWorkloadsAsync(System.Threading.CancellationToken cancellationToken) Line 27	C#
Microsoft.TemplateEngine.Edge.dll!Microsoft.TemplateEngine.Edge.Constraints.WorkloadConstraintFactory.WorkloadConstraint.ExtractWorkloadInfoAsync(System.Collections.Generic.IEnumerable<Microsoft.TemplateEngine.Abstractions.Components.IWorkloadsInfoProvider> workloadsInfoProviders, Microsoft.Extensions.Logging.ILogger logger, System.Threading.CancellationToken token) Line 105	C#
Microsoft.TemplateEngine.Edge.dll!Microsoft.TemplateEngine.Edge.Constraints.WorkloadConstraintFactory.WorkloadConstraint.CreateAsync(Microsoft.TemplateEngine.Abstractions.IEngineEnvironmentSettings environmentSettings, Microsoft.TemplateEngine.Abstractions.Constraints.ITemplateConstraintFactory factory, System.Threading.CancellationToken cancellationToken) Line 52	C#
Microsoft.TemplateEngine.Edge.dll!Microsoft.TemplateEngine.Edge.Constraints.WorkloadConstraintFactory.Microsoft.TemplateEngine.Abstractions.Constraints.ITemplateConstraintFactory.CreateTemplateConstraintAsync(Microsoft.TemplateEngine.Abstractions.IEngineEnvironmentSettings environmentSettings, System.Threading.CancellationToken cancellationToken) Line 27	C#
System.Private.CoreLib.dll!System.Threading.Tasks.Task<System.Threading.Tasks.Task<Microsoft.TemplateEngine.Abstractions.Constraints.ITemplateConstraint>>.InnerInvoke() Line 209	C#
System.Private.CoreLib.dll!System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(System.Threading.Thread threadPoolThread, System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state) Line 192	C#
System.Private.CoreLib.dll!System.Threading.Tasks.Task.ExecuteWithThreadLocal(ref System.Threading.Tasks.Task currentTaskSlot, System.Threading.Thread threadPoolThread) Line 2194	C#
System.Private.CoreLib.dll!System.Threading.ThreadPoolWorkQueue.Dispatch() Line 690	C#
System.Private.CoreLib.dll!System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart() Line 1328	C#
Microsoft.TemplateEngine.Edge.dll!Microsoft.TemplateEngine.Edge.TemplateConstraintManager.TemplateConstraintManager(Microsoft.TemplateEngine.Abstractions.IEngineEnvironmentSettings engineEnvironmentSettings) Line 32	C#
Microsoft.TemplateEngine.Cli.dll!Microsoft.TemplateEngine.Cli.TemplatePackageCoordinator.TemplatePackageCoordinator(Microsoft.TemplateEngine.Abstractions.IEngineEnvironmentSettings environmentSettings, Microsoft.TemplateEngine.Edge.Settings.TemplatePackageManager templatePackageManager)	Unknown
Microsoft.TemplateEngine.Cli.dll!Microsoft.TemplateEngine.Cli.Commands.TemplateCommand.InvokeAsync(System.CommandLine.ParseResult parseResult, System.Threading.CancellationToken cancellationToken)	Unknown
Microsoft.TemplateEngine.Cli.dll!Microsoft.TemplateEngine.Cli.Commands.InstantiateCommand.HandleTemplateInstantiationAsync(Microsoft.TemplateEngine.Cli.Commands.InstantiateCommandArgs args, Microsoft.TemplateEngine.Abstractions.IEngineEnvironmentSettings environmentSettings, Microsoft.TemplateEngine.Edge.Settings.TemplatePackageManager templatePackageManager, Microsoft.TemplateEngine.Cli.TemplateGroup templateGroup, System.Threading.CancellationToken cancellationToken)	Unknown
Microsoft.TemplateEngine.Cli.dll!Microsoft.TemplateEngine.Cli.Commands.InstantiateCommand.ExecuteIntAsync(Microsoft.TemplateEngine.Cli.Commands.InstantiateCommandArgs instantiateArgs, Microsoft.TemplateEngine.Abstractions.IEngineEnvironmentSettings environmentSettings, Microsoft.TemplateEngine.Edge.Settings.TemplatePackageManager templatePackageManager, System.CommandLine.ParseResult parseResult, System.Threading.CancellationToken cancellationToken)	Unknown
[Resuming Async Method]	
System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<Microsoft.TemplateEngine.Cli.NewCommandStatus>.AsyncStateMachineBox<Microsoft.TemplateEngine.Cli.Commands.InstantiateCommand.<ExecuteIntAsync>d__17>.ExecutionContextCallback(object s) Line 131	C#
System.Private.CoreLib.dll!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state) Line 138	C#
System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<Microsoft.TemplateEngine.Cli.NewCommandStatus>.AsyncStateMachineBox<Microsoft.TemplateEngine.Cli.Commands.InstantiateCommand.<ExecuteIntAsync>d__17>.MoveNext(System.Threading.Thread threadPoolThread) Line 163	C#
System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<Microsoft.TemplateEngine.Cli.NewCommandStatus>.AsyncStateMachineBox<Microsoft.TemplateEngine.Cli.Commands.InstantiateCommand.<ExecuteIntAsync>d__17>.MoveNext() Line 146	C#
System.Private.CoreLib.dll!System.Runtime.CompilerServices.TaskAwaiter.OutputWaitEtwEvents.AnonymousMethod__12_0(System.Action innerContinuation, System.Threading.Tasks.Task innerTask) Line 140	C#
System.Private.CoreLib.dll!System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Action action, bool allowInlining) Line 162	C#
System.Private.CoreLib.dll!System.Threading.Tasks.Task.RunContinuations(object continuationObject) Line 2850	C#
System.Private.CoreLib.dll!System.Threading.Tasks.Task<System.__Canon>.TrySetResult(System.__Canon result) Line 171	C#
System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<System.__Canon>.SetExistingTaskResult(System.Threading.Tasks.Task<System.__Canon> task, System.__Canon result) Line 389	C#
[Completed] Microsoft.TemplateEngine.Cli.dll!Microsoft.TemplateEngine.Cli.Commands.InstantiateCommand.GetTemplateGroupsAsync(Microsoft.TemplateEngine.Edge.Settings.TemplatePackageManager templatePackageManager, Microsoft.TemplateEngine.Cli.HostSpecificDataLoader hostSpecificDataLoader, System.Threading.CancellationToken cancellationToken)	Unknown
System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<System.__Canon>.AsyncStateMachineBox<Microsoft.TemplateEngine.Cli.Commands.InstantiateCommand.<GetTemplateGroupsAsync>d__11>.ExecutionContextCallback(object s) Line 131	C#
System.Private.CoreLib.dll!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state) Line 138	C#
System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<System.Collections.Generic.IEnumerable<Microsoft.TemplateEngine.Cli.TemplateGroup>>.AsyncStateMachineBox<Microsoft.TemplateEngine.Cli.Commands.InstantiateCommand.<GetTemplateGroupsAsync>d__11>.MoveNext(System.Threading.Thread threadPoolThread) Line 163	C#
System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<System.__Canon>.AsyncStateMachineBox<Microsoft.TemplateEngine.Cli.Commands.InstantiateCommand.<GetTemplateGroupsAsync>d__11>.MoveNext() Line 146	C#
System.Private.CoreLib.dll!System.Runtime.CompilerServices.TaskAwaiter.OutputWaitEtwEvents.AnonymousMethod__12_0(System.Action innerContinuation, System.Threading.Tasks.Task innerTask) Line 140	C#
System.Private.CoreLib.dll!System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Action action, bool allowInlining) Line 162	C#
System.Private.CoreLib.dll!System.Threading.Tasks.Task.RunContinuations(object continuationObject) Line 2850	C#
System.Private.CoreLib.dll!System.Threading.Tasks.Task<System.__Canon>.TrySetResult(System.__Canon result) Line 171	C#
System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<System.__Canon>.SetExistingTaskResult(System.Threading.Tasks.Task<System.__Canon> task, System.__Canon result) Line 389	C#
[Completed] Microsoft.TemplateEngine.Edge.dll!Microsoft.TemplateEngine.Edge.Settings.TemplatePackageManager.GetTemplatesAsync(System.Threading.CancellationToken cancellationToken) Line 163	C#
System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<System.__Canon>.AsyncStateMachineBox<Microsoft.TemplateEngine.Edge.Settings.TemplatePackageManager.<GetTemplatesAsync>d__16>.ExecutionContextCallback(object s) Line 131	C#
System.Private.CoreLib.dll!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state) Line 138	C#
System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<System.Collections.Generic.IReadOnlyList<Microsoft.TemplateEngine.Abstractions.ITemplateInfo>>.AsyncStateMachineBox<Microsoft.TemplateEngine.Edge.Settings.TemplatePackageManager.<GetTemplatesAsync>d__16>.MoveNext(System.Threading.Thread threadPoolThread) Line 163	C#
System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<System.__Canon>.AsyncStateMachineBox<Microsoft.TemplateEngine.Edge.Settings.TemplatePackageManager.<GetTemplatesAsync>d__16>.MoveNext() Line 146	C#
System.Private.CoreLib.dll!System.Runtime.CompilerServices.TaskAwaiter.OutputWaitEtwEvents.AnonymousMethod__12_0(System.Action innerContinuation, System.Threading.Tasks.Task innerTask) Line 140	C#
System.Private.CoreLib.dll!System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Action action, bool allowInlining) Line 162	C#
System.Private.CoreLib.dll!System.Threading.Tasks.Task.RunContinuations(object continuationObject) Line 2850	C#
System.Private.CoreLib.dll!System.Threading.Tasks.Task<System.__Canon>.TrySetResult(System.__Canon result) Line 171	C#
System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<System.__Canon>.SetExistingTaskResult(System.Threading.Tasks.Task<System.__Canon> task, System.__Canon result) Line 389	C#
[Completed] Microsoft.TemplateEngine.Edge.dll!Microsoft.TemplateEngine.Edge.Settings.TemplatePackageManager.UpdateTemplateCacheAsync(bool needsRebuild, System.Threading.CancellationToken cancellationToken) Line 383	C#
System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<System.__Canon>.AsyncStateMachineBox<Microsoft.TemplateEngine.Edge.Settings.TemplatePackageManager.<UpdateTemplateCacheAsync>d__23>.ExecutionContextCallback(object s) Line 131	C#
System.Private.CoreLib.dll!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state) Line 138	C#
System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<Microsoft.TemplateEngine.Edge.Settings.TemplateCache>.AsyncStateMachineBox<Microsoft.TemplateEngine.Edge.Settings.TemplatePackageManager.<UpdateTemplateCacheAsync>d__23>.MoveNext(System.Threading.Thread threadPoolThread) Line 163	C#
System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<System.__Canon>.AsyncStateMachineBox<Microsoft.TemplateEngine.Edge.Settings.TemplatePackageManager.<UpdateTemplateCacheAsync>d__23>.MoveNext() Line 146	C#
System.Private.CoreLib.dll!System.Runtime.CompilerServices.TaskAwaiter.OutputWaitEtwEvents.AnonymousMethod__12_0(System.Action innerContinuation, System.Threading.Tasks.Task innerTask) Line 140	C#
System.Private.CoreLib.dll!System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Action action, bool allowInlining) Line 162	C#
System.Private.CoreLib.dll!System.Threading.Tasks.Task.RunContinuations(object continuationObject) Line 2850	C#
System.Private.CoreLib.dll!System.Threading.Tasks.Task<System.__Canon>.TrySetResult(System.__Canon result) Line 171	C#
System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<System.__Canon>.SetExistingTaskResult(System.Threading.Tasks.Task<System.__Canon> task, System.__Canon result) Line 389	C#
[Completed] Microsoft.TemplateEngine.Edge.dll!Microsoft.TemplateEngine.Edge.Settings.TemplatePackageManager.GetTemplatePackagesAsync(bool force, System.Threading.CancellationToken cancellationToken) Line 121	C#
System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<System.__Canon>.AsyncStateMachineBox<Microsoft.TemplateEngine.Edge.Settings.TemplatePackageManager.<GetTemplatePackagesAsync>d__13>.ExecutionContextCallback(object s) Line 131	C#
System.Private.CoreLib.dll!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state) Line 138	C#
System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<System.Collections.Generic.IReadOnlyList<Microsoft.TemplateEngine.Abstractions.TemplatePackage.ITemplatePackage>>.AsyncStateMachineBox<Microsoft.TemplateEngine.Edge.Settings.TemplatePackageManager.<GetTemplatePackagesAsync>d__13>.MoveNext(System.Threading.Thread threadPoolThread) Line 163	C#
System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<System.__Canon>.AsyncStateMachineBox<Microsoft.TemplateEngine.Edge.Settings.TemplatePackageManager.<GetTemplatePackagesAsync>d__13>.MoveNext() Line 146	C#
System.Private.CoreLib.dll!System.Runtime.CompilerServices.TaskAwaiter.OutputWaitEtwEvents.AnonymousMethod__12_0(System.Action innerContinuation, System.Threading.Tasks.Task innerTask) Line 140	C#
System.Private.CoreLib.dll!System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Action action, bool allowInlining) Line 162	C#
System.Private.CoreLib.dll!System.Threading.Tasks.Task.RunContinuations(object continuationObject) Line 2850	C#
System.Private.CoreLib.dll!System.Threading.Tasks.Task<System.__Canon>.TrySetResult(System.__Canon result) Line 171	C#
System.Private.CoreLib.dll!System.Threading.Tasks.UnwrapPromise<System.Collections.Generic.IReadOnlyList<Microsoft.TemplateEngine.Abstractions.TemplatePackage.ITemplatePackage>>.TrySetFromTask(System.Threading.Tasks.Task task, bool lookForOce) Line 122	C#
System.Private.CoreLib.dll!System.Threading.Tasks.UnwrapPromise<System.__Canon>.ProcessInnerTask(System.Threading.Tasks.Task task) Line 134	C#
System.Private.CoreLib.dll!System.Threading.Tasks.Task.RunContinuations(object continuationObject) Line 2741	C#
System.Private.CoreLib.dll!System.Threading.Tasks.Task.FinishSlow(bool userDelegateExecute) Line 1971	C#
System.Private.CoreLib.dll!System.Threading.Tasks.Task.ExecuteWithThreadLocal(ref System.Threading.Tasks.Task currentTaskSlot, System.Threading.Thread threadPoolThread) Line 2205	C#
System.Private.CoreLib.dll!System.Threading.ThreadPoolWorkQueue.Dispatch() Line 690	C#
System.Private.CoreLib.dll!System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart() Line 1328	C#
[Async Call Stack]	
[Async] Microsoft.TemplateEngine.Cli.dll!Microsoft.TemplateEngine.Cli.Commands.BaseCommand<Microsoft.TemplateEngine.Cli.Commands.NewCommandArgs>.CommandAction.InvokeAsync(System.CommandLine.ParseResult parseResult, System.Threading.CancellationToken cancellationToken)	Unknown

@dsplaisted
Copy link
Member

Because it's buried in the linked issue, here's what the failure would actually look like when you tried to create a project using a template that used the workload constraint:

Failed to instatiate template '.NET Aspire Application', the following constraints are not met:
  workload: The constraint 'workload' failed to initialize: Manifest provider Microsoft.NET.Sdk.WorkloadManifestReader.SdkDirectoryWorkloadManifestProvider returned a duplicate manifest ID 'microsoft.net.sdk.android' [/vmr/artifacts/obj/extracted-dotnet-sdk/sdk-manifests/9.0.100-preview.1/microsoft.net.sdk.android/34.99.0-preview.1.151/WorkloadManifest.json] that conflicts with existing manifest [/vmr/artifacts/obj/extracted-dotnet-sdk/sdk-manifests/9.0.100-preview.1/microsoft.net.sdk.android/34.99.0-preview.1.151/WorkloadManifest.json]

@rmarinho
Copy link
Member

rmarinho commented Oct 22, 2024

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants