SignTool can fail if it repacks too much big stuff at once

[mmitche]

• This issue is blocking
• This issue is causing unreasonable pain

@dibarbet, when adding a new nupkg to the roslyn build, started hitting OOMs on the repack steps. Upon further investigation, it seems likely that the repack step, which runs in parallel, is OOMing when there are too many large packages in memory at once. I think we have not hit this in the past because most repos have only a few large unpkgs or zip files, thus it is unlikely that their streams will be open at the same time.

Release Note Category

• Feature changes/additions
• Bug fixes
• Internal Infrastructure Improvements

Release Note Description

[mmitche]

There is one potential issue, and another potential fix:

• https://github.com/dotnet/arcade/blob/main/src/Microsoft.DotNet.SignTool/src/ZipData.cs#L116 - The File.Open here may not be immediately getting disposed.
• It's possible that if you had 16 very large files in parallel, you would always hit this issue. Instead, we should determine some size limit, then repack those larger than the threshold in serial.

[mmitche]

Closed as fixed by serialization of large containers

[RikkiGibson]

We may be experiencing this issue again in the following builds:

https://dev.azure.com/dnceng/internal/_build/results?buildId=2394040&view=results
https://dev.azure.com/dnceng/internal/_build/results?buildId=2394161&view=results

edit: ignore, we think this was resolved by dotnet/roslyn#72322

[mmitche]

Hmm, might want to check the logs and see what parallelism is getting passed to the task.

[mmitche]

Great, let me know if you see it again after the change.

[jjonescz]

The change has been merged but the OOM ocurred again:

• https://dev.azure.com/dnceng/internal/_build/results?buildId=2395027&view=results

Also other signing errors ocurred, not sure if related:

• https://dev.azure.com/dnceng/internal/_build/results?buildId=2394384&view=results
• https://dev.azure.com/dnceng/internal/_build/results?buildId=2394392&view=results

[mmitche]

The change has been merged but the OOM ocurred again:

• https://dev.azure.com/dnceng/internal/_build/results?buildId=2395027&view=results

Also other signing errors ocurred, not sure if related:

• https://dev.azure.com/dnceng/internal/_build/results?buildId=2394384&view=results
• https://dev.azure.com/dnceng/internal/_build/results?buildId=2394392&view=results

I think the other errors are unrelated. It looks like a network error I haven't seen before, on the Microbuild side. I suggest setting SignToolRepackMaximumParallelFileSize to 64 and see whether that helps things. The problem with roslyn is that it has 3 copies of basically every file, and a lot of files in the mid-range size.