Resolve dual 'source of truth' for publishing info #4101
Comments
dotnet-issue-labeler
bot
added
area-infra
mmitche
added
the
Epic
|
Other notes. The way we do things today leads to differences between VMR and individual builds around how repos are finding upstream input artifacts. For instance: https://github.com/dotnet/installer/blob/main/src/redist/targets/GenerateLayout.targets#L264 Because VMR builds don't really "publish" the outputs of the repo build based on the layout the repo specifies, they just copy them over, a repo downstream will need to check the flag for VMR build, then alter the relative path that it's looking in. Here's how I generally envision this works:
The main difference between how repos move artifacts between themselves and this is just that the former has one extra step we don't need: Repo automation:
In the VMR case, if you wanted to match this exactly, you'd do:
But we don't really need the additional intermediate storage location and so a couple steps are degenerate. We can have Publish To Build artifacts do more work:
|
Couple weeks ago when Nikolai implemented the removal of the intermediate packages in the VMR, we decided that the inner repo's responsibility shouldn't be to lay the files out to a specific location inside the VMR. Instead we created the manifest file that the VMR orchestrator then reads to copy the files to the shared location. Just to provide context... I think that you propose makes sense and that it should be fine if the Publish step in the inner repo copies the artifacts directly into that shared location. And I'm all for merging these two different publishing paths. Maintaining both is cumbersome. |
|
@mmitche publishing is conditioned to not run in product or inner-build. Were you thinking of enabling this in VMR, in inner-, or outer- repo build? If outer, we'd still be using |
Yeah I was thinking that enabling publishing in the inner builds and product builds. Essentially no diff in when publishing is run. I think you still need the nupkg categorization like you say (for repo source build) to correctly divide up the artifacts for the intermediates. But I think this can be done within the confines of the existing publishing infra. Brainstorming: So in repo-source build you don't want to publish the inner-source artifacts directly (since they collide with the officially built MSFT ones), you want to wrap them and publish those. I wonder whether you could integrate the categorization into the manifest artifact data though, then use that to create intermediates like we do today. For example, let's say runtime produces 50 nupkgs. Inner source build runs, using the publishing infra to identify those 50 nupkgs and produce a manifest like it would with any normal build. But we disable the push to AzDO part and just generate the manifest and a temporary layout on disk. This would be something like that change of "PushToAzureDevOpsArtifacts->PushToBuildArtifacts", where outer-source build would just specify that things get published to disk, not emit VSO logging commands. This would be very similar to how it would work in the full product build (copy to a specified shared location). Then we add manifest data for those artifacts which indicates that category. This would fit in exactly with the way the repos work today with their other types of categorization (e.g. nonshipping vs. shipping). The repo knows this categorization today, it just lives in outer vs. inner SB. So we move it and use the extensible ManifestArtifactData to indicate the category. : https://github.com/dotnet/aspnetcore/blob/main/eng/Publishing.props#L68-L72 Maybe for this artifact this goes to: So inner-source build is using the normal publish infra and let's say the shared location specified for publishing is Then, in full orchestrated mode, the main differences end up pretty minimal from repo source build:
|
|
That should work. Would orchestrator still read the, new, manifest and copy the artifacts, or would outer repo build copy them to local storage during publish? |
|
I think the orchestrator doesn't have to re-copy the artifacts. I think it would just tell the outer repo where the common publish location is, and outer repo would pass to inner repo. But, I think it would read the manifest to determine the versions of outputted artifacts. This could then replace the logic today that determines versions for *PackageVersions.props. Rather than reading the output directory, read the manifests of the builds that a repo mentions as its dependencies. So even if msbuild builds before roslyn, unless we identify that msbuild should flow to roslyn, we don't actually read the manifests and generate PackageVersions.props This would then fit rather nicely with the need to only update versions based on what a repo depends on for @mthalman's multhreaded work. Again, this is mostly brainstorming. Needs to be designed and tested. I can kind of see how it would fit together though and get rid of a lot of the custom SB infra in the process. |
|
Few more notes for brainstorming. 1 - Source-build package consumption expects both shipping and non-shipping packages in the same folder. So, we either 1) change that and introduce shipping and non-shipping feeds, or 2) allow new publishing process to copy both package types to the same output location. 2 - Outer build is producing a symbols package by harvesting all PDBs from inner-repo artifacts ( 3 - Outer build is currently filtering out all 4 - Outer build is currently building a list of non-shipping packages and including it in the Intermediate package. The list was used to filter-out non-shipping packages during VMR's prebuilt detection. As we do not use Intermediate packages in VMR anymore, this list isn't used and all related code should be removed. 5 - There is already a simple target, that's currently blocked from running in inner build, that copies artifacts to source-build storage - |
For a second I wondered if NuGet respects subfolders in local feeds as well but it doesn't. I think introducing a second local feed for the shipping vs non-shipping folder makes sense. It would make what I'm currently doing for #4104 much cleaner.
In general it would be good to not rely on the outer repo build at all for publishing. The MSFT build doesn't have that concept. That would help with eventually collapsing the outer and the inner repo build.
Exactly.
Interesting. Can you please point me to that code in question? It might help with the above issue that I'm currently working on. |
Agreed. That maps very nicely onto what we do for individual repo builds.
Prebuilt detection should just be able to read the manifest data for non-shipping vs. shipping. I think we would include the manifest in the intermediate packages. |
|
OK - by using relative paths for publishing assets, we get something like this in Linux build - previously all these files were in a single directory: It's a bit confusing for consumption by distro maintainers. There are also two versioned sub-directories under Please ignore the issue with missing version in PSB artifacts and symbols archive - working on a fix. |
|
It does match what we ship to customers though (this is the dotnetcli layout). What happens to the sdk outputs in the current layout? |
OK, that makes sense. Today, all these files are in root assets directory, no sub-directories. |
|
The dual directory is also point in time for .NET 9, since the installer+sdk repo merge will happen eventually and then there will be one dir for the sdk. |
|
We could preserve the flat assets layout in source-only build, without introducing new task parameter (i.e. we could use existing We would need to keep the source-only conditioning in |
|
I think We could also just take the final outputs and put them to a flat directory in the VMR, but let the individual repo flow use the correct layout. |
Perhaps we could just copy the SDK to assets root, after all repos finish building. SDK, related symbols archive and PSB archive would be in root, and those are the key source-build artifacts. |
|
This is mostly completed aside from runtime and windowsdesktop. For those two repositories I realized that while we now unify publishing inside the VMR, outside they continue to use prepare-artifacts.proj and don't pass the They generate at least checksum files. The windowsdesktop prepare-artifacts.proj should be easy to remove (maybe it's really just the checksum generation that is missing). The runtime one will take more time. I think the remaining work for those two repositories can be done in follow-ups but we shouldn't close this issues until that work is completed. Note that this doesn't need to be done for aspnetcore (which also generates checksums) as they already use Arcade's Publish.proj with the Publishing.props extension point. |
|
With dotnet/windowsdesktop#4251, windowsdesktop's publishing infrastructure is now unified. deployment-tools and runtime remain. |
|
Deployment-tools ✅ dotnet/deployment-tools#348 I don't plan to work on runtime, that's much more difficult. I think we should get back to it when we have signing support on non-Windows. Meanwhile, this issue should be parked. |
|
Just filed #4239. Closing the epic. |
github-project-automation
bot
moved this from In Progress
to Done
in .NET Unified Build
Right now we have two sources of truth for publishing information. We have the standard arcade publishing routines that involve Publishing.props, shipping/non-shipping info, relative paths for blobs, etc. Then we have the source-build method, which involves a couple targets (GetCategorizedIntermediateNupkgContents). This method isn't the source of truth for the product at the moment, and also doesn't preserve a lot of interesting information like relative blob path layouts, what assets are produced by a repo leg (it assumes zips files and nupkgs). We could add that kind of info in there, but we would just end up duplicating existing info. It also doesn't generate manifests that are usable by downstream infra like the staging pipelines.
This needs to get resolved. Ideally, we would use the same infra for both VMR and individual repo builds. We would go through the same standard arcade paths for publishing, generating the same manifest data, between VMR and individual repo builds.
So basically, when arcade repos build, they runs the normal Publish.proj, creating a manifest using PushToAzureDevOpsArtifacts with the outputs that arcade says it produced. Same for runtime, aspnetcore, etc. this would include the manifest. When we get to the end of the VMR build, we now have 1 manifest for each repo built in each vertical. These get merged together just like any other build.
The important bits here are:
Work Items
The text was updated successfully, but these errors were encountered: