Start publishing Unified Builds to BAR

[premun]

Depends on #4157

• Figure out if packages are placed in a common directory in the UB build -> packages are now under artifacts/packages/<config>/Shipping/** and assets are under artifacts/assets/<config>.
• Figure out if we want to use /src/arcade/eng/common or /eng/common. I think the first as the other doesn't get updated really -> we will use /eng/common
• Create and Publish VerticalManifests during unified builds #4198
• Create and publish a MergedManifest for unified builds #4199
• Add a Publish using Darc stage that will publish a build to a darc channel

The UB build: https://dnceng.visualstudio.com/internal/_build?definitionId=1330

Depends On

• Resolve dual 'source of truth' for publishing info #4101

[premun]

There's no urgency with that task yet as we don't have assets in the build yet but maybe we could publish an empty BAR build from the VMR so that we can trigger fake subscriptions.

It might be better to start publishing this in the INT Maestro BAR then.

[ViktorHofer]

Figure out if we want to use /src/arcade/eng/common or /eng/common. I think the first as the other doesn't get updated really

We should use /eng/common. src/arcade/eng/common, src/installer/eng/common and /eng/common are always in sync (identical).

Figure out if packages are placed in a common directory in the UB build

I'm working on that right now.

[ViktorHofer]

Depends on #4157

[ViktorHofer]

FYI @akoeplinger

[dkurepa]

Currently, every repo uses the following publishing flow:

flowchart TB
subgraph emsdk
    subgraph "win x64 leg"
        winBuild[Build] --> winSign[Sign]
        winSign --> winAssetManifest[Create Asset Manifest]
        winAssetManifest --> winPublish[Publish to Artifacts]
    end
    subgraph "linux x64 leg"
        linuxBuild[Build] --> linuxSign[Sign]
        linuxSign --> linuxAssetManifest[Create Asset Manifest]
        linuxAssetManifest --> linuxPublish[Publish to Artifacts]
    end
    subgraph "macOS x64 leg"
        macOSBuild[Build] --> macOSSign[Sign]
        macOSSign --> macOSAssetManifest[Create Asset Manifest]
        macOSAssetManifest --> macOSPublish[Publish to Artifacts]
    end
    subgraph bar["Publish to Bar"]
        mergedManifest["Create Merged Manifest"] --> publishBar["Publish build to bar, using Merged Manifest"]
    end
    winPublish --> bar
    linuxPublish --> bar
    macOSPublish --> bar
end

Loading

• Every build leg creates and publishes packages as artifacts and creates its own Asset Manifest
• After all build legs are finished, a MergedManifest is created using all Asset Manifests

Things will work a bit differently for Unified builds:

flowchart TB
subgraph UnifiedBuild
    subgraph win-x64
        subgraph winEmsdk["emsdk"]
            emsdkWinBuild["Build"] --> emsdkWinPublish[Publish produced bits and AssetManifest]
        end
        subgraph winRuntime["runtime"]
            runtimeWinBuild["Build"] --> runtimeWinPublish[Publish produced bits and AssetManifest]
        end
        subgraph winAspnetcore["aspnetcore"]
            aspWinBuild["Build"] --> aspWinPublish[Publish produced bits and AssetManifest]
        end
        emsdkWinPublish --> winMergeManifests[Merge Windows Asset Manifests into a VerticalManifest]
        runtimeWinPublish --> winMergeManifests
        aspWinPublish --> winMergeManifests
    end
    subgraph linux-x64
        subgraph linuxEmsdk["emsdk"]
            emsdkLinuxBuild["Build"] --> emsdkLinuxPublish[Publish produced bits and AssetManifest]
        end
        subgraph linuxRuntime["runtime"]
            runtimeLinuxBuild["Build"] --> runtimeLinuxPublish[Publish produced bits and AssetManifest]
        end
        subgraph linuxAspnetcore["aspnetcore"]
            aspLinuxBuild["Build"] --> aspLinuxPublish[Publish produced bits and AssetManifest]
        end
        emsdkLinuxPublish --> linuxMergeManifests[Merge Linux Asset Manifests into a VerticalManifest]
        runtimeLinuxPublish --> linuxMergeManifests
        aspLinuxPublish --> linuxMergeManifests
    end
    winMergeManifests --> mergedManifest["Generate Merged Manifest"]
    linuxMergeManifests --> mergedManifest
    mergedManifest --> barPublish["Publish Unified Build to BAR"]
end

Loading

• Every leg will build its own vertical.
• In a vertical, every repo will be built using arcade, so every repo will have its own AssetManifest
• After every vertical build, we'll create a VerticalAssetManifest, which will combine all AssetManifests, and also add additional data if needed
• After all legs are done, we'll create a MergedManifest by merging all of the VerticalAsseetManifests and use it to create a BAR build

[ViktorHofer]

cc @mmitche

[dkurepa]

I'd like to get some additional clarity on the scope of this issue, and its dependencies.
Currently we have two other related issues:

• Resolve dual 'source of truth' for publishing info #4101 that @NikolaMilosavljevic is working on
  • One of the results of this issue is that we'll have AssetManifests produced for each repo.
  • We'll use these AssetManifests to produce and publish a VerticalAssetManifest for each leg. What's unclear is which issue this part belongs to.
• Implement a strategy to select the assets to publish per vertical #4157
  • The result of this issue should be a strategy that we'll use when creating the MergedManifest from VerticalAssetManifests, that will be used for the Publish using Darc stage.

So with that said, for this issue we will:

• If determined that creating VerticalAssetManifests belongs here, create VerticalAssetManifests and publish them as artifacts
• Create a MergedManifest from these VerticalAssetManifests using the strategy determined in Resolve dual 'source of truth' for publishing info #4101, once all legs are finished, and use it to create a BAR build
• Publish the new BAR build to a channel, using the Publish using Darc stage

We will also have to incorporate assets produced in join points.

@mmitche @ViktorHofer is this all this correct?

[mmitche]

Resolve dual 'source of truth' for publishing info #4101 that @NikolaMilosavljevic is working on

• One of the results of this issue is that we'll have AssetManifests produced for each repo.
• We'll use these AssetManifests to produce and publish a VerticalAssetManifest for each leg. What's unclear is which issue this part belongs to.

I'm not sure how far exactly @NikolaMilosavljevic's change will go beyond the per-repo publishing. If he is not covering vertical manifest creation in #4101, then I think it belongs here. All the work is highly related, it's just a matter of how you want to divide it up between folks working in the area.

The rest of that looks correct to me.

[dkurepa]

@NikolaMilosavljevic just an FYI, we discussed the Vertical Manifest generation on the weekly Product construction meting and decided that we'll handle it as a part of this issue

[NikolaMilosavljevic]

@NikolaMilosavljevic just an FYI, we discussed the Vertical Manifest generation on the weekly Product construction meting and decided that we'll handle it as a part of this issue

Thanks. I presume that applies to Vertical merged manifest. Repo manifests will be produced by work I'm completing.

[pavel-purma]

I have started on this one. Already created the channel for .NET 10 named ".NET 10.0.1xx UB" and set up the default channel for the main unified build pipeline.
I already did few test runs. It seems there is an issue in the MergedManifest are not information provided about the repo ... fixing that