Local live-live builds #494

jkoritzinsky · 2019-12-03T22:26:43Z

This PR implements local live-live builds. The eng/liveBuilds.targets MSBuild file is the central location for live dependency file discovery. There are various MSBuild property hooks in there to enable overriding the location of the various consumed live components.

Work completed in this PR:

Libraries consumes:
- A live copy of the CoreCLR build's shared framework and System.Private.CoreLib output.
Installer consumes:
- A live copy of the CoreCLR build's shared framework and System.Private.CoreLib output
- A live copy of the Libraries shared framework ref and implementation assemblies.
- A live copy of Microsoft.NETCore.Platforms and Microsoft.NETCore.Targets
CoreCLR managed test build consumes:
- A live copy of the Libraries shared framework and OOB ref assemblies.
CoreCLR Core_Root generation:
- A live copy of the Libraries shared framework and OOB implementation assemblies.
Customization points for each "unit" of output that is consumed between subset categories.
Subsets that have dependencies are serialized such that a root build will build subsets in the correct order for inter-subset dependencies.

Future work for another PR once we are off the critical path:

Refactor out target overrides in src/installer/pkg/projects/netcoreapp/src/localnetcoreapp.override.targets so we can send back a patch to Arcade and remove localnetcoreapp.override.targets.
Extend eng/liveBuilds.targets to have a target for consuming a live build of the hosting layer.
Consume a live build of the hosting layer in the CoreFX test build instead of consuming packages.

Superscedes #55 (needed to be recreated after the repo went public).

…dependencies are in the building subset/subset category.

…live-builds

…ndencies.

…ding a new BinplaceConfiguration. Update test_dependencies in the CoreCLR tree to use that to restore OOB libraries.

…he reference is only needed at runtime and Core_Root already has System.Drawing.Common).

…live world, we'll run this test configuration by pointing the CoreFX test host build step at the artifacts from a checked CoreCLR build (possible in both local and CI builds).

…sembly set.

…(breaks the live-live build on a clean system).

…ms to be unneeded since the SDK now packages a RID graph. Use the live RID graph in test_runtime.csproj where we may actually be looking for the live RID graph.

…live-builds

…eferences from the installers.

…lean build.

…ckaged Microsoft.NETCore.Platforms package.

…ntime into live-live-builds

…oreclr-and-libraries-job template.

…build.

…PackageName).versions.txt file that other shared frameworks also don't generate.

jkoritzinsky · 2019-12-11T21:56:22Z

I am so confused by the failures here. The failures say that the CoreCLR subset was not found at path/to/artifacts/transport/coreclr, but just 3 steps earlier, the CoreCLR artifacts were downloaded and extracted to that exact folder...

jkoritzinsky · 2019-12-11T21:57:56Z

And with the exact same parameters it passes locally....

jkoritzinsky · 2019-12-11T22:05:28Z

And the exact same thing works perfectly in the libraries builds.

…t the CoreCLR download.

…ler build.

…the LibrariesConfiguration is different from the installer build configuration.

… be consistent with the libraries build so our package versions match.

…typo in TargetPath for CoreCLR assets (runtime instead of runtimes). Fix paths for CoreCLR cross-target files as well as crossgen to place them in the correct directories.

jkoritzinsky · 2019-12-12T23:15:38Z

Everything is passing except the check for the "platform manifest" in the Microsoft.NETCore.App.Ref targeting pack. This is related to https://github.com/dotnet/core-setup/issues/8853.

ViktorHofer · 2019-12-12T23:18:49Z

That's great! Does Davis's note help?

I think I should work on this, and in the meantime I think it's reasonable to disable "full" platform manifest generation (set BuildFullPlatformManifest to false) to unblock dotnet/runtime "live live" builds.

jkoritzinsky · 2019-12-12T23:20:20Z

I already followed that when I started this PR and I think that might be part of what's causing the issue now.

… generate.

ViktorHofer · 2019-12-13T10:31:12Z

@jkoritzinsky is the PR ready to be merged?

jkoritzinsky · 2019-12-13T16:37:01Z

Yep!

ViktorHofer · 2019-12-13T17:20:03Z

Niceeee. I assume you will merge it after sending out the mail? :)

jkoritzinsky · 2019-12-13T17:29:23Z

Will do!

jkoritzinsky added 30 commits November 14, 2019 15:59

Convert libraries and installer to use a live CoreCLR.

9c85652

Installer uses a live libraries build for DLLs.

88f21a9

Remove newline in path.

6164fa8

Enable live-live builds to build required live dependencies when the …

94a436b

…dependencies are in the building subset/subset category.

Merge branch 'master' of https://github.com/dotnet/runtime into live-…

3b57b00

…live-builds

Update path to coreclr output.

ae23413

Fix typo in coreclr.proj.

02f2afa

Remove last usage of CoreCLRPDBOverridePath.

42f2954

Update path of binplaced runtime to include OS and Arch.

106ea54

Fix typo in coreclr/tests/src/Common/Directory.Build.targets.

39ab7ff

Restore CoreFX shared framework from live build for CoreCLR test depe…

23f8963

…ndencies.

Remove dependency on Microsoft.NETCore.CoreCLR.TestDependencies package.

c2b930c

Enable referencing live builds of the OOB libraries by temporarily ad…

f73ad73

…ding a new BinplaceConfiguration. Update test_dependencies in the CoreCLR tree to use that to restore OOB libraries.

Remove reference to System.Drawing.Common from NETClientPrimitives (t…

eb0e522

…he reference is only needed at runtime and Core_Root already has System.Drawing.Common).

Remove package-based checked CoreCLR + CoreFX test runs. In the live-…

68e746b

…live world, we'll run this test configuration by pointing the CoreFX test host build step at the artifacts from a checked CoreCLR build (possible in both local and CI builds).

Only copy the UCRT on Windows.

cba76e8

CoreCLR test build now builds against a live libraries sfx+oob ref as…

3d3f216

…sembly set.

Override GetCorePackagePaths target to not reference unused packages …

2f9d951

…(breaks the live-live build on a clean system).

Remove live references to Microsoft.NETCore.Platforms now that it see…

6601e93

…ms to be unneeded since the SDK now packages a RID graph. Use the live RID graph in test_runtime.csproj where we may actually be looking for the live RID graph.

Merge branch 'master' of https://github.com/dotnet/runtime into live-…

48c45f0

…live-builds

Fix coreclr build arg generation bug that was causing build failures.

9f35f9d

Remove references to Microsoft.NETCore.Platforms outside of package r…

8b4791d

…eferences from the installers.

Fix problems in installer build in live-live that were found from a c…

0344a53

…lean build.

Merge branch 'master' into live-live-builds

aa4b349

Remove unused dependency on Microsoft.NETCore.Targets.

314c4a0

Fix Microsoft.NETCore.App -> Microsoft.NETCore.Platforms dependency.

f640bd9

Remove Microsoft.NETCore.Targets version from Versions.props.

80669fa

Remove unused deps.csproj file that was the last dependency on the pa…

cea542d

…ckaged Microsoft.NETCore.Platforms package.

Merge branch 'live-live-builds' of https://github.com/jkoritzinsky/ru…

01f5130

…ntime into live-live-builds

Fix OSGroup calculation for dependent projects off-Windows.

425d034

jkoritzinsky added 6 commits December 10, 2019 18:12

Inclue doc files in the Microsoft.NETCore.App.Internal package

3cbbb3e

Try making the runtime-installer pipeline live-live using the build-c…

d307095

…oreclr-and-libraries-job template.

Update artifacts download paths and msbuild properties for installer …

fbdde3e

…build.

Mark cross-component files as native.

60ecad1

Remove FrameworkPackageName to remove error about missing $(Framework…

9ae4d08

…PackageName).versions.txt file that other shared frameworks also don't generate.

Restore GetFilesFromPackageResolve override.

cd636c8

jkoritzinsky added 2 commits December 11, 2019 14:27

Don't clean the unpack folder for the libraries download. It wipes ou…

77f3ec7

…t the CoreCLR download.

Make sure that we restore the WinRT targetting pack during the instal…

7257b1e

…ler build.

jkoritzinsky mentioned this pull request Dec 12, 2019

Removing "netcoreapp" and "netfx" TargetGroup #457

Merged

jkoritzinsky and others added 4 commits December 11, 2019 17:05

Add libraries package directory to RestoreSources for the case where …

cdf9a29

…the LibrariesConfiguration is different from the installer build configuration.

Don't pass OfficialBuildId unless it is an official build. We want to…

9b62aff

… be consistent with the libraries build so our package versions match.

Give default value for OfficialBuildArg variable

35bfb81

Ensure installer tests can see the libraries-generated packages. Fix …

ef5144c

…typo in TargetPath for CoreCLR assets (runtime instead of runtimes). Fix paths for CoreCLR cross-target files as well as crossgen to place them in the correct directories.

Add back FrameworkPackageName to get the partial platform manifest to…

836c9b7

… generate.

ViktorHofer approved these changes Dec 13, 2019

View reviewed changes

jkoritzinsky merged commit 2458930 into dotnet:master Dec 13, 2019

jkoritzinsky deleted the live-live-builds branch December 13, 2019 18:31

jkoritzinsky mentioned this pull request Dec 13, 2019

The dotnet/runtime repo is now live-live! #839

Closed

dagood mentioned this pull request Jun 29, 2020

Update baseline infra with specifics for deployment-tools repo dotnet/deployment-tools#1

Merged

ghost locked as resolved and limited conversation to collaborators Dec 11, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Local live-live builds #494

Local live-live builds #494

jkoritzinsky commented Dec 3, 2019 •

edited

Loading

jkoritzinsky commented Dec 11, 2019

jkoritzinsky commented Dec 11, 2019

jkoritzinsky commented Dec 11, 2019

jkoritzinsky commented Dec 12, 2019

ViktorHofer commented Dec 12, 2019

jkoritzinsky commented Dec 12, 2019

ViktorHofer commented Dec 13, 2019 •

edited

Loading

jkoritzinsky commented Dec 13, 2019

ViktorHofer commented Dec 13, 2019

jkoritzinsky commented Dec 13, 2019

Local live-live builds #494

Local live-live builds #494

Conversation

jkoritzinsky commented Dec 3, 2019 • edited Loading

jkoritzinsky commented Dec 11, 2019

jkoritzinsky commented Dec 11, 2019

jkoritzinsky commented Dec 11, 2019

jkoritzinsky commented Dec 12, 2019

ViktorHofer commented Dec 12, 2019

jkoritzinsky commented Dec 12, 2019

ViktorHofer commented Dec 13, 2019 • edited Loading

jkoritzinsky commented Dec 13, 2019

ViktorHofer commented Dec 13, 2019

jkoritzinsky commented Dec 13, 2019

jkoritzinsky commented Dec 3, 2019 •

edited

Loading

ViktorHofer commented Dec 13, 2019 •

edited

Loading