Local Live live builds #55

jkoritzinsky · 2019-11-15T02:17:12Z

This PR implements local live-live builds. The eng/liveBuilds.targets MSBuild file is the central location for live dependency file discovery. There are various MSBuild property hooks in there to enable overriding the location of the various consumed live components.

Work completed in this PR:

Libraries consumes:
- A live copy of the CoreCLR build's shared framework and System.Private.CoreLib output.
Installer consumes:
- A live copy of the CoreCLR build's shared framework and System.Private.CoreLib output
- A live copy of the Libraries shared framework ref and implementation assemblies.
- A live copy of Microsoft.NETCore.Platforms and Microsoft.NETCore.Targets
CoreCLR managed test build consumes:
- A live copy of the Libraries shared framework and OOB ref assemblies.
CoreCLR Core_Root generation:
- A live copy of the Libraries shared framework and OOB implementation assemblies.
Customization points for each "unit" of output that is consumed between subset categories.
Subset category dependencies will automatically rebuild only when their subset category is included in the current build. Otherwise the build will let you use out-of-date live builds.

This PR depends on:

Disable tests on CoreCLR using new attribute #118 for moving CoreFX test disabling on Checked/stress CoreCLR to attributes and off CoreFX.issues.rsp.
Work for getting live-live CI up and running so we can get this running in CI..

Future work for another PR once we are off the critical path:

Refactor out target overrides in src/installer/pkg/projects/netcoreapp/src/localnetcoreapp.override.targets so we can send back a patch to Arcade and remove localnetcoreapp.override.targets.
Extend eng/liveBuilds.targets to have a target for consuming a live build of the hosting layer.
Consume a live build of the hosting layer in the CoreFX test build instead of consuming packages.

src/libraries/restore/runtime/runtime.depproj

eng/liveBuilds.targets

dagood · 2019-11-15T17:18:10Z

eng/liveBuilds.targets

+<Project>
+
+  <PropertyGroup>
+    <HostArch>$([System.Runtime.InteropServices.RuntimeInformation]::ProcessArchitecture.ToString().ToLowerInvariant)</HostArch>


Do these static properties have to live in a targets file?

The static properties don't have to. I just decided to keep them localized to where they're used. The path properties should be here since they depend on the configuration properties, which can be overridden in a subtree (libraries overrides them since there ConfigurationGroup represents what Configuration represents everywhere else).

I would prefer if they were in a target so they're truly localized. But without seeing an MSBuild style guide anywhere ever, I'm not going to claim consistency here or anything like that. No objection to merging.

dagood · 2019-11-15T17:29:26Z

Add an extensibility point to Arcade to force the required build ordering.

The way I do this in Core-Setup is with MSBuild tasks, nothing special. This is the same mechanism project references use. (When running an MSBuild task against the same project with the same properties from multiple projects, MSBuild only runs it once with safe parallelism and all that. This lets you build up the build graph with as fine-grained parallelism as you can stomach.)

To make one chunk of the build depend on another, I used <MSBuild Projects="@(CorehostProjectToBuild)" Targets="Build" />. That's why eng/Subsets.props is imported by the installer build.

dagood · 2019-11-15T17:41:02Z

I'm going to be flying tomorrow and very busy this weekend, so if you or others could help drive this closer to the finish line while I'm gone that would be fantastic!

Unfortunate timing... my team's boxing up stuff to be moved to another part of the building over the weekend so I won't have access to my main hardware. I'll try to make some progress on these two if I can manage with laptops:

Consume the live build of the files from Microsoft.NETCore.Platforms and Microsoft.NETCore.Targets in src/installer/pkg/projects/netcoreapp/src/netcoreapp.depproj
Add an extensibility point to Arcade to force the required build ordering.

("Refactor out changes to GetCrossgenToolPaths" doesn't sound necessary for the initial dotnet/runtime opening, I can help with that in the future.)

…dependencies are in the building subset/subset category.

…live-builds

jkotas · 2019-11-18T09:52:52Z

One more TODO: Update documentation (e.g. https://github.com/dotnet/runtime/blob/master/docs/coreclr/building/testing-with-corefx.md)

…ndencies.

…ding a new BinplaceConfiguration. Update test_dependencies in the CoreCLR tree to use that to restore OOB libraries.

…he reference is only needed at runtime and Core_Root already has System.Drawing.Common).

…live world, we'll run this test configuration by pointing the CoreFX test host build step at the artifacts from a checked CoreCLR build (possible in both local and CI builds).

jkoritzinsky · 2019-11-18T23:09:12Z

Update though today:

CoreCLR tests now populate Core_Root with a live libraries build including out-of-box libraries.
Live-live dependencies builds are done when their outputs are needed when their in-place output path (where the build is dropped in artifacts) is used as their location, their included in the currently building subset, and the dependency build hasn't been explicitly disabled (the CoreCLR test build disables it for now until that build is Arcade-ified).
The CoreCLR-specific infra for running CoreFX tests against a checked CoreCLR has been removed since we can now run the libraries tests against a live drop of checked CoreCLR. The CoreCLR-specific exludes in CoreFX.rsp will be covered via new attributes on the CoreFX tests that @safern has added to Arcade and will apply to the tests once we get the Arcade update.
Remove a dependency on an old package that contained the UCRT for the test build and replace it by copying the UCRT directly when targeting Windows.

…lean build.

jkoritzinsky · 2019-11-19T22:43:47Z

This PR now has support for live builds of coreclr and libraries being consumed in both product and test builds for everything except the dependency of the legacy Microsoft.NETCore.App package on Microsoft.NETCore.Platforms on Windows. There are some issues I need to work out on non-Windows and I need to figure out how to get the version of the live Microsoft.NETCore.Platforms package for building the legacy Microsoft.NETCore.App package.

Any additional work past these pieces that is not documentation (ie consuming the live hosting layer in the CoreFX tests) will wait until a future PR since that is not on the critical path.

…ckaged Microsoft.NETCore.Platforms package.

…ntime into live-live-builds

eng/liveBuilds.targets

ViktorHofer · 2019-11-20T09:17:30Z

eng/liveBuilds.targets

+    <PropertyGroup>
+      <CoreCLRArtifactsPath>$([MSBuild]::NormalizeDirectory('$(CoreCLRArtifactsPath)'))</CoreCLRArtifactsPath>
+    </PropertyGroup>
+    <PropertyGroup>


nit: can we merge these two property groups instead?

eng/liveBuilds.targets

ViktorHofer · 2019-11-20T09:26:39Z

src/coreclr/tests/src/Directory.Build.targets

@@ -221,9 +234,18 @@
    <ProjectAssetsFile></ProjectAssetsFile>
  </PropertyGroup>

-  <PropertyGroup Condition="'$(ReferenceSystemPrivateCoreLib)' == 'true' and '$(UsingMicrosoftNETSdk)' == 'true'">
+  <PropertyGroup Condition="'$(UsingMicrosoftNETSdk)' == 'true'">


do we actually need this remaining condition? Usually we just disable the property entirely without checking if we use an SDK which honors it.

I think I can remove the remaining part of the condition

ViktorHofer · 2019-11-20T09:34:37Z

src/libraries/Directory.Build.targets

      <!-- enable trimming for any runtime project that's part of the shared framework and hasn't already set ILLinkTrimAssembly -->
      <SetProperties Condition="'$(BinPlaceRuntime)' == 'true' and '$(ILLinkTrimAssembly)' == ''">ILLinkTrimAssembly=true</SetProperties>
    </BinPlaceConfiguration>

+    <!-- Binplace the OOB libraries built in the netcoreapp build. -->


Why don't you instead just use the artifacts/bin/ref/netcoreapp and artifacts/bin/runtime/netcoreapp folders instead of binplacing new ones?

Yeah I think to build coreclr tests we can use ref/netcoreapp and runtime/netcoreapp. That way we only binplace the inbox assemblies which are needed by the installer to produce the shared framework package.

From what I'm seeing, the artifacts/bin/runtime/netcoreapp folder has more than just the implementation assemblies. It looks like it has the restored runtime from the CoreCLR build as well.

We can change that if necessary. We don't use those artifacts there.

Yeah. We just need to think about the ReferenceFromRuntime scenario as we special case it for S.P.CoreLib to take the assets from runtime.depproj output, however for all others we get them from runtime/netcoreapp

That's a good point. I think we should treat CoreLib as part of libraries.

src/libraries/restore/runtime/runtime.depproj

ViktorHofer · 2019-11-20T09:42:42Z

Very impressive! Good work.

…live-builds

…test build.

jkoritzinsky · 2019-12-03T22:27:10Z

Superseded by #494

) Do not generate the hml per default. This is a feature for xamarin-macios and has to be explicitly enabled. fixes: dotnet#55 Co-authored-by: Premek Vysoky <premek.vysoky@microsoft.com>

jkoritzinsky added 2 commits November 14, 2019 15:59

Convert libraries and installer to use a live CoreCLR.

9c85652

Installer uses a live libraries build for DLLs.

88f21a9

jkoritzinsky added the area-Infrastructure label Nov 15, 2019

jkoritzinsky added this to the 5.0 milestone Nov 15, 2019

jkoritzinsky requested review from jashook, ViktorHofer and dagood November 15, 2019 02:17

Remove newline in path.

6164fa8

jkoritzinsky commented Nov 15, 2019

View reviewed changes

src/libraries/restore/runtime/runtime.depproj Outdated Show resolved Hide resolved

safern reviewed Nov 15, 2019

View reviewed changes

eng/liveBuilds.targets Outdated Show resolved Hide resolved

eng/liveBuilds.targets Outdated Show resolved Hide resolved

dagood reviewed Nov 15, 2019

View reviewed changes

jkoritzinsky added 4 commits November 17, 2019 12:01

Enable live-live builds to build required live dependencies when the …

94a436b

…dependencies are in the building subset/subset category.

Merge branch 'master' of https://github.com/dotnet/runtime into live-…

3b57b00

…live-builds

Update path to coreclr output.

ae23413

Fix typo in coreclr.proj.

02f2afa

jkotas mentioned this pull request Nov 18, 2019

Update the Coreclr docs #83

Merged

ViktorHofer mentioned this pull request Nov 18, 2019

Switch to live built dependencies #92

Closed

3 tasks

jkoritzinsky added 9 commits November 18, 2019 08:41

Remove last usage of CoreCLRPDBOverridePath.

42f2954

Update path of binplaced runtime to include OS and Arch.

106ea54

Fix typo in coreclr/tests/src/Common/Directory.Build.targets.

39ab7ff

Restore CoreFX shared framework from live build for CoreCLR test depe…

23f8963

…ndencies.

Remove dependency on Microsoft.NETCore.CoreCLR.TestDependencies package.

c2b930c

Enable referencing live builds of the OOB libraries by temporarily ad…

f73ad73

…ding a new BinplaceConfiguration. Update test_dependencies in the CoreCLR tree to use that to restore OOB libraries.

Remove reference to System.Drawing.Common from NETClientPrimitives (t…

eb0e522

…he reference is only needed at runtime and Core_Root already has System.Drawing.Common).

Remove package-based checked CoreCLR + CoreFX test runs. In the live-…

68e746b

…live world, we'll run this test configuration by pointing the CoreFX test host build step at the artifacts from a checked CoreCLR build (possible in both local and CI builds).

Only copy the UCRT on Windows.

cba76e8

jkoritzinsky added 2 commits November 19, 2019 14:21

Fix problems in installer build in live-live that were found from a c…

0344a53

…lean build.

Merge branch 'master' into live-live-builds

aa4b349

jkoritzinsky added 3 commits November 19, 2019 14:47

Remove unused dependency on Microsoft.NETCore.Targets.

314c4a0

Fix Microsoft.NETCore.App -> Microsoft.NETCore.Platforms dependency.

f640bd9

Remove Microsoft.NETCore.Targets version from Versions.props.

80669fa

dagood mentioned this pull request Nov 19, 2019

Set installer.tasks.csproj to Platform AnyCPU for arch-agnostic output path #135

Merged

jkoritzinsky added 2 commits November 19, 2019 15:11

Remove unused deps.csproj file that was the last dependency on the pa…

cea542d

…ckaged Microsoft.NETCore.Platforms package.

Merge branch 'live-live-builds' of https://github.com/jkoritzinsky/ru…

01f5130

…ntime into live-live-builds

ViktorHofer reviewed Nov 20, 2019

View reviewed changes

eng/liveBuilds.targets Outdated Show resolved Hide resolved

ViktorHofer reviewed Nov 20, 2019

View reviewed changes

eng/liveBuilds.targets Outdated Show resolved Hide resolved

ViktorHofer reviewed Nov 20, 2019

View reviewed changes

src/libraries/restore/runtime/runtime.depproj Show resolved Hide resolved

ViktorHofer reviewed Nov 20, 2019

View reviewed changes

src/libraries/restore/runtime/runtime.depproj Outdated Show resolved Hide resolved

jkoritzinsky added 3 commits November 20, 2019 09:28

Fix OSGroup calculation for dependent projects off-Windows.

425d034

Fix construction of Microsoft.NETCore.App.Internal package.

364c621

Remove unneeded CoreRun copying.

002e664

jkoritzinsky changed the title ~~WIP: Local Live live builds~~ Local Live live builds Nov 21, 2019

jkoritzinsky added 2 commits November 21, 2019 14:42

Merge branch 'master' of https://github.com/dotnet/runtime into live-…

2956eef

…live-builds

Remove optional consumption of CoreCLR transport packages in CoreCLR …

ad29f1b

…test build.

jkoritzinsky mentioned this pull request Nov 22, 2019

Remove optional consumption of CoreCLR transport packages in CoreCLR test build. #223

Merged

ViktorHofer mentioned this pull request Dec 2, 2019

Add root build definition which builds the entire repo #98

Closed

jkoritzinsky mentioned this pull request Dec 3, 2019

Local live-live builds #494

Merged

jkoritzinsky closed this Dec 3, 2019

ghost locked as resolved and limited conversation to collaborators Dec 11, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Local Live live builds #55

Local Live live builds #55

jkoritzinsky commented Nov 15, 2019 •

edited

Loading

dagood Nov 15, 2019

jkoritzinsky Nov 18, 2019

dagood Nov 18, 2019

dagood commented Nov 15, 2019

dagood commented Nov 15, 2019

jkotas commented Nov 18, 2019

jkoritzinsky commented Nov 18, 2019

jkoritzinsky commented Nov 19, 2019

ViktorHofer Nov 20, 2019 •

edited

Loading

ViktorHofer Nov 20, 2019

jkoritzinsky Nov 20, 2019

ViktorHofer Nov 20, 2019

safern Nov 20, 2019

jkoritzinsky Nov 21, 2019

ViktorHofer Nov 25, 2019

safern Nov 25, 2019 •

edited

Loading

ViktorHofer Nov 25, 2019

ViktorHofer commented Nov 20, 2019

jkoritzinsky commented Dec 3, 2019

Local Live live builds #55

Local Live live builds #55

Conversation

jkoritzinsky commented Nov 15, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dagood commented Nov 15, 2019

dagood commented Nov 15, 2019

jkotas commented Nov 18, 2019

jkoritzinsky commented Nov 18, 2019

jkoritzinsky commented Nov 19, 2019

ViktorHofer Nov 20, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

safern Nov 25, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ViktorHofer commented Nov 20, 2019

jkoritzinsky commented Dec 3, 2019

jkoritzinsky commented Nov 15, 2019 •

edited

Loading

ViktorHofer Nov 20, 2019 •

edited

Loading

safern Nov 25, 2019 •

edited

Loading