Add CentOS 8 CI build #1539

omajid · 2020-03-25T13:59:18Z

We have had some issues building in RHEL 8, so lets set up a CentOS 8 CI system to try and catch those issues earlier.

omajid · 2020-03-25T14:53:36Z

The builds are failing trying to build coreclr because they are trying to find a libunwind-devel which doesn't exist.

I have locally been doing this:

sed -i 's|-DCLR_CMAKE_USE_SYSTEM_LIBUNWIND=TRUE|-DCLR_CMAKE_USE_SYSTEM_LIBUNWIND=FALSE|' repos/coreclr.common.props

before running ./build.sh.

What would be a good way to integrate this into CI without hardcoding? We want CentOS 7 CI to use the system libuwind, but the CentOS 8 CI to use the copy bundled with coreclr.

dagood · 2020-03-25T14:57:09Z

@crummel, looks like this is resolving #1536. What do you do to generate the libunwindless tarballs (I assume we'd want to do the same here)?

.vsts.pipelines/builds/matrix.yml

omajid · 2020-03-25T21:43:36Z

The current CentOS 8 build failure is:

/tb/tarball_573224/packages/source-built/coreclr-tools/ilasm: error while loading shared libraries: libunwind-x86_64.so.8: cannot open shared object file: No such file or directory

That's the very bug we were running into earlier. The CI seems to be working as expected 😄

dir.props

dagood · 2020-03-26T15:36:10Z

@crummel @dseefeld @adaggarwal can you help out with the CI error? Does this have to do with downloading source-built artifacts from some other platform, rather than using what was built during the production build on the same job? Or the mixups around whether that tar.gz contains portable or nonportable CoreCLR tools?

omajid · 2020-03-26T15:37:23Z

This looks like the same "Permission Denied" issue that @adaggarwal was running into yesterday.

adaggarwal · 2020-03-26T15:37:54Z

@crummel @dseefeld @adaggarwal can you help out with the CI error? Does this have to do with downloading source-built artifacts from some other platform, rather than using what was built during the production build on the same job? Or the mixups around whether that tar.gz contains portable or nonportable CoreCLR tools?

I see it's the permissions with core-clr tools. Looking.

crummel · 2020-03-26T15:38:20Z

I think this might just be because it was started before Aditya's fix got merged, if AzDo picks up the new base here I think it should work:

crummel · 2020-03-26T15:38:24Z

/azp run

azure-pipelines · 2020-03-26T15:38:35Z

Azure Pipelines successfully started running 1 pipeline(s).

dagood · 2020-03-26T15:39:38Z

I think this might just be because it was started before Aditya's fix got merged, if AzDo picks up the new base here I think it should work:

It probably won't--GitHub does the merge so there needs to be something on the PR side that triggers a remerge.

adaggarwal · 2020-03-26T15:39:41Z

I think this might just be because it was started before Aditya's fix got merged, if AzDo picks up the new base here I think it should work:

Right. Because The other PR uses the same artifacts - https://github.com/dotnet/source-build/runs/534431675 and we had green on centos71 offline.

.vsts.pipelines/jobs/ci-linux.yml

We have had some issues building in RHEL 8, so lets set up a CentOS 8 CI system to try and catch those issues earlier.

adaggarwal · 2020-03-26T16:16:15Z

I had a brief discussion with Davis about the problem. What could be happening here is - in the last PR that changed the version number for artifacts, I also mutated the 101.1 coreclr bits to have execute permissions.

The change was in-place, it did show up in #1535 but this PR's CI used a cached copy(with no x permissions for coreclr tools). This is because we use cdn(azureedge) system which doesn't handle mutations as efficiently. I'll push a change with a new copy of artifacts which has execute permissions for coreclr bits in 101.2 instead of 101.1 so that the CI doesn't confuses.

adaggarwal · 2020-03-27T15:48:46Z

Known timeout issues, merging.

* Add CentOS 8 CI build We have had some issues building in RHEL 8, so lets set up a CentOS 8 CI system to try and catch those issues earlier. * Pin to artifacts version with executable coreclr bits Co-authored-by: Aditya Aggarwal <aditya.aggarwal@microsoft.com> Conflicts: eng/Versions.props repos/coreclr.common.props

* Update arcade and standard shas * Add runtime repo and remove coreclr, corefx and core-setup * Update standard to build with new arcade version * Update version of roslyn to work with updated Arcade * Update patches to enable runtime repo to build * Update aspnet-extensions commit and get it building * Add runtime portable build * Add OverrideTargetRid for Linux * Retain portable logs when cleaning up after portable build * Include project.assets.json files in files to be saved from portable build * Update version of source-build archive * Remove old corefx, coreclr and core-setup projects and update known-good * Add patch for ilasm round trip issue * Add patch for ilasm round trip issue * Update commit shas for repos in known good for preview 2 * Reconcile patches for core-sdk * Reconcile patches for linkere * Reconcile patches for fsharp repo * Reconcile patches for nuget-client repo * Reconcile patches for roslyn repo * Reconcile patches for runtime repo * Reconcile patches for sdk repo * Remove dependency flow sources -- update Arcade version * Update commit sha for standard * Update Arcade and SDK versions * Reconcile patches for arcade repo * Remove ArcadeConfigurationOverride * Fixup output placement for standard repo * Remove unneccessary roslyn patch * Correct invalid merge * Remove linker reflection heuristics patch per comment from sbomer on #777 * Update PackagesOutput for linker * Add patch to set nopgooptimize in coreclr build * Update coreclr build arguments * Only build through coreclr for now * Only build through runtime * Add CentOS 8 CI build (#1539) * Add CentOS 8 CI build We have had some issues building in RHEL 8, so lets set up a CentOS 8 CI system to try and catch those issues earlier. * Pin to artifacts version with executable coreclr bits Co-authored-by: Aditya Aggarwal <aditya.aggarwal@microsoft.com> Conflicts: eng/Versions.props repos/coreclr.common.props * Update docker images * Update version of xliff tasks * Include 1 addtional arcade patch * Add patch to exclude Helix tests from source-build in arcade * Add reference to PlatformAbstractions * Skip prebuilt check for now * Update version of previously source-built archive * Update online prebuilt baseline * Disable smoke-tests and unit tests until core-sdk build works * Don't remove placeholder pdb files * Add back temporary ildasm workaround until AspNetCore builds * Update baseline with OSX only prebuilts * Update offline baseline * Turn off processing of smoke-test-packages * Update debian & ubuntu images and turn off leak detection * Update Centos8 docker image to one that includes updated cmake * Another try at updating the centos8 docker image * Update to latest version of Private.SourceBuild.ReferencePackages * Remove detection of and de-assemble/re-assemble of read only ref pkgs With the completion of #1497, the need to detect and remove these read only ref packages has been removed. All packages that we have identified as read-only in the past have been included in source-build-reference-packages. If any new ones show up, they will show up as prebuilts and can be added to the source-build-reference-packages repo. * Revert "Skip prebuilt check for now" This reverts commit 5766e7e. * Update offline baseline * Add comments & issue links to disabled items * Updates based on PR review comments * Update to document temporary smoke-test removal * Remove standard project Going forward the repo will only be used for servicing. * Update runtime dependency name Co-authored-by: Omair Majid <omajid@redhat.com>

* Add CentOS 8 CI build We have had some issues building in RHEL 8, so lets set up a CentOS 8 CI system to try and catch those issues earlier. * Pin to artifacts version with executable coreclr bits Co-authored-by: Aditya Aggarwal <aditya.aggarwal@microsoft.com> Conflicts: eng/Versions.props repos/coreclr.common.props

* Add runtime repo and remove coreclr, corefx and core-setup * Update arcade and standard shas * Update patches to enable runtime repo to build * Add patch for ilasm round trip issue * Add patch for ilasm round trip issue * Reconcile patches for runtime repo * Reconcile patches for arcade repo * Add patch to set nopgooptimize in coreclr build * Add CentOS 8 CI build (#1539) * Add CentOS 8 CI build We have had some issues building in RHEL 8, so lets set up a CentOS 8 CI system to try and catch those issues earlier. * Pin to artifacts version with executable coreclr bits Co-authored-by: Aditya Aggarwal <aditya.aggarwal@microsoft.com> Conflicts: eng/Versions.props repos/coreclr.common.props * Only build through coreclr for now * Include 1 addtional arcade patch * Add patch to exclude Helix tests from source-build in arcade * Add reference to PlatformAbstractions * Skip prebuilt check for now * Disable smoke-tests and unit tests until core-sdk build works * Update Centos8 docker image to one that includes updated cmake * Another try at updating the centos8 docker image * Revert "Skip prebuilt check for now" This reverts commit 5766e7e. * Add comments & issue links to disabled items * Update to document temporary smoke-test removal * Update build script to take additional params for all builds Add patch to not build harvestPackages in source-build Remove PVP from dependencies.props and eng/Versions.props * Don't default version of MsbuildTaskMicrosoftCodeAnalysisCSharpVersion in arcade * Update versions of arcade and SBRP * Add new sdk overrides * Re-enable installer build * Update version of source-built artifacts * Build crossgen 2 and update arcade patches * Update dotnet and source-built versions * don't override NuGetVersion in arcade * Temporary fix to save arcade specific prebuilts that are ref-pkgs * Update prebuilt baselines * Remove extra files from rebase * Remove extra centos8 job * Update portable runtime settings from PR#1530 in the release/3.1 branch * Updating allowed prebuilts * Patch to conditionally update CodeAnalysisCSharpVersion * Turn off prebuilt check. Issue to re-enable #1577 * Update portable build properties * One prebuilt update for osx * Updates from PR review * Remove extraneous space * One more extraneous space Co-authored-by: Omair Majid <omajid@redhat.com>

dagood reviewed Mar 25, 2020

View reviewed changes

.vsts.pipelines/builds/matrix.yml Show resolved Hide resolved

omajid force-pushed the centos-8-testing branch from 4736c47 to 48af3a7 Compare March 25, 2020 19:35

omajid force-pushed the centos-8-testing branch from 48af3a7 to 0ca8d19 Compare March 26, 2020 13:35

dagood reviewed Mar 26, 2020

View reviewed changes

dir.props Outdated Show resolved Hide resolved

omajid force-pushed the centos-8-testing branch from 0ca8d19 to 169b6db Compare March 26, 2020 15:40

dagood reviewed Mar 26, 2020

View reviewed changes

.vsts.pipelines/jobs/ci-linux.yml Show resolved Hide resolved

Add CentOS 8 CI build

b7fec0d

We have had some issues building in RHEL 8, so lets set up a CentOS 8 CI system to try and catch those issues earlier.

omajid force-pushed the centos-8-testing branch from 169b6db to b7fec0d Compare March 26, 2020 15:45

Pin to artifacts version with executable coreclr bits

e11b8c4

dseefeld mentioned this pull request Mar 26, 2020

Can we test a strictly-portable scenarios in CI? #1536

Open

dagood approved these changes Mar 26, 2020

View reviewed changes

crummel approved these changes Mar 26, 2020

View reviewed changes

adaggarwal merged commit ed88943 into dotnet:release/3.1 Mar 27, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add CentOS 8 CI build #1539

Add CentOS 8 CI build #1539

omajid commented Mar 25, 2020

omajid commented Mar 25, 2020 •

edited

Loading

dagood commented Mar 25, 2020

omajid commented Mar 25, 2020

dagood commented Mar 26, 2020

omajid commented Mar 26, 2020

adaggarwal commented Mar 26, 2020

crummel commented Mar 26, 2020

crummel commented Mar 26, 2020

azure-pipelines bot commented Mar 26, 2020

dagood commented Mar 26, 2020

adaggarwal commented Mar 26, 2020

adaggarwal commented Mar 26, 2020 •

edited

Loading

adaggarwal commented Mar 27, 2020

Add CentOS 8 CI build #1539

Add CentOS 8 CI build #1539

Conversation

omajid commented Mar 25, 2020

omajid commented Mar 25, 2020 • edited Loading

dagood commented Mar 25, 2020

omajid commented Mar 25, 2020

dagood commented Mar 26, 2020

omajid commented Mar 26, 2020

adaggarwal commented Mar 26, 2020

crummel commented Mar 26, 2020

crummel commented Mar 26, 2020

azure-pipelines bot commented Mar 26, 2020

dagood commented Mar 26, 2020

adaggarwal commented Mar 26, 2020

adaggarwal commented Mar 26, 2020 • edited Loading

adaggarwal commented Mar 27, 2020

omajid commented Mar 25, 2020 •

edited

Loading

adaggarwal commented Mar 26, 2020 •

edited

Loading