Build in Azure Pipelines with cached vcpkg artifacts #114

lukka · 2019-09-18T08:06:59Z

Please note that acceptance of community PRs will be delayed while we are
bringing our test and CI systems online. For more information, see the
README.md.

Description

I hope this PR is either useful and could be an inspiration on how to set up a CI system.
This PR is not meant to be merged as is, but it is created in order to show how it is possible to run a build using CMake and Ninja with cached vcpkg artifacts on Hosted Build Agents.

Hosted agents do not have the required MSVC 14.23, hence the build also install that toolset on the fly (considerably increasing build time).

The build time on Hosted agents with cached vcpkg artifacts is about:

restoring cache 30 seconds;
building 4 minutes.
building vcpkg artifacts 15 minutes;

See here for all build samples.

No tests are run.

Also, this PR shows how to explicitly define which vcpkg version is to be used by using a git submodule.

Checklist:

I understand README.md.
The STL builds and test harnesses have passed (must be manually verified
by an STL maintainer before CI is online, leave this unchecked for initial
submission).

BillyONeal · 2019-09-19T22:44:26Z

We are absolutely interested in setting up an Azure Pipelines (or maybe GitHub Actions) script for this repo, but we are looking into solutions for hosting the build agents. The default provided "hosted" agents are little dual core boxes, and once we get our tests in here you'd likely be waiting 8+ hours for an answer.

Unfortunately, with a self hosted build agent system, there don't appear to be good options to handle untrusted pull request submissions. We may just put Azure DevOps in "manually initiated" mode but we'd like to give people more immediate feedback (before review) if possible.

I've asked the Pipelines team if they have any ideas but haven't gotten reasonable answers back yet.

Thanks for your contribution!

sylveon · 2019-09-19T22:47:33Z

Just a thought, there could be a thing running on the hosted agents which only builds the tests to do basic validity checking and then use a GitHub bot to trigger the full test suite internally.

lukka · 2019-09-19T22:57:25Z

@BillyONeal self hosted agents are the way to go indeed (the Microsoft hosted agents do not have the prerequisites to build anyway).
What about expendable VMs created off an existing image on Azure Compute? or the biggest size not powerful enough? If those are not and you need some extremely powerful physical machine, all they need is to not be part of corporate network and then you save any security concern.

BillyONeal · 2019-09-21T02:53:26Z

Just a thought, there could be a thing running on the hosted agents which only builds the tests to do basic validity checking and then use a GitHub bot to trigger the full test suite internally.

That wouldn't prevent a PR from sending malicious code to our internal test infrastructure that isn't prepared to deal with that.

What about expendable VMs created off an existing image on Azure Compute?

That would work fine but I don't think there's a great way to get Azure DevOps to recycle the machine upon completion (which is necessary if we want to build and run arbitrary internet submitted code).

If those are not and you need some extremely powerful physical machine, all they need is to not be part of corporate network and then you save any security concern.

Just because the machine isn't on the corporate network doesn't mean there aren't serious problems with letting an attacker gain persistence. For example, using Azure bandwidth to attack other Internet entities, or mining bitcoins on our hardware.

Azure DevOps does have an "OK to test check" thing ( https://docs.microsoft.com/en-us/azure/devops/pipelines/repos/github?view=azure-devops&tabs=yaml#comment-triggers ) we'll probably end up using for now.

sylveon · 2019-09-21T03:15:04Z

I meant a manually triggered GitHub bot, basically like the comment triggers of DevOps.

lukka · 2019-09-23T07:16:33Z

Running the build agent on a docker image stored in Azure Container Registry (and that contains all pre-reqs for building) could be a good fit, as long it is going to be cached locally on the host to start up quickly. Have not tried this scenario yet though.

.gitmodules

BillyONeal · 2019-09-25T00:14:53Z

@StephanTLavavej @CaseyCarter @barcharcraz Are you OK with the submodule to get vcpkg here? Since we're using a self-hosted agent I can just put that in the agent setup instructions but given that we already have a contribution that does that setup for us which will work if/when we can use the hosted system, I'm reluctant to throw that work away.

ado/run_build.yml

barcharcraz · 2019-09-25T00:16:23Z

vcpkg_windows.txt

@@ -0,0 +1,5 @@
+boost-build:x86-windows


why do we need boost build explicitly listed?

in my experience, when running 'vcpkg install boost-math:x86-windows', it failed asking me to install 'boost-build:x86-windows' as well. And doing this failed with same error: 'vcpkg install boost-build:x86-windows boost-math:x86-windows'. I had to run first the installation of boost-build:x86-windows, and then run the installation of boost-math:x86-windows separately to succeed.

this might be a issue worth reporting in vcpkg, but I'm not sure.

barcharcraz · 2019-09-25T00:17:32Z

@StephanTLavavej Stephan T. Lavavej FTE @CaseyCarter Casey Carter FTE @barcharcraz Charlie Barto FTE Are you OK with the submodule to get vcpkg here? Since we're using a self-hosted agent I can just put that in the agent setup instructions but given that we already have a contribution that does that setup for us which will work if/when we can use the hosted system, I'm reluctant to throw that work away.

Also, I'm OK with it as long as we know it can work with the new caching feature.

VCPKG itself is pretty small, it's the built artifacts that take a long time.

BillyONeal · 2019-09-25T00:22:19Z

Hmmm this PR was created without the ability for us to edit it, so I can't try it out to see if it'll work on our self-hosted agent. Do you mind if I recreate this in my fork?

Thanks for your contribution!

BillyONeal · 2019-09-25T00:32:22Z

/AzurePipelines run STL

azure-pipelines · 2019-09-25T00:32:28Z

No pipelines are associated with this pull request.

lukka · 2019-09-25T00:55:33Z

i can also do the requested changes here, just let me know what works better for you @BillyONeal

CaseyCarter · 2019-09-25T01:14:15Z

@StephanTLavavej Stephan T. Lavavej FTE @CaseyCarter Casey Carter FTE @barcharcraz Charlie Barto FTE Are you OK with the submodule to get vcpkg here? Since we're using a self-hosted agent I can just put that in the agent setup instructions but given that we already have a contribution that does that setup for us which will work if/when we can use the hosted system, I'm reluctant to throw that work away.

Also, I'm OK with it as long as we know it can work with the new caching feature.

VCPKG itself is pretty small, it's the built artifacts that take a long time.

Despite my earlier private comments that we should avoid adding a submodule, I'm now in favor of whatever gets us up and running the quickest. We can refine things later if we find improvements, but for now we shouldn't let the perfect be the enemy of the good.

BillyONeal · 2019-09-25T02:33:40Z

i can also do the requested changes here, just let me know what works better for you @BillyONeal

If you merge with master (please preserve the clang-format validation in at least one configuration) it should have the bits working. Note that it won't actually attempt to run the changes automatically without pre-review from us.

BillyONeal · 2019-09-25T02:35:06Z

i can also do the requested changes here, just let me know what works better for you @BillyONeal

If you merge with origin/master it should have the bits that indicate to run using the machine on my desk. It will require pre-review from us though... since it's a self-hosted agent and we don't have a mechanism to automatically recycle it we need pre-review to ensure folks aren't submitting malware or similar.

BillyONeal · 2019-09-25T02:51:53Z

/AzurePipelines run

azure-pipelines · 2019-09-25T02:52:03Z

Azure Pipelines failed to run 1 pipeline(s).

BillyONeal · 2019-09-25T02:53:46Z

That agent was set up like this: https://github.com/microsoft/STL/wiki/Setting-up-an-Azure-Pipelines-Agent

lukka · 2019-09-25T02:53:48Z

@BillyONeal the tasks needs to be installed from the marketplace

azure-devops/install_msvc_preview.ps1

azure-devops/run_build.yml

CaseyCarter · 2019-09-25T21:16:41Z

azure-devops/run_build.yml

@@ -0,0 +1,51 @@
+parameters:
+  targetPlatform: 'x64'


Why have a parameters block here when the parameters come from azure-pipelines.yml?

Here is declaration and default value of accepted params. The actual value is indeed in a-p.yml

Rephrasing: do we need to have this declaration here, or can we leave it out so this template will fail when it's not used correctly? It would be a shame if someone accidentally introduced a typo in the parameter names (e.g., targetP1atform) in azure-pipelines.yml and we consequently build x64 four times for every PR.

I suppose if it MUST be here we could give it an invalid value like potato to ensure it will fail very loudly.

azure-devops/run_build.yml

azure-devops/install_msvc_preview.ps1

BillyONeal · 2019-09-25T22:30:01Z

/azp run

azure-pipelines · 2019-09-25T22:30:13Z

Azure Pipelines successfully started running 1 pipeline(s).

.gitmodules

StephanTLavavej · 2019-09-25T22:46:22Z

azure-devops/install_msvc_preview.ps1

+    [string]$bootstrapperExe = Join-Path ${env:Temp} ([System.IO.Path]::GetRandomFileName() + ".exe")
+    Invoke-WebRequest -Uri $VSBootstrapperURL -OutFile $bootstrapperExe
+
+    $Arguments = ('/c', $bootstrapperExe, $WorkLoads, '--quiet', '--norestart', '--wait', '--nocache' )


Nitpick: Why is there a space before the final )?

StephanTLavavej · 2019-09-25T22:47:03Z

azure-devops/install_msvc_preview.ps1

+    }
+    else
+    {
+      Write-Host -Object "Non zero exit code returned by the installation process : $exitCode."


Nitpick: Nonzero or Non-zero.

StephanTLavavej · 2019-09-25T22:48:01Z

azure-devops/install_msvc_preview.ps1

+}
+
+# Invalidate the standard installation of VS on the hosted agent.
+Move-Item "C:/Program Files (x86)/Microsoft Visual Studio/2019/Enterprise/" "C:/Program Files (x86)/Microsoft Visual Studio/2019/nouse/" -Verbose


This seems highly invasive. Is it desirable? Could we simply fail if we detect an existing installation of VS?

This script is only run on the "hosted" machines which are blown away after each build, so invasive changes are fine/normal.

then you would always fail on the hosted one.

StephanTLavavej · 2019-09-25T22:49:23Z

azure-devops/install_msvc_preview.ps1

+              '--add Microsoft.VisualStudio.Component.VC.Tools.x86.x64 ' + `
+              '--add Microsoft.VisualStudio.Component.VC.Tools.ARM64 ' + `
+              '--add Microsoft.VisualStudio.Component.VC.Tools.ARM ' + `
+              '--add Microsoft.VisualStudio.Component.Windows10SDK.18362 '


Is there any way to request the latest Win10 SDK, instead of encoding this version number here that might need to be updated?

Not as far as I am aware.

StephanTLavavej · 2019-09-25T22:53:03Z

azure-devops/run_build.yml

+      inputs:
+        cmakeListsTxtPath: 'CMakeSettings.json'
+        useVcpkgToolchainFile: true
+        configurationRegexFilter: '.*${{ parameters.targetPlatform }}.*'


Is this doing a whole-string or a substring match? If the latter, the .* at the front and back are unnecessary.

I'm not positive which it is but I also think it doesn't matter. (And we should merge this as-is)

StephanTLavavej · 2019-09-25T22:53:40Z

azure-pipelines.yml

+# Copyright (c) Microsoft Corporation.
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+# Build STL targeting x64, arm64, x86, arm.


Our usual order is x86, x64, arm, arm64. Is this order significant in some way?

StephanTLavavej · 2019-09-25T22:57:34Z

vcpkg_windows.txt

+boost-math:x64-windows
+boost-math:x86-windows
+boost-math:arm-windows
+boost-math:arm64-windows


Does this file have to be in the root of the repo? Would azure-devops be a better location? It's mentioned by azure-devops/run_build.yml.

BillyONeal · 2019-09-26T00:02:47Z

/azp run

azure-pipelines · 2019-09-26T00:02:58Z

Azure Pipelines successfully started running 1 pipeline(s).

StephanTLavavej · 2019-09-26T02:04:49Z

azure-pipelines.yml

+  parameters:
+    targetPlatform: x64
+
+


Is there a reason for there to be two newlines here?

StephanTLavavej · 2019-09-26T02:05:12Z

vcpkg_windows.txt

@@ -0,0 +1,5 @@
+boost-build:x86-windows
+boost-math:x64-windows
+boost-math:x86-windows


I observe that this mentions x64 before x86.

BillyONeal · 2019-09-26T02:08:19Z

/azp run

azure-pipelines · 2019-09-26T02:08:29Z

Azure Pipelines successfully started running 1 pipeline(s).

This is provided by azure-pipelines.yml. See: microsoft#114 (comment)

lukka requested review from barcharcraz, BillyONeal, CaseyCarter and StephanTLavavej as code owners September 18, 2019 08:06

BillyONeal self-assigned this Sep 19, 2019

barcharcraz reviewed Sep 25, 2019

View reviewed changes

.gitmodules Show resolved Hide resolved

barcharcraz reviewed Sep 25, 2019

View reviewed changes

ado/run_build.yml Outdated Show resolved Hide resolved

barcharcraz reviewed Sep 25, 2019

View reviewed changes

add azure-pipeline for x64, arm64, x86, arm

24f4cc6

lukka force-pushed the master branch from 584c744 to 24f4cc6 Compare September 25, 2019 02:49

lukka requested a review from a team as a code owner September 25, 2019 02:49

BillyONeal mentioned this pull request Sep 25, 2019

P0439R0 enum class memory_order #124

Merged

5 tasks

CaseyCarter suggested changes Sep 25, 2019

View reviewed changes

BillyONeal added 2 commits September 25, 2019 15:28

Add newlines and license blocks, and remove Spectre libs.

44d4d31

Deindent install preview comment

ab4391f

BillyONeal requested a review from CaseyCarter September 25, 2019 22:29

barcharcraz approved these changes Sep 25, 2019

View reviewed changes

StephanTLavavej reviewed Sep 25, 2019

View reviewed changes

CaseyCarter approved these changes Sep 25, 2019

View reviewed changes

lukka and others added 2 commits September 25, 2019 16:42

cosmetic fixes

3a5d74e

Reorder platforms and add license to .gitmodules.

87912e8

BillyONeal requested a review from StephanTLavavej September 26, 2019 00:09

StephanTLavavej approved these changes Sep 26, 2019

View reviewed changes

Stephan's nitpicks

f777520

StephanTLavavej approved these changes Sep 26, 2019

View reviewed changes

BillyONeal merged commit d7c7691 into microsoft:master Sep 26, 2019

BillyONeal mentioned this pull request Sep 26, 2019

Move cgmanifest.json #138

Merged

7 tasks

BillyONeal added a commit to BillyONeal/cpprestsdk that referenced this pull request Sep 27, 2019

Attempt templates and other tricks I learned from microsoft/STL#114

ad30215

BillyONeal added a commit to BillyONeal/cpprestsdk that referenced this pull request Sep 27, 2019

Attempt templates and other tricks I learned from microsoft/STL#114

a1affaa

BillyONeal added a commit to BillyONeal/cpprestsdk that referenced this pull request Sep 27, 2019

Attempt templates and other tricks I learned from microsoft/STL#114

5787f4a

BillyONeal added a commit to BillyONeal/cpprestsdk that referenced this pull request Sep 27, 2019

Attempt templates and other tricks I learned from microsoft/STL#114

0e547bc

BillyONeal added a commit to BillyONeal/cpprestsdk that referenced this pull request Sep 27, 2019

Attempt templates and other tricks I learned from microsoft/STL#114

ada27eb

BillyONeal added a commit to BillyONeal/cpprestsdk that referenced this pull request Sep 27, 2019

Attempt templates and other tricks I learned from microsoft/STL#114

8f0a93e

StephanTLavavej added a commit to StephanTLavavej/STL that referenced this pull request Nov 5, 2019

Attempt to remove default targetPlatform from run_build.yml.

a178840

This is provided by azure-pipelines.yml. See: microsoft#114 (comment)

StephanTLavavej mentioned this pull request Apr 29, 2020

README.md: Use the vcpkg submodule instead of separately cloning #761

Closed

Build in Azure Pipelines with cached vcpkg artifacts #114

Build in Azure Pipelines with cached vcpkg artifacts #114

Conversation

lukka commented Sep 18, 2019 • edited by BillyONeal Loading

Description

Checklist:

BillyONeal commented Sep 19, 2019

sylveon commented Sep 19, 2019 • edited Loading

lukka commented Sep 19, 2019

BillyONeal commented Sep 21, 2019

sylveon commented Sep 21, 2019

lukka commented Sep 23, 2019

BillyONeal commented Sep 25, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

barcharcraz commented Sep 25, 2019

BillyONeal commented Sep 25, 2019

BillyONeal commented Sep 25, 2019

azure-pipelines bot commented Sep 25, 2019

lukka commented Sep 25, 2019

CaseyCarter commented Sep 25, 2019

BillyONeal commented Sep 25, 2019

BillyONeal commented Sep 25, 2019

BillyONeal commented Sep 25, 2019

azure-pipelines bot commented Sep 25, 2019

BillyONeal commented Sep 25, 2019

lukka commented Sep 25, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

CaseyCarter Sep 25, 2019 • edited Loading

Choose a reason for hiding this comment

CaseyCarter Sep 25, 2019 • edited Loading

Choose a reason for hiding this comment

BillyONeal commented Sep 25, 2019

azure-pipelines bot commented Sep 25, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

BillyONeal Sep 26, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

BillyONeal commented Sep 26, 2019

azure-pipelines bot commented Sep 26, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

BillyONeal commented Sep 26, 2019

azure-pipelines bot commented Sep 26, 2019

lukka commented Sep 18, 2019 •

edited by BillyONeal

Loading

sylveon commented Sep 19, 2019 •

edited

Loading

CaseyCarter Sep 25, 2019 •

edited

Loading

CaseyCarter Sep 25, 2019 •

edited

Loading

BillyONeal Sep 26, 2019 •

edited

Loading