Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Build in Azure Pipelines with cached vcpkg artifacts #114

Merged
merged 16 commits into from
Sep 26, 2019
Merged
Show file tree
Hide file tree
Changes from 15 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions .gitmodules
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
# Copyright (c) Microsoft Corporation.
# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception

[submodule "vcpkg"]
StephanTLavavej marked this conversation as resolved.
Show resolved Hide resolved
path = vcpkg
url = https://github.com/microsoft/vcpkg.git
lukka marked this conversation as resolved.
Show resolved Hide resolved
fetchRecurseSubmodules = false
2 changes: 1 addition & 1 deletion azure-devops/enforce-clang-format.cmd
Original file line number Diff line number Diff line change
Expand Up @@ -3,4 +3,4 @@
@echo off
clang-format -style=file -i stl/inc/* stl/inc/cvt/* stl/inc/experimental/* stl/src/* 2>&1
echo If your build fails here, you need to format the following files with clang-format 8.0.1.
git status --porcelain 1>&2
git status --porcelain stl 1>&2
64 changes: 64 additions & 0 deletions azure-devops/install_msvc_preview.ps1
Original file line number Diff line number Diff line change
@@ -0,0 +1,64 @@
# Copyright (c) Microsoft Corporation.
# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
$ErrorActionPreference = "Stop"
StephanTLavavej marked this conversation as resolved.
Show resolved Hide resolved

Function InstallVS
{
# Microsoft hosted agents do not have the required MSVC Preview for building MSVC STL.
# This step installs MSVC Preview, only on Microsoft hosted agents.

Param(
[String]$WorkLoads,
[String]$Sku,
[String]$VSBootstrapperURL)
$exitCode = -1
try
{
Write-Host "Downloading bootstrapper ..."
[string]$bootstrapperExe = Join-Path ${env:Temp} ([System.IO.Path]::GetRandomFileName() + ".exe")
Invoke-WebRequest -Uri $VSBootstrapperURL -OutFile $bootstrapperExe

$Arguments = ('/c', $bootstrapperExe, $WorkLoads, '--quiet', '--norestart', '--wait', '--nocache')

Write-Host "Starting Install: $Arguments"
$process = Start-Process -FilePath cmd.exe -ArgumentList $Arguments -Wait -PassThru
$exitCode = $process.ExitCode

if ($exitCode -eq 0 -or $exitCode -eq 3010)
{
Write-Host -Object 'Installation successful!'
}
else
{
Write-Host -Object "Nonzero exit code returned by the installation process : $exitCode."
}
}
catch
{
Write-Host -Object "Failed to install Visual Studio!"
Write-Host -Object $_.Exception.Message
exit $exitCode
}

exit $exitCode
}

# Invalidate the standard installation of VS on the hosted agent.
Move-Item "C:/Program Files (x86)/Microsoft Visual Studio/2019/Enterprise/" "C:/Program Files (x86)/Microsoft Visual Studio/2019/nouse/" -Verbose
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems highly invasive. Is it desirable? Could we simply fail if we detect an existing installation of VS?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This script is only run on the "hosted" machines which are blown away after each build, so invasive changes are fine/normal.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

then you would always fail on the hosted one.


$WorkLoads = '--add Microsoft.VisualStudio.Component.VC.CLI.Support ' + `
'--add Microsoft.VisualStudio.Component.VC.Tools.x86.x64 ' + `
'--add Microsoft.VisualStudio.Component.VC.Tools.ARM64 ' + `
'--add Microsoft.VisualStudio.Component.VC.Tools.ARM ' + `
'--add Microsoft.VisualStudio.Component.Windows10SDK.18362 '
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there any way to request the latest Win10 SDK, instead of encoding this version number here that might need to be updated?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not as far as I am aware.


$ReleaseInPath = 'Preview'
$Sku = 'Enterprise'
$VSBootstrapperURL = 'https://aka.ms/vs/16/pre/vs_buildtools.exe'

$ErrorActionPreference = 'Stop'

# Install VS
$exitCode = InstallVS -WorkLoads $WorkLoads -Sku $Sku -VSBootstrapperURL $VSBootstrapperURL

exit $exitCode
53 changes: 53 additions & 0 deletions azure-devops/run_build.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
# Copyright (c) Microsoft Corporation.
# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
parameters:
StephanTLavavej marked this conversation as resolved.
Show resolved Hide resolved
targetPlatform: 'x64'
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why have a parameters block here when the parameters come from azure-pipelines.yml?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here is declaration and default value of accepted params. The actual value is indeed in a-p.yml

Copy link
Contributor

@CaseyCarter CaseyCarter Sep 25, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rephrasing: do we need to have this declaration here, or can we leave it out so this template will fail when it's not used correctly? It would be a shame if someone accidentally introduced a typo in the parameter names (e.g., targetP1atform) in azure-pipelines.yml and we consequently build x64 four times for every PR.

Copy link
Contributor

@CaseyCarter CaseyCarter Sep 25, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suppose if it MUST be here we could give it an invalid value like potato to ensure it will fail very loudly.


jobs:
- job: ${{ parameters.targetPlatform }}
pool:
name: STL

variables:
vcpkgLocation: '$(Build.SourcesDirectory)/vcpkg'
vcpkgResponseFile: $(Build.SourcesDirectory)/vcpkg_windows.txt
steps:
- checkout: self
lukka marked this conversation as resolved.
Show resolved Hide resolved
submodules: recursive
- task: BatchScript@1
BillyONeal marked this conversation as resolved.
Show resolved Hide resolved
displayName: 'Enforce clang-format'
condition: eq('${{ parameters.targetPlatform }}', 'x86')
inputs:
filename: 'azure-devops/enforce-clang-format.cmd'
failOnStandardError: true
- task: PowerShell@2
displayName: 'Install MSVC preview'
# on "Hosted" agents only:
condition: or(contains(variables['Agent.Name'], 'Azure Pipelines'), contains(variables['Agent.Name'], 'Hosted Agent'))
inputs:
targetType: filePath
filePath: $(Build.SourcesDirectory)/azure-devops/install_msvc_preview.ps1
- task: CacheBeta@0
displayName: Cache vcpkg
inputs:
key: $(vcpkgResponseFile) | $(Build.SourcesDirectory)/.git/modules/vcpkg/HEAD
CaseyCarter marked this conversation as resolved.
Show resolved Hide resolved
path: '$(vcpkgLocation)'
- task: run-vcpkg@0
displayName: 'Run vcpkg to install boost-build'
inputs:
vcpkgArguments: 'boost-build:x86-windows'
vcpkgDirectory: '$(vcpkgLocation)'
- task: run-vcpkg@0
displayName: 'Run vcpkg'
inputs:
vcpkgArguments: '@$(vcpkgResponseFile)'
vcpkgDirectory: '$(vcpkgLocation)'
vcpkgTriplet: ${{ parameters.targetPlatform }}-windows
- task: run-cmake@0
displayName: 'Run CMake with Ninja'
enabled: true
inputs:
cmakeListsTxtPath: 'CMakeSettings.json'
useVcpkgToolchainFile: true
configurationRegexFilter: '.*${{ parameters.targetPlatform }}.*'
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this doing a whole-string or a substring match? If the latter, the .* at the front and back are unnecessary.

Copy link
Member

@BillyONeal BillyONeal Sep 26, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not positive which it is but I also think it doesn't matter. (And we should merge this as-is)

buildDirectory: $(Build.ArtifactStagingDirectory)/${{ parameters.targetPlatform }}
32 changes: 22 additions & 10 deletions azure-pipelines.yml
Original file line number Diff line number Diff line change
@@ -1,10 +1,22 @@
# Copyright (c) Microsoft Corporation.
# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
pool: 'STL'

steps:
- task: BatchScript@1
inputs:
filename: 'azure-devops/enforce-clang-format.cmd'
failOnStandardError: true
displayName: 'Enforce clang-format'
# Copyright (c) Microsoft Corporation.
# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception

# Build STL targeting x86, x64, arm, arm64

jobs:
- template: azure-devops/run_build.yml
parameters:
targetPlatform: x86

- template: azure-devops/run_build.yml
parameters:
targetPlatform: x64


Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a reason for there to be two newlines here?

- template: azure-devops/run_build.yml
parameters:
targetPlatform: arm

- template: azure-devops/run_build.yml
parameters:
targetPlatform: arm64
1 change: 1 addition & 0 deletions vcpkg
Submodule vcpkg added at 8413b9
5 changes: 5 additions & 0 deletions vcpkg_windows.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
boost-build:x86-windows
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why do we need boost build explicitly listed?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

in my experience, when running 'vcpkg install boost-math:x86-windows', it failed asking me to install 'boost-build:x86-windows' as well. And doing this failed with same error: 'vcpkg install boost-build:x86-windows boost-math:x86-windows'. I had to run first the installation of boost-build:x86-windows, and then run the installation of boost-math:x86-windows separately to succeed.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this might be a issue worth reporting in vcpkg, but I'm not sure.

boost-math:x64-windows
boost-math:x86-windows
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I observe that this mentions x64 before x86.

boost-math:arm-windows
boost-math:arm64-windows
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this file have to be in the root of the repo? Would azure-devops be a better location? It's mentioned by azure-devops/run_build.yml.