From 69eff7e9fdf07afc9f6c29788466cb1e12cc5938 Mon Sep 17 00:00:00 2001 From: "Dustin L. Howett" Date: Fri, 11 Aug 2023 14:06:30 -0500 Subject: [PATCH] Rewrite the entire Azure DevOps build system (#15808) This pull request rewrites the entire Azure DevOps build system. The guiding principles behind this rewrite are: - No pipeline definitions should contain steps (or tasks) directly. - All jobs should be in template files. - Any set of steps that is reused across multiple jobs must be in template files. - All artifact names can be customized (via a property called `artifactStem` on all templates that produce or consume artifacts). - No compilation happens outside of the "Build" phase, to consolidate the production and indexing of PDBs. - **Building the project produces a `bin` directory.** That `bin` directory is therefore the primary currency of the build. Jobs will either produce or consume `bin` if they want to do anything with the build outputs. - All step and job templates are named with `step` or `job` _first_, which disambiguates them in the templates directory. - Most jobs can be run on different `pool`s, so that we can put expensive jobs on expensive build agents and cheap jobs on cheap build agents. Some jobs handle pool selection on their own, however. Our original build pipelines used the `VSBuild` task _all over the place._ This resulted in Terminal being built in myriad ways, different for every pipeline. There was an attempt at standardization early on, where `ci.yml` consumed jobs and steps templates... but when `release.yml` was added, all of that went out the window. The new pipelines are consistent and focus on a small, well-defined set of jobs: - `job-build-project` - This is the big one! - Takes a list of build configurations and platforms. - Produces an artifact named `build-PLATFORM-CONFIG` for the entire matrix of possibilities. - Optionally signs the output and produces a bill of materials. - Admittedly has a lot going on. - `job-build-package-wpf` - Takes a list of build configurations and platforms. - Consumes the `build-` artifact for every config/platform possibility, plus one for "Any CPU" (hardcoded; this is where the .NET code builds) - Produces one `wpf-nupkg-CONFIG` for each configuration, merging all platforms. - Optionally signs the output and produces a bill of materials. - `job-merge-msix-into-bundle` - Takes a list of build configurations and platforms. - Consumes the `build-` artifact for every config/platform - Produces one `appxbundle-CONFIG` for each configuration, merging all platforms for that config into one `msixbundle`. - Optionally signs the output and produces a bill of materials. - `job-package-conpty` - Takes a list of build configurations and platforms. - Consumes the `build-` artifact for every config/platform - Produces one `conpty-nupkg-CONFIG` for each configuration, merging all platforms. - Optionally signs the output and produces a bill of materials. - `job-test-project` - Takes **one** build config and **one** platform. - Consumes `build-PLATFORM-CONFIG` - Selects its own pools (hardcoded) because it knows about architectures and must choose the right agent arch. - Runs tests (directly on the build agent). - `job-run-pgo-tests` - Just like the above, but runs tests where `IsPgo` is `true` - Collects all of the PGO counts and publishes a `pgc-intermediates` artifact for that platform and configuration. - `job-pgo-merge-pgd` - Takes **one** build config and multiple platforms. - Consumes `build-$platform-CONFIG` for each platform. - Consumes `pgc-intermediates-$platform-CONFIG` for each platform. - Merges the `pgc` files into `pgd` files - Produces a new `pgd-` artifact. - `job-pgo-build-nuget-and-publish` - Consumes the `pgd-` artifact from above. - Packs it into a `nupkg` and publishes it. - `job-submit-windows-vpack` - Only expected to run against `Release`. - Consumes the `appxbundle-CONFIG` artifact. - Publishes it to a vpack for Windows to consume. - `job-check-code-format` - Does not use artifacts. Runs `clang-format`. - `job-index-github-codenav` - Does not use artifacts. Fuzz submission is broken due to changes in the `onefuzz` client. I have removed the compliance and security build because it is no longer supported. Finally, this pull request has some additional benefits: - I've expanded the PGO build phase to cover ARM64! - We can remove everything Helix-related except the WTT parser - We no longer depend on Helix submission or Helix pools - The WPF control's inner DLLs are now codesigned (#15404) - Symbols for the WPF control, both .NET and C++, are published alongside all other symbols. - The files we submit to ESRP for signing are batched up into a single step[^1] Closes #11874 Closes #11974 Closes #15404 [^1]: This will have to change if we want to sign the individual per-architecture `.appx` files before bundling so that they can be directly installed. --- build/Helix/AzurePipelinesHelperScripts.ps1 | 175 ---- build/Helix/EnsureMachineState.ps1 | 112 --- build/Helix/GenerateTestProjFile.ps1 | 336 -------- build/Helix/InstallTestAppDependencies.ps1 | 12 - build/Helix/OutputFailedTestQuery.ps1 | 8 - build/Helix/OutputSubResultsJsonFiles.ps1 | 32 - build/Helix/OutputTestResults.ps1 | 131 --- build/Helix/PrepareHelixPayload.ps1 | 62 -- build/Helix/ProcessHelixFiles.ps1 | 130 --- build/Helix/RunTestsInHelix.proj | 20 - build/Helix/UpdateUnreliableTests.ps1 | 136 --- build/Helix/global.json | 5 - build/Helix/packages.config | 9 - build/Helix/readme.md | 32 - build/Helix/runtests.cmd | 105 --- build/config/ESRPSigning_ConPTY.json | 51 -- build/config/ESRPSigning_Terminal.json | 72 -- build/config/esrp.build.batch.conpty.json | 47 + ...srp.build.batch.terminal_constituents.json | 65 ++ build/config/esrp.build.batch.wpf.json | 46 + build/config/esrp.build.batch.wpfdotnet.json | 47 + build/pgo/PGO.DB.proj | 3 +- build/pgo/Terminal.PGO.DB.nuspec | 1 + build/pgo/Terminal.PGO.props | 3 +- build/pipelines/ci.yml | 58 +- build/pipelines/daily-loc-submission.yml | 1 + build/pipelines/feature-flag-ci.yml | 21 +- build/pipelines/fuzz.yml | 80 +- build/pipelines/pgo.yml | 79 +- build/pipelines/release.yml | 804 +++--------------- .../templates-v2/job-build-package-wpf.yml | 134 +++ .../templates-v2/job-build-project.yml | 263 ++++++ .../templates-v2/job-check-code-format.yml | 15 + .../job-index-github-codenav.yml} | 10 +- .../job-merge-msix-into-bundle.yml | 133 +++ .../templates-v2/job-package-conpty.yml | 118 +++ .../job-pgo-build-nuget-and-publish.yml} | 54 +- .../templates-v2/job-pgo-merge-pgd.yml | 75 ++ .../templates-v2/job-publish-symbols.yml | 81 ++ .../templates-v2/job-run-pgo-tests.yml | 83 ++ .../templates-v2/job-submit-windows-vpack.yml | 70 ++ .../job-test-project.yml} | 30 +- .../steps-create-signing-config.yml | 35 + .../steps-download-bin-dir-artifact.yml | 24 + .../steps-ensure-nuget-version.yml | 5 + .../steps-fetch-and-prepare-localizations.yml | 27 + .../steps-restore-nuget.yml} | 6 +- .../templates/build-console-audit-job.yml | 34 - .../pipelines/templates/build-console-ci.yml | 31 - .../build-console-compliance-job.yml | 203 ----- .../templates/build-console-fuzzing.yml | 90 -- .../pipelines/templates/build-console-pgo.yml | 55 -- .../templates/build-console-steps.yml | 137 --- .../pipelines/templates/check-formatting.yml | 17 - .../templates/console-ci-helix-job.yml | 25 - .../templates/helix-createprojfile-steps.yml | 15 - .../helix-processtestresults-job.yml | 71 -- .../templates/helix-runtests-job.yml | 164 ---- .../pipelines/templates/pgo-merge-pgd-job.yml | 70 -- build/scripts/Run-Tests.ps1 | 5 +- .../WpfTerminalControl.csproj | 13 + src/common.nugetversions.props | 2 +- src/common.nugetversions.targets | 7 +- 63 files changed, 1637 insertions(+), 3148 deletions(-) delete mode 100644 build/Helix/AzurePipelinesHelperScripts.ps1 delete mode 100644 build/Helix/EnsureMachineState.ps1 delete mode 100644 build/Helix/GenerateTestProjFile.ps1 delete mode 100644 build/Helix/InstallTestAppDependencies.ps1 delete mode 100644 build/Helix/OutputFailedTestQuery.ps1 delete mode 100644 build/Helix/OutputSubResultsJsonFiles.ps1 delete mode 100644 build/Helix/OutputTestResults.ps1 delete mode 100644 build/Helix/PrepareHelixPayload.ps1 delete mode 100644 build/Helix/ProcessHelixFiles.ps1 delete mode 100644 build/Helix/RunTestsInHelix.proj delete mode 100644 build/Helix/UpdateUnreliableTests.ps1 delete mode 100644 build/Helix/global.json delete mode 100644 build/Helix/packages.config delete mode 100644 build/Helix/readme.md delete mode 100644 build/Helix/runtests.cmd delete mode 100644 build/config/ESRPSigning_ConPTY.json delete mode 100644 build/config/ESRPSigning_Terminal.json create mode 100644 build/config/esrp.build.batch.conpty.json create mode 100644 build/config/esrp.build.batch.terminal_constituents.json create mode 100644 build/config/esrp.build.batch.wpf.json create mode 100644 build/config/esrp.build.batch.wpfdotnet.json create mode 100644 build/pipelines/templates-v2/job-build-package-wpf.yml create mode 100644 build/pipelines/templates-v2/job-build-project.yml create mode 100644 build/pipelines/templates-v2/job-check-code-format.yml rename build/pipelines/{templates/codenav-indexer.yml => templates-v2/job-index-github-codenav.yml} (57%) create mode 100644 build/pipelines/templates-v2/job-merge-msix-into-bundle.yml create mode 100644 build/pipelines/templates-v2/job-package-conpty.yml rename build/pipelines/{templates/pgo-build-and-publish-nuget-job.yml => templates-v2/job-pgo-build-nuget-and-publish.yml} (57%) create mode 100644 build/pipelines/templates-v2/job-pgo-merge-pgd.yml create mode 100644 build/pipelines/templates-v2/job-publish-symbols.yml create mode 100644 build/pipelines/templates-v2/job-run-pgo-tests.yml create mode 100644 build/pipelines/templates-v2/job-submit-windows-vpack.yml rename build/pipelines/{templates/test-console-ci.yml => templates-v2/job-test-project.yml} (72%) create mode 100644 build/pipelines/templates-v2/steps-create-signing-config.yml create mode 100644 build/pipelines/templates-v2/steps-download-bin-dir-artifact.yml create mode 100644 build/pipelines/templates-v2/steps-ensure-nuget-version.yml create mode 100644 build/pipelines/templates-v2/steps-fetch-and-prepare-localizations.yml rename build/pipelines/{templates/restore-nuget-steps.yml => templates-v2/steps-restore-nuget.yml} (88%) delete mode 100644 build/pipelines/templates/build-console-audit-job.yml delete mode 100644 build/pipelines/templates/build-console-ci.yml delete mode 100644 build/pipelines/templates/build-console-compliance-job.yml delete mode 100644 build/pipelines/templates/build-console-fuzzing.yml delete mode 100644 build/pipelines/templates/build-console-pgo.yml delete mode 100644 build/pipelines/templates/build-console-steps.yml delete mode 100644 build/pipelines/templates/check-formatting.yml delete mode 100644 build/pipelines/templates/console-ci-helix-job.yml delete mode 100644 build/pipelines/templates/helix-createprojfile-steps.yml delete mode 100644 build/pipelines/templates/helix-processtestresults-job.yml delete mode 100644 build/pipelines/templates/helix-runtests-job.yml delete mode 100644 build/pipelines/templates/pgo-merge-pgd-job.yml diff --git a/build/Helix/AzurePipelinesHelperScripts.ps1 b/build/Helix/AzurePipelinesHelperScripts.ps1 deleted file mode 100644 index 8934a8548d1..00000000000 --- a/build/Helix/AzurePipelinesHelperScripts.ps1 +++ /dev/null @@ -1,175 +0,0 @@ -function GetAzureDevOpsBaseUri -{ - Param( - [string]$CollectionUri, - [string]$TeamProject - ) - - return $CollectionUri + $TeamProject -} - -function GetQueryTestRunsUri -{ - Param( - [string]$CollectionUri, - [string]$TeamProject, - [string]$BuildUri, - [switch]$IncludeRunDetails - ) - - if ($IncludeRunDetails) - { - $includeRunDetailsParameter = "&includeRunDetails=true" - } - else - { - $includeRunDetailsParameter = "" - } - - $baseUri = GetAzureDevOpsBaseUri -CollectionUri $CollectionUri -TeamProject $TeamProject - $queryUri = "$baseUri/_apis/test/runs?buildUri=$BuildUri$includeRunDetailsParameter&api-version=5.0" - return $queryUri -} - -function Get-HelixJobTypeFromTestRun -{ - Param ($testRun) - - $testRunSingleResultUri = "$($testRun.url)/results?`$top=1&`$skip=0&api-version=5.1" - $singleTestResult = Invoke-RestMethod -Uri $testRunSingleResultUri -Method Get -Headers $azureDevOpsRestApiHeaders - $count = $singleTestResult.value.Length - if($count -eq 0) - { - # If the count is 0, then results have not yet been reported for this run. - # We only care about completed runs with results, so it is ok to just return 'UNKNOWN' for this run. - return "UNKNOWN" - } - else - { - $info = ConvertFrom-Json $singleTestResult.value.comment - $helixJobId = $info.HelixJobId - $job = Invoke-RestMethodWithRetries "https://helix.dot.net/api/2019-06-17/jobs/${helixJobId}?access_token=${HelixAccessToken}" - return $job.Type - } -} - -function Append-HelixAccessTokenToUrl -{ - Param ([string]$url, [string]$token) - if($url.Contains("?")) - { - $url = "$($url)&access_token=$($token)" - } - else - { - $url = "$($url)?access_token=$($token)" - } - return $url -} - - -# The Helix Rest api is sometimes unreliable. So we call these apis with retry logic. -# Note: The Azure DevOps apis are stable and do not need to be called with this retry logic. -$helixApiRetries = 0 -$helixApiRetriesMax = 10 - -function Download-StringWithRetries -{ - Param ([string]$fileName, [string]$url) - - $result = "" - $done = $false - - while(!($done)) - { - try - { - Write-Host "Downloading $fileName" - $result = (New-Object System.Net.WebClient).DownloadString($url) - $done = $true - } - catch - { - Write-Host "Failed to download $fileName $($PSItem.Exception)" - - $helixApiRetries = $helixApiRetries + 1 - if($helixApiRetries -lt $helixApiRetriesMax) - { - Write-Host "Sleep and retry download of $fileName" - Start-Sleep 60 - } - else - { - throw "Failed to download $fileName" - } - } - } - - return $result -} - -function Invoke-RestMethodWithRetries -{ - Param ([string]$url,$Headers) - - $result = @() - $done = $false - - while(!($done)) - { - try - { - $result = Invoke-RestMethod -Uri $url -Method Get -Headers $Headers - $done = $true - } - catch - { - Write-Host "Failed to invoke Rest method $($PSItem.Exception)" - - $helixApiRetries = $helixApiRetries + 1 - if($helixApiRetries -lt $helixApiRetriesMax) - { - Write-Host "Sleep and retry invoke" - Start-Sleep 60 - } - else - { - throw "Failed to invoke Rest method" - } - } - } - - return $result -} - -function Download-FileWithRetries -{ - Param ([string]$fileurl, [string]$destination) - - $done = $false - - while(!($done)) - { - try - { - Write-Host "Downloading $destination" - $webClient.DownloadFile($fileurl, $destination) - $done = $true - } - catch - { - Write-Host "Failed to download $destination $($PSItem.Exception)" - - $helixApiRetries = $helixApiRetries + 1 - if($helixApiRetries -lt $helixApiRetriesMax) - { - Write-Host "Sleep and retry download of $destination" - Start-Sleep 60 - } - else - { - throw "Failed to download $destination" - } - } - } -} \ No newline at end of file diff --git a/build/Helix/EnsureMachineState.ps1 b/build/Helix/EnsureMachineState.ps1 deleted file mode 100644 index 6f26da63a56..00000000000 --- a/build/Helix/EnsureMachineState.ps1 +++ /dev/null @@ -1,112 +0,0 @@ -$scriptDirectory = $script:MyInvocation.MyCommand.Path | Split-Path -Parent - -# List all processes to aid debugging: -Write-Host "All processes running:" -Get-Process - -tasklist /svc - -# Add this test directory as an exclusion for Windows Defender -Write-Host "Add $scriptDirectory as Exclusion Path" -Add-MpPreference -ExclusionPath $scriptDirectory -Write-Host "Add $($env:HELIX_CORRELATION_PAYLOAD) as Exclusion Path" -Add-MpPreference -ExclusionPath $env:HELIX_CORRELATION_PAYLOAD -Get-MpPreference -Get-MpComputerStatus - - -# Minimize all windows: -$shell = New-Object -ComObject "Shell.Application" -$shell.minimizeall() - -# Kill any instances of Windows Security Alert: -$windowTitleToMatch = "*Windows Security Alert*" -$procs = Get-Process | Where {$_.MainWindowTitle -like "*Windows Security Alert*"} -foreach ($proc in $procs) -{ - Write-Host "Found process with '$windowTitleToMatch' title: $proc" - $proc.Kill(); -} - -# Kill processes by name that are known to interfere with our tests: -$processNamesToStop = @("Microsoft.Photos", "WinStore.App", "SkypeApp", "SkypeBackgroundHost", "OneDriveSetup", "OneDrive") -foreach($procName in $processNamesToStop) -{ - Write-Host "Attempting to kill $procName if it is running" - Stop-Process -ProcessName $procName -Verbose -ErrorAction Ignore -} -Write-Host "All processes running after attempting to kill unwanted processes:" -Get-Process - -tasklist /svc - -$platform = $env:testbuildplatform -if(!$platform) -{ - $platform = "x86" -} - -function UninstallApps { - Param([string[]]$appsToUninstall) - - foreach($pkgName in $appsToUninstall) - { - foreach($pkg in (Get-AppxPackage $pkgName).PackageFullName) - { - Write-Output "Removing: $pkg" - Remove-AppxPackage $pkg - } - } -} - -function UninstallTestApps { - Param([string[]]$appsToUninstall) - - foreach($pkgName in $appsToUninstall) - { - foreach($pkg in (Get-AppxPackage $pkgName).PackageFullName) - { - Write-Output "Removing: $pkg" - Remove-AppxPackage $pkg - } - - # Sometimes an app can get into a state where it is no longer returned by Get-AppxPackage, but it is still present - # which prevents other versions of the app from being installed. - # To handle this, we can directly call Remove-AppxPackage against the full name of the package. However, without - # Get-AppxPackage to find the PackageFullName, we just have to manually construct the name. - $packageFullName = "$($pkgName)_1.0.0.0_$($platform)__8wekyb3d8bbwe" - Write-Host "Removing $packageFullName if installed" - Remove-AppPackage $packageFullName -ErrorVariable appxerror -ErrorAction SilentlyContinue - if($appxerror) - { - foreach($error in $appxerror) - { - # In most cases, Remove-AppPackage will fail due to the package not being found. Don't treat this as an error. - if(!($error.Exception.Message -match "0x80073CF1")) - { - Write-Error $error - } - } - } - else - { - Write-Host "Successfully removed $packageFullName" - } - } -} - -Write-Host "Uninstall AppX packages that are known to cause issues with our tests" -UninstallApps("*Skype*", "*Windows.Photos*") - -Write-Host "Uninstall any of our test apps that may have been left over from previous test runs" -UninstallTestApps("NugetPackageTestApp", "NugetPackageTestAppCX", "IXMPTestApp", "MUXControlsTestApp") - -Write-Host "Uninstall MUX Framework package that may have been left over from previous test runs" -# We don't want to uninstall all versions of the MUX Framework package, as there may be other apps preinstalled on the system -# that depend on it. We only uninstall the Framework package that corresponds to the version of MUX that we are testing. -[xml]$versionData = (Get-Content "version.props") -$versionMajor = $versionData.GetElementsByTagName("MUXVersionMajor").'#text' -$versionMinor = $versionData.GetElementsByTagName("MUXVersionMinor").'#text' -UninstallApps("Microsoft.UI.Xaml.$versionMajor.$versionMinor") - -Get-Process \ No newline at end of file diff --git a/build/Helix/GenerateTestProjFile.ps1 b/build/Helix/GenerateTestProjFile.ps1 deleted file mode 100644 index 07a3550a79a..00000000000 --- a/build/Helix/GenerateTestProjFile.ps1 +++ /dev/null @@ -1,336 +0,0 @@ -[CmdLetBinding()] -Param( - [Parameter(Mandatory = $true)] - [string]$TestFile, - - [Parameter(Mandatory = $true)] - [string]$OutputProjFile, - - [Parameter(Mandatory = $true)] - [string]$JobTestSuiteName, - - [Parameter(Mandatory = $true)] - [string]$TaefPath, - - [string]$TaefQuery -) - -Class TestCollection -{ - [string]$Name - [string]$SetupMethodName - [string]$TeardownMethodName - [System.Collections.Generic.Dictionary[string, string]]$Properties - - TestCollection() - { - if ($this.GetType() -eq [TestCollection]) - { - throw "This class should never be instantiated directly; it should only be derived from." - } - } - - TestCollection([string]$name) - { - $this.Init($name) - } - - hidden Init([string]$name) - { - $this.Name = $name - $this.Properties = @{} - } -} - -Class Test : TestCollection -{ - Test([string]$name) - { - $this.Init($name) - } -} - -Class TestClass : TestCollection -{ - [System.Collections.Generic.List[Test]]$Tests - - TestClass([string]$name) - { - $this.Init($name) - $this.Tests = @{} - } -} - -Class TestModule : TestCollection -{ - [System.Collections.Generic.List[TestClass]]$TestClasses - - TestModule([string]$name) - { - $this.Init($name) - $this.TestClasses = @{} - } -} - -function Parse-TestInfo([string]$taefOutput) -{ - enum LineType - { - None - TestModule - TestClass - Test - Setup - Teardown - Property - } - - [string]$testModuleIndentation = " " - [string]$testClassIndentation = " " - [string]$testIndentation = " " - [string]$setupBeginning = "Setup: " - [string]$teardownBeginning = "Teardown: " - [string]$propertyBeginning = "Property[" - - function Get-LineType([string]$line) - { - if ($line.Contains($setupBeginning)) - { - return [LineType]::Setup; - } - elseif ($line.Contains($teardownBeginning)) - { - return [LineType]::Teardown; - } - elseif ($line.Contains($propertyBeginning)) - { - return [LineType]::Property; - } - elseif ($line.StartsWith($testModuleIndentation) -and -not $line.StartsWith("$testModuleIndentation ")) - { - return [LineType]::TestModule; - } - elseif ($line.StartsWith($testClassIndentation) -and -not $line.StartsWith("$testClassIndentation ")) - { - return [LineType]::TestClass; - } - elseif ($line.StartsWith($testIndentation) -and -not $line.StartsWith("$testIndentation ")) - { - return [LineType]::Test; - } - else - { - return [LineType]::None; - } - } - - [string[]]$lines = $taefOutput.Split(@([Environment]::NewLine, "`n"), [StringSplitOptions]::RemoveEmptyEntries) - [System.Collections.Generic.List[TestModule]]$testModules = @() - - [TestModule]$currentTestModule = $null - [TestClass]$currentTestClass = $null - [Test]$currentTest = $null - - [TestCollection]$lastTestCollection = $null - - foreach ($rawLine in $lines) - { - [LineType]$lineType = (Get-LineType $rawLine) - - # We don't need the whitespace around the line anymore, so we'll discard it to make things easier. - [string]$line = $rawLine.Trim() - - if ($lineType -eq [LineType]::TestModule) - { - if ($currentTest -ne $null -and $currentTestClass -ne $null) - { - $currentTestClass.Tests.Add($currentTest) - } - - if ($currentTestClass -ne $null -and $currentTestModule -ne $null) - { - $currentTestModule.TestClasses.Add($currentTestClass) - } - - if ($currentTestModule -ne $null) - { - $testModules.Add($currentTestModule) - } - - $currentTestModule = [TestModule]::new($line) - $currentTestClass = $null - $currentTest = $null - $lastTestCollection = $currentTestModule - } - elseif ($lineType -eq [LineType]::TestClass) - { - if ($currentTest -ne $null -and $currentTestClass -ne $null) - { - $currentTestClass.Tests.Add($currentTest) - } - - if ($currentTestClass -ne $null -and $currentTestModule -ne $null) - { - $currentTestModule.TestClasses.Add($currentTestClass) - } - - $currentTestClass = [TestClass]::new($line) - $currentTest = $null - $lastTestCollection = $currentTestClass - } - elseif ($lineType -eq [LineType]::Test) - { - if ($currentTest -ne $null -and $currentTestClass -ne $null) - { - $currentTestClass.Tests.Add($currentTest) - } - - $currentTest = [Test]::new($line) - $lastTestCollection = $currentTest - } - elseif ($lineType -eq [LineType]::Setup) - { - if ($lastTestCollection -ne $null) - { - $lastTestCollection.SetupMethodName = $line.Replace($setupBeginning, "") - } - } - elseif ($lineType -eq [LineType]::Teardown) - { - if ($lastTestCollection -ne $null) - { - $lastTestCollection.TeardownMethodName = $line.Replace($teardownBeginning, "") - } - } - elseif ($lineType -eq [LineType]::Property) - { - if ($lastTestCollection -ne $null) - { - foreach ($match in [Regex]::Matches($line, "Property\[(.*)\]\s+=\s+(.*)")) - { - [string]$propertyKey = $match.Groups[1].Value; - [string]$propertyValue = $match.Groups[2].Value; - $lastTestCollection.Properties.Add($propertyKey, $propertyValue); - } - } - } - } - - if ($currentTest -ne $null -and $currentTestClass -ne $null) - { - $currentTestClass.Tests.Add($currentTest) - } - - if ($currentTestClass -ne $null -and $currentTestModule -ne $null) - { - $currentTestModule.TestClasses.Add($currentTestClass) - } - - if ($currentTestModule -ne $null) - { - $testModules.Add($currentTestModule) - } - - return $testModules -} - -Write-Verbose "TaefQuery = $TaefQuery" - -$TaefSelectQuery = "" -$TaefQueryToAppend = "" -if($TaefQuery) -{ - $TaefSelectQuery = "/select:`"$TaefQuery`"" - $TaefQueryToAppend = " and $TaefQuery" -} -Write-Verbose "TaefSelectQuery = $TaefSelectQuery" - - -$taefExe = "$TaefPath\te.exe" -[string]$taefOutput = & "$taefExe" /listproperties $TaefSelectQuery $TestFile | Out-String - -[System.Collections.Generic.List[TestModule]]$testModules = (Parse-TestInfo $taefOutput) - -$projFileContent = @" - - -"@ - -foreach ($testModule in $testModules) -{ - foreach ($testClass in $testModules.TestClasses) - { - Write-Host "Generating Helix work item for test class $($testClass.Name)..." - [System.Collections.Generic.List[string]]$testSuiteNames = @() - - $testSuiteExists = $false - $suitelessTestExists = $false - - foreach ($test in $testClass.Tests) - { - # A test method inherits its 'TestSuite' property from its TestClass - if (!$test.Properties.ContainsKey("TestSuite") -and $testClass.Properties.ContainsKey("TestSuite")) - { - $test.Properties["TestSuite"] = $testClass.Properties["TestSuite"] - } - - if ($test.Properties.ContainsKey("TestSuite")) - { - [string]$testSuite = $test.Properties["TestSuite"] - - if (-not $testSuiteNames.Contains($testSuite)) - { - Write-Host " Found test suite $testSuite. Generating Helix work item for it as well." - $testSuiteNames.Add($testSuite) - } - - $testSuiteExists = $true - } - else - { - $suitelessTestExists = $true - } - } - - $testClassSelectPattern = "$($testClass.Name).*" - if($testClass.Name.Contains("::")) - { - $testClassSelectPattern = "$($testClass.Name)::*" - } - $testNameQuery= "(@Name='$testClassSelectPattern')" - - $workItemName = $testClass.Name - # Native tests use '::' as a separator, which is not valid for workItem names. - $workItemName = $workItemName -replace "::", "-" - - if ($suitelessTestExists) - { - $projFileContent += @" - - - 00:30:00 - call %HELIX_CORRELATION_PAYLOAD%\runtests.cmd /select:"(@Name='$($testClass.Name)*'$(if ($testSuiteExists) { "and not @TestSuite='*'" }))$($TaefQueryToAppend)" - -"@ - } - - foreach ($testSuiteName in $testSuiteNames) - { - $projFileContent += @" - - - 00:30:00 - call %HELIX_CORRELATION_PAYLOAD%\runtests.cmd /select:"(@Name='$($testClass.Name)*' and @TestSuite='$testSuiteName')$($TaefQueryToAppend)" - -"@ - } - } -} - -$projFileContent += @" - - - -"@ - -Set-Content $OutputProjFile $projFileContent -NoNewline -Encoding UTF8 \ No newline at end of file diff --git a/build/Helix/InstallTestAppDependencies.ps1 b/build/Helix/InstallTestAppDependencies.ps1 deleted file mode 100644 index da0a049d27d..00000000000 --- a/build/Helix/InstallTestAppDependencies.ps1 +++ /dev/null @@ -1,12 +0,0 @@ -# Displaying progress is unnecessary and is just distracting. -$ProgressPreference = "SilentlyContinue" - -$dependencyFiles = Get-ChildItem -Filter "*Microsoft.VCLibs.*.appx" - -foreach ($file in $dependencyFiles) -{ - Write-Host "Adding dependency $($file)..." - - Add-AppxPackage $file - -} \ No newline at end of file diff --git a/build/Helix/OutputFailedTestQuery.ps1 b/build/Helix/OutputFailedTestQuery.ps1 deleted file mode 100644 index 3ea7abf80ad..00000000000 --- a/build/Helix/OutputFailedTestQuery.ps1 +++ /dev/null @@ -1,8 +0,0 @@ -Param( - [Parameter(Mandatory = $true)] - [string]$WttInputPath -) - -Add-Type -Language CSharp -ReferencedAssemblies System.Xml,System.Xml.Linq,System.Runtime.Serialization,System.Runtime.Serialization.Json (Get-Content $PSScriptRoot\HelixTestHelpers.cs -Raw) - -[HelixTestHelpers.FailedTestDetector]::OutputFailedTestQuery($WttInputPath) \ No newline at end of file diff --git a/build/Helix/OutputSubResultsJsonFiles.ps1 b/build/Helix/OutputSubResultsJsonFiles.ps1 deleted file mode 100644 index 476502e9878..00000000000 --- a/build/Helix/OutputSubResultsJsonFiles.ps1 +++ /dev/null @@ -1,32 +0,0 @@ -Param( - [Parameter(Mandatory = $true)] - [string]$WttInputPath, - - [Parameter(Mandatory = $true)] - [string]$WttSingleRerunInputPath, - - [Parameter(Mandatory = $true)] - [string]$WttMultipleRerunInputPath, - - [Parameter(Mandatory = $true)] - [string]$TestNamePrefix -) - -# Ideally these would be passed as parameters to the script. However ps makes it difficult to deal with string literals containing '&', so we just -# read the values directly from the environment variables -$helixResultsContainerUri = $Env:HELIX_RESULTS_CONTAINER_URI -$helixResultsContainerRsas = $Env:HELIX_RESULTS_CONTAINER_RSAS - -Add-Type -Language CSharp -ReferencedAssemblies System.Xml,System.Xml.Linq,System.Runtime.Serialization,System.Runtime.Serialization.Json (Get-Content $PSScriptRoot\HelixTestHelpers.cs -Raw) - -$testResultParser = [HelixTestHelpers.TestResultParser]::new($TestNamePrefix, $helixResultsContainerUri, $helixResultsContainerRsas) -[System.Collections.Generic.Dictionary[string, string]]$subResultsJsonByMethodName = $testResultParser.GetSubResultsJsonByMethodName($WttInputPath, $WttSingleRerunInputPath, $WttMultipleRerunInputPath) - -$subResultsJsonDirectory = [System.IO.Path]::GetDirectoryName($WttInputPath) - -foreach ($methodName in $subResultsJsonByMethodName.Keys) -{ - $subResultsJson = $subResultsJsonByMethodName[$methodName] - $subResultsJsonPath = [System.IO.Path]::Combine($subResultsJsonDirectory, $methodName + "_subresults.json") - Out-File $subResultsJsonPath -Encoding utf8 -InputObject $subResultsJson -} \ No newline at end of file diff --git a/build/Helix/OutputTestResults.ps1 b/build/Helix/OutputTestResults.ps1 deleted file mode 100644 index a23b456afe8..00000000000 --- a/build/Helix/OutputTestResults.ps1 +++ /dev/null @@ -1,131 +0,0 @@ -Param( - [Parameter(Mandatory = $true)] - [int]$MinimumExpectedTestsExecutedCount, - - [string]$AccessToken = $env:SYSTEM_ACCESSTOKEN, - [string]$CollectionUri = $env:SYSTEM_COLLECTIONURI, - [string]$TeamProject = $env:SYSTEM_TEAMPROJECT, - [string]$BuildUri = $env:BUILD_BUILDURI, - [bool]$CheckJobAttempt -) - -$azureDevOpsRestApiHeaders = @{ - "Accept"="application/json" - "Authorization"="Basic $([System.Convert]::ToBase64String([System.Text.ASCIIEncoding]::ASCII.GetBytes(":$AccessToken")))" -} - -. "$PSScriptRoot/AzurePipelinesHelperScripts.ps1" - -Write-Host "Checking test results..." - -$queryUri = GetQueryTestRunsUri -CollectionUri $CollectionUri -TeamProject $TeamProject -BuildUri $BuildUri -IncludeRunDetails -Write-Host "queryUri = $queryUri" - -$testRuns = Invoke-RestMethodWithRetries $queryUri -Headers $azureDevOpsRestApiHeaders -[System.Collections.Generic.List[string]]$failingTests = @() -[System.Collections.Generic.List[string]]$unreliableTests = @() -[System.Collections.Generic.List[string]]$unexpectedResultTest = @() - -[System.Collections.Generic.List[string]]$namesOfProcessedTestRuns = @() -$totalTestsExecutedCount = 0 - -# We assume that we only have one testRun with a given name that we care about -# We only process the last testRun with a given name (based on completedDate) -# The name of a testRun is set to the Helix queue that it was run on (e.g. windows.10.amd64.client21h1.xaml) -# If we have multiple test runs on the same queue that we care about, we will need to re-visit this logic -foreach ($testRun in ($testRuns.value | Sort-Object -Property "completedDate" -Descending)) -{ - if ($CheckJobAttempt) - { - if ($namesOfProcessedTestRuns -contains $testRun.name) - { - Write-Host "Skipping test run '$($testRun.name)', since we have already processed a test run of that name." - continue - } - } - - Write-Host "Processing results from test run '$($testRun.name)'" - $namesOfProcessedTestRuns.Add($testRun.name) - - $totalTestsExecutedCount += $testRun.totalTests - - $testRunResultsUri = "$($testRun.url)/results?api-version=5.0" - $testResults = Invoke-RestMethodWithRetries "$($testRun.url)/results?api-version=5.0" -Headers $azureDevOpsRestApiHeaders - - foreach ($testResult in $testResults.value) - { - $shortTestCaseTitle = $testResult.testCaseTitle -replace "[a-zA-Z0-9]+.[a-zA-Z0-9]+.Windows.UI.Xaml.Tests.MUXControls.","" - - if ($testResult.outcome -eq "Failed") - { - if (-not $failingTests.Contains($shortTestCaseTitle)) - { - $failingTests.Add($shortTestCaseTitle) - } - } - elseif ($testResult.outcome -eq "Warning") - { - if (-not $unreliableTests.Contains($shortTestCaseTitle)) - { - $unreliableTests.Add($shortTestCaseTitle) - } - } - elseif ($testResult.outcome -ne "Passed") - { - # We should only see tests with result "Passed", "Failed" or "Warning" - if (-not $unexpectedResultTest.Contains($shortTestCaseTitle)) - { - $unexpectedResultTest.Add($shortTestCaseTitle) - } - } - } -} - -if ($unreliableTests.Count -gt 0) -{ - Write-Host @" -##vso[task.logissue type=warning;]Unreliable tests: -##vso[task.logissue type=warning;]$($unreliableTests -join "$([Environment]::NewLine)##vso[task.logissue type=warning;]") - -"@ -} - -if ($failingTests.Count -gt 0) -{ - Write-Host @" -##vso[task.logissue type=error;]Failing tests: -##vso[task.logissue type=error;]$($failingTests -join "$([Environment]::NewLine)##vso[task.logissue type=error;]") - -"@ -} - -if ($unexpectedResultTest.Count -gt 0) -{ - Write-Host @" -##vso[task.logissue type=error;]Tests with unexpected results: -##vso[task.logissue type=error;]$($unexpectedResultTest -join "$([Environment]::NewLine)##vso[task.logissue type=error;]") - -"@ -} - -if($totalTestsExecutedCount -lt $MinimumExpectedTestsExecutedCount) -{ - Write-Host "Expected at least $MinimumExpectedTestsExecutedCount tests to be executed." - Write-Host "Actual executed test count is: $totalTestsExecutedCount" - Write-Host "##vso[task.complete result=Failed;]" -} -elseif ($failingTests.Count -gt 0) -{ - Write-Host "At least one test failed." - Write-Host "##vso[task.complete result=Failed;]" -} -elseif ($unreliableTests.Count -gt 0) -{ - Write-Host "All tests eventually passed, but some initially failed." - Write-Host "##vso[task.complete result=Succeeded;]" -} -else -{ - Write-Host "All tests passed." - Write-Host "##vso[task.complete result=Succeeded;]" -} diff --git a/build/Helix/PrepareHelixPayload.ps1 b/build/Helix/PrepareHelixPayload.ps1 deleted file mode 100644 index 42ba9c6f77b..00000000000 --- a/build/Helix/PrepareHelixPayload.ps1 +++ /dev/null @@ -1,62 +0,0 @@ -[CmdLetBinding()] -Param( - [string]$Platform, - [string]$Configuration, - [string]$ArtifactName='drop' -) - -$payloadDir = "HelixPayload\$Configuration\$Platform" - -$repoDirectory = Join-Path (Split-Path -Parent $script:MyInvocation.MyCommand.Path) "..\..\" -$nugetPackagesDir = Join-Path (Split-Path -Parent $script:MyInvocation.MyCommand.Path) "packages" - -# Create the payload directory. Remove it if it already exists. -If(test-path $payloadDir) -{ - Remove-Item $payloadDir -Recurse -} -New-Item -ItemType Directory -Force -Path $payloadDir - -# Copy files from nuget packages -Copy-Item "$nugetPackagesDir\microsoft.windows.apps.test.1.0.181203002\lib\netcoreapp2.1\*.dll" $payloadDir -Copy-Item "$nugetPackagesDir\Microsoft.Taef.10.60.210621002\build\Binaries\$Platform\*" $payloadDir -Copy-Item "$nugetPackagesDir\Microsoft.Taef.10.60.210621002\build\Binaries\$Platform\NetFx4.5\*" $payloadDir -New-Item -ItemType Directory -Force -Path "$payloadDir\.NETCoreApp2.1\" -Copy-Item "$nugetPackagesDir\runtime.win-$Platform.microsoft.netcore.app.2.1.0\runtimes\win-$Platform\lib\netcoreapp2.1\*" "$payloadDir\.NETCoreApp2.1\" -Copy-Item "$nugetPackagesDir\runtime.win-$Platform.microsoft.netcore.app.2.1.0\runtimes\win-$Platform\native\*" "$payloadDir\.NETCoreApp2.1\" -New-Item -ItemType Directory -Force -Path "$payloadDir\content\" -Copy-Item "$nugetPackagesDir\Microsoft.Internal.Windows.Terminal.TestContent.1.0.1\content\*" "$payloadDir\content\" - -function Copy-If-Exists -{ - Param($source, $destinationDir) - - if (Test-Path $source) - { - Write-Host "Copy from '$source' to '$destinationDir'" - Copy-Item -Force $source $destinationDir - } - else - { - Write-Host "'$source' does not exist." - } -} - -# Copy files from the 'drop' artifact dir -Copy-Item "$repoDirectory\Artifacts\$ArtifactName\$Configuration\$Platform\Test\*" $payloadDir -Recurse - -# Copy files from the repo -New-Item -ItemType Directory -Force -Path "$payloadDir" -Copy-Item "build\helix\ConvertWttLogToXUnit.ps1" "$payloadDir" -Copy-Item "build\helix\OutputFailedTestQuery.ps1" "$payloadDir" -Copy-Item "build\helix\OutputSubResultsJsonFiles.ps1" "$payloadDir" -Copy-Item "build\helix\HelixTestHelpers.cs" "$payloadDir" -Copy-Item "build\helix\runtests.cmd" $payloadDir -Copy-Item "build\helix\InstallTestAppDependencies.ps1" "$payloadDir" -Copy-Item "build\Helix\EnsureMachineState.ps1" "$payloadDir" - -# Extract the unpackaged distribution of Windows Terminal to the payload directory, -# where it will create a subdirectory named terminal-0.0.1.0 -# This is referenced in TerminalApp.cs later as part of the test harness. -& tar -x -v -f "$repoDirectory\Artifacts\$ArtifactName\unpackaged\WindowsTerminalDev_0.0.1.0_x64.zip" -C "$payloadDir" -Copy-Item "res\fonts\*.ttf" "$payloadDir\terminal-0.0.1.0" diff --git a/build/Helix/ProcessHelixFiles.ps1 b/build/Helix/ProcessHelixFiles.ps1 deleted file mode 100644 index ae9f7582812..00000000000 --- a/build/Helix/ProcessHelixFiles.ps1 +++ /dev/null @@ -1,130 +0,0 @@ -Param( - [string]$AccessToken = $env:SYSTEM_ACCESSTOKEN, - [string]$HelixAccessToken = $env:HelixAccessToken, - [string]$CollectionUri = $env:SYSTEM_COLLECTIONURI, - [string]$TeamProject = $env:SYSTEM_TEAMPROJECT, - [string]$BuildUri = $env:BUILD_BUILDURI, - [string]$OutputFolder = "HelixOutput" -) - -$helixLinkFile = "$OutputFolder\LinksToHelixTestFiles.html" - - -function Generate-File-Links -{ - Param ([Array[]]$files,[string]$sectionName) - if($files.Count -gt 0) - { - Out-File -FilePath $helixLinkFile -Append -InputObject "
" - Out-File -FilePath $helixLinkFile -Append -InputObject "

$sectionName

" - Out-File -FilePath $helixLinkFile -Append -InputObject "" - Out-File -FilePath $helixLinkFile -Append -InputObject "
" - } -} - -function Append-HelixAccessTokenToUrl -{ - Param ([string]$url, [string]$token) - if($token) - { - if($url.Contains("?")) - { - $url = "$($url)&access_token=$($token)" - } - else - { - $url = "$($url)?access_token=$($token)" - } - } - return $url -} - -#Create output directory -New-Item $OutputFolder -ItemType Directory - -$azureDevOpsRestApiHeaders = @{ - "Accept"="application/json" - "Authorization"="Basic $([System.Convert]::ToBase64String([System.Text.ASCIIEncoding]::ASCII.GetBytes(":$AccessToken")))" -} - -. "$PSScriptRoot/AzurePipelinesHelperScripts.ps1" - -$queryUri = GetQueryTestRunsUri -CollectionUri $CollectionUri -TeamProject $TeamProject -BuildUri $BuildUri -IncludeRunDetails -Write-Host "queryUri = $queryUri" - -$testRuns = Invoke-RestMethodWithRetries $queryUri -Headers $azureDevOpsRestApiHeaders -$webClient = New-Object System.Net.WebClient -[System.Collections.Generic.List[string]]$workItems = @() - -foreach ($testRun in $testRuns.value) -{ - Write-Host "testRunUri = $testRun.url" - $testResults = Invoke-RestMethodWithRetries "$($testRun.url)/results?api-version=5.0" -Headers $azureDevOpsRestApiHeaders - $isTestRunNameShown = $false - - foreach ($testResult in $testResults.value) - { - $info = ConvertFrom-Json ([System.Web.HttpUtility]::HtmlDecode($testResult.comment)) - $helixJobId = $info.HelixJobId - $helixWorkItemName = $info.HelixWorkItemName - - $workItem = "$helixJobId-$helixWorkItemName" - - Write-Host "Helix Work Item = $workItem" - - if (-not $workItems.Contains($workItem)) - { - $workItems.Add($workItem) - $filesQueryUri = "https://helix.dot.net/api/2019-06-17/jobs/$helixJobId/workitems/$helixWorkItemName/files" - $filesQueryUri = Append-HelixAccessTokenToUrl $filesQueryUri $helixAccessToken - $files = Invoke-RestMethodWithRetries $filesQueryUri - - $screenShots = $files | where { $_.Name.EndsWith(".jpg") } - $dumps = $files | where { $_.Name.EndsWith(".dmp") } - $pgcFiles = $files | where { $_.Name.EndsWith(".pgc") } - if ($screenShots.Count + $dumps.Count + $pgcFiles.Count -gt 0) - { - if(-Not $isTestRunNameShown) - { - Out-File -FilePath $helixLinkFile -Append -InputObject "

$($testRun.name)

" - $isTestRunNameShown = $true - } - Out-File -FilePath $helixLinkFile -Append -InputObject "

$helixWorkItemName

" - Generate-File-Links $screenShots "Screenshots" - Generate-File-Links $dumps "CrashDumps" - Generate-File-Links $pgcFiles "PGC files" - $misc = $files | where { ($screenShots -NotContains $_) -And ($dumps -NotContains $_) -And ($visualTreeVerificationFiles -NotContains $_) -And ($pgcFiles -NotContains $_) } - Generate-File-Links $misc "Misc" - - foreach($pgcFile in $pgcFiles) - { - $flavorPath = $testResult.automatedTestName.Split('.')[0] - $archPath = $testResult.automatedTestName.Split('.')[1] - $fileName = $pgcFile.Name - $fullPath = "$OutputFolder\PGO\$flavorPath\$archPath" - $destination = "$fullPath\$fileName" - - Write-Host "Copying $($pgcFile.Name) to $destination" - - if (-Not (Test-Path $fullPath)) - { - New-Item $fullPath -ItemType Directory - } - - $link = $pgcFile.Link - - Write-Host "Downloading $link to $destination" - - $link = Append-HelixAccessTokenToUrl $link $HelixAccessToken - Download-FileWithRetries $link $destination - } - } - } - } -} diff --git a/build/Helix/RunTestsInHelix.proj b/build/Helix/RunTestsInHelix.proj deleted file mode 100644 index db00314ff06..00000000000 --- a/build/Helix/RunTestsInHelix.proj +++ /dev/null @@ -1,20 +0,0 @@ - - - pr/terminal/$(BUILD_SOURCEBRANCH)/ - true - true - true - $(HelixPreCommands);set testnameprefix=$(Configuration).$(Platform);set testbuildplatform=$(Platform);set rerunPassesRequiredToAvoidFailure=$(rerunPassesRequiredToAvoidFailure) - ..\..\bin\$(Platform)\$(Configuration)\ - - - - - - - - - - - - \ No newline at end of file diff --git a/build/Helix/UpdateUnreliableTests.ps1 b/build/Helix/UpdateUnreliableTests.ps1 deleted file mode 100644 index ecf4e8bf7a4..00000000000 --- a/build/Helix/UpdateUnreliableTests.ps1 +++ /dev/null @@ -1,136 +0,0 @@ -[CmdLetBinding()] -Param( - [Parameter(Mandatory = $true)] - [int]$RerunPassesRequiredToAvoidFailure, - - [string]$AccessToken = $env:SYSTEM_ACCESSTOKEN, - [string]$CollectionUri = $env:SYSTEM_COLLECTIONURI, - [string]$TeamProject = $env:SYSTEM_TEAMPROJECT, - [string]$BuildUri = $env:BUILD_BUILDURI -) - -. "$PSScriptRoot/AzurePipelinesHelperScripts.ps1" - - -$azureDevOpsRestApiHeaders = @{ - "Accept"="application/json" - "Authorization"="Basic $([System.Convert]::ToBase64String([System.Text.ASCIIEncoding]::ASCII.GetBytes(":$AccessToken")))" -} - -$queryUri = GetQueryTestRunsUri -CollectionUri $CollectionUri -TeamProject $TeamProject -BuildUri $BuildUri -Write-Host "queryUri = $queryUri" - -# To account for unreliable tests, we'll iterate through all of the tests associated with this build, check to see any tests that were unreliable -# (denoted by being marked as "skipped"), and if so, we'll instead mark those tests with a warning and enumerate all of the attempted runs -# with their pass/fail states as well as any relevant error messages for failed attempts. -$testRuns = Invoke-RestMethodWithRetries $queryUri -Headers $azureDevOpsRestApiHeaders - -$timesSeenByRunName = @{} - -foreach ($testRun in $testRuns.value) -{ - $testRunResultsUri = "$($testRun.url)/results?api-version=5.0" - - Write-Host "Marking test run `"$($testRun.name)`" as in progress so we can change its results to account for unreliable tests." - Invoke-RestMethod "$($testRun.url)?api-version=5.0" -Method Patch -Body (ConvertTo-Json @{ "state" = "InProgress" }) -Headers $azureDevOpsRestApiHeaders -ContentType "application/json" | Out-Null - - Write-Host "Retrieving test results..." - $testResults = Invoke-RestMethodWithRetries $testRunResultsUri -Headers $azureDevOpsRestApiHeaders - - foreach ($testResult in $testResults.value) - { - $testNeedsSubResultProcessing = $false - if ($testResult.outcome -eq "NotExecuted") - { - $testNeedsSubResultProcessing = $true - } - elseif($testResult.outcome -eq "Failed") - { - $testNeedsSubResultProcessing = $testResult.errorMessage -like "*_subresults.json*" - } - - if ($testNeedsSubResultProcessing) - { - Write-Host " Test $($testResult.testCaseTitle) was detected as unreliable. Updating..." - - # The errorMessage field contains a link to the JSON-encoded rerun result data. - $resultsJson = Download-StringWithRetries "Error results" $testResult.errorMessage - $rerunResults = ConvertFrom-Json $resultsJson - [System.Collections.Generic.List[System.Collections.Hashtable]]$rerunDataList = @() - $attemptCount = 0 - $passCount = 0 - $totalDuration = 0 - - foreach ($rerun in $rerunResults.results) - { - $rerunData = @{ - "displayName" = "Attempt #$($attemptCount + 1) - $($testResult.testCaseTitle)"; - "durationInMs" = $rerun.duration; - "outcome" = $rerun.outcome; - } - - if ($rerun.outcome -eq "Passed") - { - $passCount++ - } - - if ($attemptCount -gt 0) - { - $rerunData["sequenceId"] = $attemptCount - } - - Write-Host " Attempt #$($attemptCount + 1): $($rerun.outcome)" - - if ($rerun.outcome -ne "Passed") - { - $screenshots = "$($rerunResults.blobPrefix)/$($rerun.screenshots -join @" -$($rerunResults.blobSuffix) -$($rerunResults.blobPrefix) -"@)$($rerunResults.blobSuffix)" - - # We subtract 1 from the error index because we added 1 so we could use 0 - # as a default value not injected into the JSON in order to keep its size down. - # We did this because there's a maximum size enforced for the errorMessage parameter - # in the Azure DevOps REST API. - $fullErrorMessage = @" -Log: $($rerunResults.blobPrefix)/$($rerun.log)$($rerunResults.blobSuffix) - -Screenshots: -$screenshots - -Error log: -$($rerunResults.errors[$rerun.errorIndex - 1]) -"@ - - $rerunData["errorMessage"] = $fullErrorMessage - } - - $attemptCount++ - $totalDuration += $rerun.duration - $rerunDataList.Add($rerunData) - } - - $overallOutcome = "Warning" - - if ($attemptCount -eq 2) - { - Write-Host " Test $($testResult.testCaseTitle) passed on the immediate rerun, so we'll mark it as unreliable." - } - elseif ($passCount -gt $RerunPassesRequiredToAvoidFailure) - { - Write-Host " Test $($testResult.testCaseTitle) passed on $passCount of $attemptCount attempts, which is greater than or equal to the $RerunPassesRequiredToAvoidFailure passes required to avoid being marked as failed. Marking as unreliable." - } - else - { - Write-Host " Test $($testResult.testCaseTitle) passed on only $passCount of $attemptCount attempts, which is less than the $RerunPassesRequiredToAvoidFailure passes required to avoid being marked as failed. Marking as failed." - $overallOutcome = "Failed" - } - - $updateBody = ConvertTo-Json @(@{ "id" = $testResult.id; "outcome" = $overallOutcome; "errorMessage" = " "; "durationInMs" = $totalDuration; "subResults" = $rerunDataList; "resultGroupType" = "rerun" }) -Depth 5 - Invoke-RestMethod -Uri $testRunResultsUri -Method Patch -Headers $azureDevOpsRestApiHeaders -Body $updateBody -ContentType "application/json" | Out-Null - } - } - - Write-Host "Finished updates. Re-marking test run `"$($testRun.name)`" as completed." - Invoke-RestMethod -Uri "$($testRun.url)?api-version=5.0" -Method Patch -Body (ConvertTo-Json @{ "state" = "Completed" }) -Headers $azureDevOpsRestApiHeaders -ContentType "application/json" | Out-Null -} diff --git a/build/Helix/global.json b/build/Helix/global.json deleted file mode 100644 index 53d8ed33c45..00000000000 --- a/build/Helix/global.json +++ /dev/null @@ -1,5 +0,0 @@ -{ - "msbuild-sdks": { - "Microsoft.DotNet.Helix.Sdk": "6.0.0-beta.22525.5" - } -} diff --git a/build/Helix/packages.config b/build/Helix/packages.config deleted file mode 100644 index b101a50d6fe..00000000000 --- a/build/Helix/packages.config +++ /dev/null @@ -1,9 +0,0 @@ - - - - - - - - - diff --git a/build/Helix/readme.md b/build/Helix/readme.md deleted file mode 100644 index d51fc973c1e..00000000000 --- a/build/Helix/readme.md +++ /dev/null @@ -1,32 +0,0 @@ -This directory contains code and configuration files to run WinUI tests in Helix. - -Helix is a cloud hosted test execution environment which is accessed via the Arcade SDK. -More details: -* [Arcade](https://github.com/dotnet/arcade) -* [Helix](https://github.com/dotnet/arcade/tree/master/src/Microsoft.DotNet.Helix/Sdk) - -WinUI tests are scheduled in Helix by the Azure DevOps Pipeline: [RunHelixTests.yml](../RunHelixTests.yml). - -The workflow is as follows: -1. NuGet Restore is called on the packages.config in this directory. This downloads any runtime dependencies -that are needed to run tests. -2. PrepareHelixPayload.ps1 is called. This copies the necessary files from various locations into a Helix -payload directory. This directory is what will get sent to the Helix machines. -3. RunTestsInHelix.proj is executed. This proj has a dependency on -[Microsoft.DotNet.Helix.Sdk](https://github.com/dotnet/arcade/tree/master/src/Microsoft.DotNet.Helix/Sdk) -which it uses to publish the Helix payload directory and to schedule the Helix Work Items. The WinUI tests -are parallelized into multiple Helix Work Items. -4. Each Helix Work Item calls [runtests.cmd](runtests.cmd) with a specific query to pass to -[TAEF](https://docs.microsoft.com/en-us/windows-hardware/drivers/taef/) which runs the tests. -5. If a test is detected to have failed, we run it again, first once, then eight more times if it fails again. -If it fails all ten times, we report the test as failed; otherwise, we report it as unreliable, -which will show up as a warning, but which will not fail the build. When a test is reported as unreliable, -we include the results for each individual run via a JSON string in the original test's errorMessage field. -6. TAEF produces logs in WTT format. Helix is able to process logs in XUnit format. We run -[ConvertWttLogToXUnit.ps1](ConvertWttLogToXUnit.ps1) to convert the logs into the necessary format. -7. RunTestsInHelix.proj has EnableAzurePipelinesReporter set to true. This allows the XUnit formatted test -results to be reported back to the Azure DevOps Pipeline. -8. We process unreliable tests once all tests have been reported by reading the JSON string from the -errorMessage field and calling the Azure DevOps REST API to modify the unreliable tests to have sub-results -added to the test and to mark the test as "warning", which will enable people to see exactly how the test -failed in runs where it did. \ No newline at end of file diff --git a/build/Helix/runtests.cmd b/build/Helix/runtests.cmd deleted file mode 100644 index 5c543a141bf..00000000000 --- a/build/Helix/runtests.cmd +++ /dev/null @@ -1,105 +0,0 @@ -setlocal ENABLEDELAYEDEXPANSION - -echo %TIME% - -robocopy %HELIX_CORRELATION_PAYLOAD% . /s /NP > NUL - -echo %TIME% - -reg add HKLM\Software\Policies\Microsoft\Windows\Appx /v AllowAllTrustedApps /t REG_DWORD /d 1 /f - -rem enable dump collection for our test apps: -rem note, this script is run from a 32-bit cmd, but we need to set the native reg-key -FOR %%A IN (TestHostApp.exe,te.exe,te.processhost.exe,conhost.exe,OpenConsole.exe,WindowsTerminal.exe) DO ( - %systemroot%\sysnative\cmd.exe /c reg add "HKLM\Software\Microsoft\Windows\Windows Error Reporting\LocalDumps\%%A" /v DumpFolder /t REG_EXPAND_SZ /d %HELIX_DUMP_FOLDER% /f - %systemroot%\sysnative\cmd.exe /c reg add "HKLM\Software\Microsoft\Windows\Windows Error Reporting\LocalDumps\%%A" /v DumpType /t REG_DWORD /d 2 /f - %systemroot%\sysnative\cmd.exe /c reg add "HKLM\Software\Microsoft\Windows\Windows Error Reporting\LocalDumps\%%A" /v DumpCount /t REG_DWORD /d 10 /f -) - -echo %TIME% - -:: kill dhandler, which is a tool designed to handle unexpected windows appearing. But since our tests are -:: expected to show UI we don't want it running. -taskkill -f -im dhandler.exe - -echo %TIME% -powershell -ExecutionPolicy Bypass .\EnsureMachineState.ps1 -echo %TIME% -powershell -ExecutionPolicy Bypass .\InstallTestAppDependencies.ps1 -echo %TIME% - -set testBinaryCandidates=TerminalApp.LocalTests.dll SettingsModel.LocalTests.dll Conhost.UIA.Tests.dll WindowsTerminal.UIA.Tests.dll -set testBinaries= -for %%B in (%testBinaryCandidates%) do ( - if exist %%B ( - set "testBinaries=!testBinaries! %%B" - ) -) - -echo %TIME% -te.exe %testBinaries% /enablewttlogging /unicodeOutput:false /sessionTimeout:0:15 /testtimeout:0:10 /screenCaptureOnError %* -echo %TIME% - -powershell -ExecutionPolicy Bypass Get-Process - -move te.wtl te_original.wtl - -copy /y te_original.wtl %HELIX_WORKITEM_UPLOAD_ROOT% -copy /y WexLogFileOutput\*.jpg %HELIX_WORKITEM_UPLOAD_ROOT% -copy /y *.pgc %HELIX_WORKITEM_UPLOAD_ROOT% - -set FailedTestQuery= -for /F "tokens=* usebackq" %%I IN (`powershell -ExecutionPolicy Bypass .\OutputFailedTestQuery.ps1 te_original.wtl`) DO ( - set FailedTestQuery=%%I -) - -rem The first time, we'll just re-run failed tests once. In many cases, tests fail very rarely, such that -rem a single re-run will be sufficient to detect many unreliable tests. -if "%FailedTestQuery%" == "" goto :SkipReruns - -echo %TIME% -te.exe %testBinaries% /enablewttlogging /unicodeOutput:false /sessionTimeout:0:15 /testtimeout:0:10 /screenCaptureOnError /select:"%FailedTestQuery%" -echo %TIME% - -move te.wtl te_rerun.wtl - -copy /y te_rerun.wtl %HELIX_WORKITEM_UPLOAD_ROOT% -copy /y WexLogFileOutput\*.jpg %HELIX_WORKITEM_UPLOAD_ROOT% - -rem If there are still failing tests remaining, we'll run them eight more times, so they'll have been run a total of ten times. -rem If any tests fail all ten times, we can be pretty confident that these are actual test failures rather than unreliable tests. -if not exist te_rerun.wtl goto :SkipReruns - -set FailedTestQuery= -for /F "tokens=* usebackq" %%I IN (`powershell -ExecutionPolicy Bypass .\OutputFailedTestQuery.ps1 te_rerun.wtl`) DO ( - set FailedTestQuery=%%I -) - -if "%FailedTestQuery%" == "" goto :SkipReruns - -echo %TIME% -te.exe %testBinaries% /enablewttlogging /unicodeOutput:false /sessionTimeout:0:15 /testtimeout:0:10 /screenCaptureOnError /testmode:Loop /LoopTest:8 /select:"%FailedTestQuery%" -echo %TIME% - -powershell -ExecutionPolicy Bypass Get-Process - -move te.wtl te_rerun_multiple.wtl - -copy /y te_rerun_multiple.wtl %HELIX_WORKITEM_UPLOAD_ROOT% -copy /y WexLogFileOutput\*.jpg %HELIX_WORKITEM_UPLOAD_ROOT% -powershell -ExecutionPolicy Bypass .\CopyVisualTreeVerificationFiles.ps1 - -:SkipReruns - -powershell -ExecutionPolicy Bypass Get-Process - -echo %TIME% -powershell -ExecutionPolicy Bypass .\OutputSubResultsJsonFiles.ps1 te_original.wtl te_rerun.wtl te_rerun_multiple.wtl %testnameprefix% -powershell -ExecutionPolicy Bypass .\ConvertWttLogToXUnit.ps1 te_original.wtl te_rerun.wtl te_rerun_multiple.wtl testResults.xml %testnameprefix% -echo %TIME% - -copy /y *_subresults.json %HELIX_WORKITEM_UPLOAD_ROOT% - -type testResults.xml - -echo %TIME% diff --git a/build/config/ESRPSigning_ConPTY.json b/build/config/ESRPSigning_ConPTY.json deleted file mode 100644 index d01e9c9eabe..00000000000 --- a/build/config/ESRPSigning_ConPTY.json +++ /dev/null @@ -1,51 +0,0 @@ -{ - "Version": "1.0.0", - "UseMinimatch": false, - "SignBatches": [ - { - "MatchedPath": [ - "conpty.dll", - "OpenConsole.exe" - ], - "SigningInfo": { - "Operations": [ - { - "KeyCode": "CP-230012", - "OperationSetCode": "SigntoolSign", - "Parameters": [ - { - "parameterName": "OpusName", - "parameterValue": "Microsoft" - }, - { - "parameterName": "OpusInfo", - "parameterValue": "http://www.microsoft.com" - }, - { - "parameterName": "FileDigest", - "parameterValue": "/fd \"SHA256\"" - }, - { - "parameterName": "PageHash", - "parameterValue": "/NPH" - }, - { - "parameterName": "TimeStamp", - "parameterValue": "/tr \"http://rfc3161.gtm.corp.microsoft.com/TSS/HttpTspServer\" /td sha256" - } - ], - "ToolName": "sign", - "ToolVersion": "1.0" - }, - { - "KeyCode": "CP-230012", - "OperationSetCode": "SigntoolVerify", - "Parameters": [], - "ToolName": "sign", - "ToolVersion": "1.0" - } - ] - } - } - ] -} diff --git a/build/config/ESRPSigning_Terminal.json b/build/config/ESRPSigning_Terminal.json deleted file mode 100644 index 01780c2df3c..00000000000 --- a/build/config/ESRPSigning_Terminal.json +++ /dev/null @@ -1,72 +0,0 @@ -{ - "Version": "1.0.0", - "UseMinimatch": false, - "SignBatches": [ - { - "MatchedPath": [ - // Namespaced DLLs - "Microsoft.Terminal.*.dll", - "Microsoft.Terminal.*.winmd", - - // ConPTY and DefTerm - "OpenConsole.exe", - "OpenConsoleProxy.dll", - - // VCRT Forwarders - "*_app.dll", - - // Legacy DLLs with old names - "TerminalApp.dll", - "TerminalApp.winmd", - "TerminalConnection.dll", - "TerminalThemeHelpers.dll", - "WindowsTerminalShellExt.dll", - - // The rest - "TerminalAzBridge.exe", - "wt.exe", - "WindowsTerminal.exe", - "elevate-shim.exe" - ], - "SigningInfo": { - "Operations": [ - { - "KeyCode": "CP-230012", - "OperationSetCode": "SigntoolSign", - "Parameters": [ - { - "parameterName": "OpusName", - "parameterValue": "Microsoft" - }, - { - "parameterName": "OpusInfo", - "parameterValue": "http://www.microsoft.com" - }, - { - "parameterName": "FileDigest", - "parameterValue": "/fd \"SHA256\"" - }, - { - "parameterName": "PageHash", - "parameterValue": "/NPH" - }, - { - "parameterName": "TimeStamp", - "parameterValue": "/tr \"http://rfc3161.gtm.corp.microsoft.com/TSS/HttpTspServer\" /td sha256" - } - ], - "ToolName": "sign", - "ToolVersion": "1.0" - }, - { - "KeyCode": "CP-230012", - "OperationSetCode": "SigntoolVerify", - "Parameters": [], - "ToolName": "sign", - "ToolVersion": "1.0" - } - ] - } - } - ] -} diff --git a/build/config/esrp.build.batch.conpty.json b/build/config/esrp.build.batch.conpty.json new file mode 100644 index 00000000000..b0097056ec5 --- /dev/null +++ b/build/config/esrp.build.batch.conpty.json @@ -0,0 +1,47 @@ +[ + { + "MatchedPath": [ + "conpty.dll", + "OpenConsole.exe" + ], + "SigningInfo": { + "Operations": [ + { + "KeyCode": "CP-230012", + "OperationSetCode": "SigntoolSign", + "Parameters": [ + { + "parameterName": "OpusName", + "parameterValue": "Microsoft" + }, + { + "parameterName": "OpusInfo", + "parameterValue": "http://www.microsoft.com" + }, + { + "parameterName": "FileDigest", + "parameterValue": "/fd \"SHA256\"" + }, + { + "parameterName": "PageHash", + "parameterValue": "/NPH" + }, + { + "parameterName": "TimeStamp", + "parameterValue": "/tr \"http://rfc3161.gtm.corp.microsoft.com/TSS/HttpTspServer\" /td sha256" + } + ], + "ToolName": "sign", + "ToolVersion": "1.0" + }, + { + "KeyCode": "CP-230012", + "OperationSetCode": "SigntoolVerify", + "Parameters": [], + "ToolName": "sign", + "ToolVersion": "1.0" + } + ] + } + } +] diff --git a/build/config/esrp.build.batch.terminal_constituents.json b/build/config/esrp.build.batch.terminal_constituents.json new file mode 100644 index 00000000000..c33123a16de --- /dev/null +++ b/build/config/esrp.build.batch.terminal_constituents.json @@ -0,0 +1,65 @@ +[ + { + "MatchedPath": [ + // Namespaced DLLs + "PackageContents/Microsoft.Terminal.*.dll", + "PackageContents/Microsoft.Terminal.*.winmd", + + // ConPTY and DefTerm + "PackageContents/OpenConsole.exe", + "PackageContents/OpenConsoleProxy.dll", + + // Legacy DLLs with old names + "PackageContents/TerminalApp.dll", + "PackageContents/TerminalApp.winmd", + "PackageContents/TerminalConnection.dll", + "PackageContents/TerminalThemeHelpers.dll", + "PackageContents/WindowsTerminalShellExt.dll", + + // The rest + "PackageContents/TerminalAzBridge.exe", + "PackageContents/wt.exe", + "PackageContents/WindowsTerminal.exe", + "PackageContents/elevate-shim.exe" + ], + "SigningInfo": { + "Operations": [ + { + "KeyCode": "CP-230012", + "OperationSetCode": "SigntoolSign", + "Parameters": [ + { + "parameterName": "OpusName", + "parameterValue": "Microsoft" + }, + { + "parameterName": "OpusInfo", + "parameterValue": "http://www.microsoft.com" + }, + { + "parameterName": "FileDigest", + "parameterValue": "/fd \"SHA256\"" + }, + { + "parameterName": "PageHash", + "parameterValue": "/NPH" + }, + { + "parameterName": "TimeStamp", + "parameterValue": "/tr \"http://rfc3161.gtm.corp.microsoft.com/TSS/HttpTspServer\" /td sha256" + } + ], + "ToolName": "sign", + "ToolVersion": "1.0" + }, + { + "KeyCode": "CP-230012", + "OperationSetCode": "SigntoolVerify", + "Parameters": [], + "ToolName": "sign", + "ToolVersion": "1.0" + } + ] + } + } +] diff --git a/build/config/esrp.build.batch.wpf.json b/build/config/esrp.build.batch.wpf.json new file mode 100644 index 00000000000..e9c9af87891 --- /dev/null +++ b/build/config/esrp.build.batch.wpf.json @@ -0,0 +1,46 @@ +[ + { + "MatchedPath": [ + "PublicTerminalCore.dll" + ], + "SigningInfo": { + "Operations": [ + { + "KeyCode": "CP-230012", + "OperationSetCode": "SigntoolSign", + "Parameters": [ + { + "parameterName": "OpusName", + "parameterValue": "Microsoft" + }, + { + "parameterName": "OpusInfo", + "parameterValue": "http://www.microsoft.com" + }, + { + "parameterName": "FileDigest", + "parameterValue": "/fd \"SHA256\"" + }, + { + "parameterName": "PageHash", + "parameterValue": "/NPH" + }, + { + "parameterName": "TimeStamp", + "parameterValue": "/tr \"http://rfc3161.gtm.corp.microsoft.com/TSS/HttpTspServer\" /td sha256" + } + ], + "ToolName": "sign", + "ToolVersion": "1.0" + }, + { + "KeyCode": "CP-230012", + "OperationSetCode": "SigntoolVerify", + "Parameters": [], + "ToolName": "sign", + "ToolVersion": "1.0" + } + ] + } + } +] diff --git a/build/config/esrp.build.batch.wpfdotnet.json b/build/config/esrp.build.batch.wpfdotnet.json new file mode 100644 index 00000000000..0699353f64c --- /dev/null +++ b/build/config/esrp.build.batch.wpfdotnet.json @@ -0,0 +1,47 @@ +[ + { + "MatchedPath": [ + "WpfTerminalControl/net472/Microsoft.Terminal.Wpf.dll", + "WpfTerminalControl/net6.0-windows/Microsoft.Terminal.Wpf.dll" + ], + "SigningInfo": { + "Operations": [ + { + "KeyCode": "CP-230012", + "OperationSetCode": "SigntoolSign", + "Parameters": [ + { + "parameterName": "OpusName", + "parameterValue": "Microsoft" + }, + { + "parameterName": "OpusInfo", + "parameterValue": "http://www.microsoft.com" + }, + { + "parameterName": "FileDigest", + "parameterValue": "/fd \"SHA256\"" + }, + { + "parameterName": "PageHash", + "parameterValue": "/NPH" + }, + { + "parameterName": "TimeStamp", + "parameterValue": "/tr \"http://rfc3161.gtm.corp.microsoft.com/TSS/HttpTspServer\" /td sha256" + } + ], + "ToolName": "sign", + "ToolVersion": "1.0" + }, + { + "KeyCode": "CP-230012", + "OperationSetCode": "SigntoolVerify", + "Parameters": [], + "ToolName": "sign", + "ToolVersion": "1.0" + } + ] + } + } +] diff --git a/build/pgo/PGO.DB.proj b/build/pgo/PGO.DB.proj index f89eff931ac..ce7e1c6ebcb 100644 --- a/build/pgo/PGO.DB.proj +++ b/build/pgo/PGO.DB.proj @@ -1,4 +1,5 @@ + - \ No newline at end of file + diff --git a/build/pgo/Terminal.PGO.DB.nuspec b/build/pgo/Terminal.PGO.DB.nuspec index d49574423a8..b1437af2a0b 100644 --- a/build/pgo/Terminal.PGO.DB.nuspec +++ b/build/pgo/Terminal.PGO.DB.nuspec @@ -12,5 +12,6 @@ + diff --git a/build/pgo/Terminal.PGO.props b/build/pgo/Terminal.PGO.props index dedf48a6723..109bdd7ffa0 100644 --- a/build/pgo/Terminal.PGO.props +++ b/build/pgo/Terminal.PGO.props @@ -46,6 +46,5 @@ true - - + diff --git a/build/pipelines/ci.yml b/build/pipelines/ci.yml index 3b833d8e718..3a142688443 100644 --- a/build/pipelines/ci.yml +++ b/build/pipelines/ci.yml @@ -35,11 +35,7 @@ parameters: type: boolean default: true - name: runTests - displayName: "Run Unit and Feature Tests" - type: boolean - default: true - - name: submitHelix - displayName: "Submit to Helix for testing and PGO" + displayName: "Run Tests" type: boolean default: true - name: buildPlatforms @@ -54,28 +50,43 @@ stages: - stage: Audit_x64 displayName: Audit Mode dependsOn: [] - condition: succeeded() jobs: - - template: ./templates/build-console-audit-job.yml + - template: ./templates-v2/job-build-project.yml parameters: - platform: x64 + pool: + ${{ if eq(variables['System.CollectionUri'], 'https://dev.azure.com/ms/') }}: + name: SHINE-OSS-L + ${{ if ne(variables['System.CollectionUri'], 'https://dev.azure.com/ms/') }}: + name: SHINE-INT-L + buildPlatforms: [x64] + buildConfigurations: [AuditMode] + buildEverything: true + keepAllExpensiveBuildOutputs: false - - stage: Scripts - displayName: Code Health Scripts + - stage: CodeHealth + displayName: Code Health dependsOn: [] - condition: succeeded() jobs: - - template: ./templates/check-formatting.yml + - template: ./templates-v2/job-check-code-format.yml - ${{ each platform in parameters.buildPlatforms }}: - stage: Build_${{ platform }} displayName: Build ${{ platform }} dependsOn: [] - condition: succeeded() jobs: - - template: ./templates/build-console-ci.yml + - template: ./templates-v2/job-build-project.yml parameters: - platform: ${{ platform }} + pool: + ${{ if eq(variables['System.CollectionUri'], 'https://dev.azure.com/ms/') }}: + name: SHINE-OSS-L + ${{ if ne(variables['System.CollectionUri'], 'https://dev.azure.com/ms/') }}: + name: SHINE-INT-L + buildPlatforms: + - ${{ platform }} + buildConfigurations: [Release] + buildEverything: true + keepAllExpensiveBuildOutputs: false + - ${{ if eq(parameters.runTests, true) }}: - stage: Test_${{ platform }} displayName: Test ${{ platform }} @@ -83,22 +94,13 @@ stages: - Build_${{ platform }} condition: succeeded() jobs: - - template: ./templates/test-console-ci.yml + - template: ./templates-v2/job-test-project.yml parameters: platform: ${{ platform }} - - ${{ if and(containsValue(parameters.buildPlatforms, 'x64'), eq(parameters.submitHelix, true), ne(variables['Build.Reason'], 'PullRequest')) }}: - - stage: Helix_x64 - displayName: Helix x64 - dependsOn: [Build_x64] - jobs: - - template: ./templates/console-ci-helix-job.yml - parameters: - platform: x64 - - - ${{ if and(containsValue(parameters.buildPlatforms, 'x64'), ne(variables['Build.Reason'], 'PullRequest')) }}: + - ${{ if ne(variables['Build.Reason'], 'PullRequest') }}: - stage: CodeIndexer displayName: Github CodeNav Indexer - dependsOn: [Build_x64] + dependsOn: [] jobs: - - template: ./templates/codenav-indexer.yml + - template: ./templates-v2/job-index-github-codenav.yml diff --git a/build/pipelines/daily-loc-submission.yml b/build/pipelines/daily-loc-submission.yml index 126b2e10359..890c1f86a47 100644 --- a/build/pipelines/daily-loc-submission.yml +++ b/build/pipelines/daily-loc-submission.yml @@ -27,6 +27,7 @@ steps: clean: true submodules: false fetchDepth: 1 # Don't need a deep checkout for loc files! + fetchTags: false # Tags still result in depth > 1 fetch; we don't need them here persistCredentials: true path: s # Adding a second repo made Azure DevOps change where we're checked out. diff --git a/build/pipelines/feature-flag-ci.yml b/build/pipelines/feature-flag-ci.yml index 0ca4400463d..ead379dbf36 100644 --- a/build/pipelines/feature-flag-ci.yml +++ b/build/pipelines/feature-flag-ci.yml @@ -7,6 +7,7 @@ pr: paths: include: - src/features.xml + - build/pipelines/feature-flag-ci.yml variables: - name: runCodesignValidationInjectionBG @@ -21,9 +22,19 @@ parameters: # Dev is built automatically # WindowsInbox does not typically build with VS. -jobs: +stages: - ${{ each branding in parameters.buildBrandings }}: - - template: ./templates/build-console-ci.yml - parameters: - platform: x64 - branding: ${{ branding }} + - stage: Build_${{ branding }} + dependsOn: [] + displayName: Build ${{ branding }} + jobs: + - template: ./templates-v2/job-build-project.yml + parameters: + pool: # This only runs in CI + name: SHINE-OSS-L + buildPlatforms: [x64] + buildConfigurations: [Release] + buildEverything: true + branding: ${{ branding }} + keepAllExpensiveBuildOutputs: false + artifactStem: -${{ branding }} # Disambiguate artifacts with the same config/platform diff --git a/build/pipelines/fuzz.yml b/build/pipelines/fuzz.yml index b5604b5f227..75b781d33c6 100644 --- a/build/pipelines/fuzz.yml +++ b/build/pipelines/fuzz.yml @@ -16,44 +16,52 @@ pr: none name: 0.0.$(Date:yyMM).$(Date:dd)$(Rev:rr) stages: - - stage: Build_Fuzz_Config - displayName: Build Fuzzers + - stage: Build + displayName: Fuzzing Build dependsOn: [] condition: succeeded() jobs: - - template: ./templates/build-console-fuzzing.yml - parameters: - platform: x64 - - stage: OneFuzz - displayName: Submit OneFuzz Job - dependsOn: ['Build_Fuzz_Config'] + - template: ./templates-v2/job-build-project.yml + parameters: + pool: + ${{ if eq(variables['System.CollectionUri'], 'https://dev.azure.com/ms/') }}: + name: SHINE-OSS-L + ${{ if ne(variables['System.CollectionUri'], 'https://dev.azure.com/ms/') }}: + name: SHINE-INT-L + buildPlatforms: [x64] + buildConfigurations: [Fuzzing] + buildEverything: true + keepAllExpensiveBuildOutputs: false + + - stage: Submit + displayName: Submit to OneFuzz + dependsOn: [Build] condition: succeeded() - pool: - vmImage: 'ubuntu-latest' - variables: - artifactName: fuzzingBuildOutput jobs: - - job: - steps: - - task: DownloadBuildArtifacts@0 - inputs: - artifactName: $(artifactName) - downloadPath: $(Build.ArtifactStagingDirectory) - - task: UsePythonVersion@0 - inputs: - versionSpec: '3.x' - addToPath: true - architecture: 'x64' - - bash: | - set -ex - pip -q install onefuzz - onefuzz config --endpoint $(endpoint) --client_id $(client_id) --authority $(authority) --tenant_domain $(tenant_domain) --client_secret $(client_secret) - sed -i s/INSERT_PAT_HERE/$(ado_pat)/ build/Fuzz/notifications-ado.json - sed -i s/INSERT_ASSIGNED_HERE/$(ado_assigned_to)/ build/Fuzz/notifications-ado.json - displayName: Configure OneFuzz - - bash: | - onefuzz template libfuzzer basic --colocate_all_tasks --vm_count 1 --target_exe $target_exe_path --notification_config @./build/Fuzz/notifications-ado.json OpenConsole $test_name $(Build.SourceVersion) default - displayName: Submit OneFuzz Job - env: - target_exe_path: $(Build.ArtifactStagingDirectory)/$(artifactName)/Fuzzing/x64/test/OpenConsoleFuzzer.exe - test_name: WriteCharsLegacy + - job: + pool: + vmImage: 'ubuntu-latest' + steps: + - task: DownloadPipelineArtifact@2 + displayName: Download artifacts + inputs: + artifactName: build-x64-Fuzzing + downloadPath: $(Build.ArtifactStagingDirectory) + - task: UsePythonVersion@0 + inputs: + versionSpec: '3.x' + addToPath: true + architecture: 'x64' + - bash: | + set -ex + pip -q install onefuzz + onefuzz config --endpoint $(endpoint) --client_id $(client_id) --authority $(authority) --tenant_domain $(tenant_domain) --client_secret $(client_secret) + sed -i s/INSERT_PAT_HERE/$(ado_pat)/ build/Fuzz/notifications-ado.json + sed -i s/INSERT_ASSIGNED_HERE/$(ado_assigned_to)/ build/Fuzz/notifications-ado.json + displayName: Configure OneFuzz + - bash: | + onefuzz template libfuzzer basic --colocate_all_tasks --vm_count 1 --target_exe $target_exe_path --notification_config @./build/Fuzz/notifications-ado.json OpenConsole $test_name $(Build.SourceVersion) default + displayName: Submit OneFuzz Job + env: + target_exe_path: $(Build.ArtifactStagingDirectory)/OpenConsoleFuzzer.exe + test_name: WriteCharsLegacy diff --git a/build/pipelines/pgo.yml b/build/pipelines/pgo.yml index 1c93a56d8cd..d659c4e2520 100644 --- a/build/pipelines/pgo.yml +++ b/build/pipelines/pgo.yml @@ -1,5 +1,27 @@ trigger: none pr: none +schedules: + - cron: "0 5 * * 2-6" # Run at 05:00 UTC Tuesday through Saturday (Even later than Localization, after the work day in Pacific, Mon-Fri) + displayName: "Nightly Instrumentation Build" + branches: + include: + - main + always: false # only run if there's code changes! + +parameters: + - name: branding + displayName: "Branding (Build Type)" + type: string + default: Preview # By default, we'll PGO the Preview builds to get max coverage + values: + - Release + - Preview + - Dev + - name: buildPlatforms + type: object + default: + - x64 + - arm64 variables: - name: runCodesignValidationInjectionBG @@ -10,18 +32,57 @@ variables: name: 0.0.$(Date:yyMM).$(Date:dd)$(Rev:rr) stages: - - stage: Build_x64 - displayName: Build x64 + - stage: Build + displayName: Build dependsOn: [] condition: succeeded() jobs: - - template: ./templates/build-console-pgo.yml + - template: ./templates-v2/job-build-project.yml parameters: - platform: x64 - - stage: Publish_PGO_Databases - displayName: Publish PGO databases - dependsOn: ['Build_x64'] + pool: + ${{ if eq(variables['System.CollectionUri'], 'https://dev.azure.com/ms/') }}: + name: SHINE-OSS-L + ${{ if ne(variables['System.CollectionUri'], 'https://dev.azure.com/ms/') }}: + name: SHINE-INT-L + branding: ${{ parameters.branding }} + buildPlatforms: ${{ parameters.buildPlatforms }} + buildConfigurations: [Release] + buildEverything: true + pgoBuildMode: Instrument + artifactStem: -instrumentation + + - stage: RunPGO + displayName: Run PGO + dependsOn: [Build] + condition: succeeded() + jobs: + - ${{ each platform in parameters.buildPlatforms }}: + - template: ./templates-v2/job-run-pgo-tests.yml + parameters: + # This job chooses its own pools based on platform + buildPlatform: ${{ platform }} + buildConfiguration: Release + artifactStem: -instrumentation + + - stage: FinalizePGO + displayName: Finalize PGO and Publish + dependsOn: [RunPGO] + condition: succeeded() jobs: - - template: ./templates/pgo-build-and-publish-nuget-job.yml + # This job takes multiple platforms and fans them back in to a single artifact. + - template: ./templates-v2/job-pgo-merge-pgd.yml + parameters: + jobName: MergePGD + pool: + vmImage: 'windows-2022' + buildConfiguration: Release + buildPlatforms: ${{ parameters.buildPlatforms }} + artifactStem: -instrumentation + + - template: ./templates-v2/job-pgo-build-nuget-and-publish.yml parameters: - pgoArtifact: 'PGO' + pool: + vmImage: 'windows-2022' + dependsOn: MergePGD + buildConfiguration: Release + artifactStem: -instrumentation diff --git a/build/pipelines/release.yml b/build/pipelines/release.yml index cea5c7d0fc5..fd388ea1d0e 100644 --- a/build/pipelines/release.yml +++ b/build/pipelines/release.yml @@ -14,22 +14,11 @@ parameters: values: - Release - Preview + - Dev - name: buildTerminal displayName: "Build Windows Terminal MSIX" type: boolean default: true - - name: runCompliance - displayName: "Run Compliance and Security Build" - type: boolean - default: true - - name: publishSymbolsToPublic - displayName: "Publish Symbols to MSDL" - type: boolean - default: true - - name: buildTerminalVPack - displayName: "Build Windows Terminal VPack" - type: boolean - default: false - name: buildConPTY displayName: "Build ConPTY NuGet" type: boolean @@ -47,19 +36,44 @@ parameters: - Instrument - None - name: buildConfigurations + displayName: "Build Configurations" type: object default: - Release - name: buildPlatforms + displayName: "Build Platforms" type: object default: - x64 - x86 - arm64 + - name: codeSign + displayName: "Sign all build outputs" + type: boolean + default: true + - name: generateSbom + displayName: "Generate a Bill of Materials" + type: boolean + default: true + - name: terminalInternalPackageVersion + displayName: "Terminal Internal Package Version" + type: string + default: '0.0.8' + + - name: runCompliance + displayName: "Run Compliance and Security Build" + type: boolean + default: true + - name: publishSymbolsToPublic + displayName: "Publish Symbols to MSDL" + type: boolean + default: true + - name: publishVpackToWindows + displayName: "Publish VPack to Windows" + type: boolean + default: false variables: - MakeAppxPath: 'C:\Program Files (x86)\Windows Kits\10\bin\10.0.22621.0\x86\MakeAppx.exe' - TerminalInternalPackageVersion: "0.0.8" # If we are building a branch called "release-*", change the NuGet suffix # to "preview". If we don't do that, XES will set the suffix to "release1" # because it truncates the value after the first period. @@ -84,671 +98,113 @@ variables: NuGetPackBetaVersion: experimental name: $(BuildDefinitionName)_$(date:yyMM).$(date:dd)$(rev:rrr) + resources: repositories: - repository: self type: git ref: main -jobs: -- job: Build - pool: - name: SHINE-INT-L # Run the compilation on the large agent pool, rather than the default small one. - demands: ImageOverride -equals SHINE-VS17-Latest - strategy: - matrix: - ${{ each config in parameters.buildConfigurations }}: - ${{ each platform in parameters.buildPlatforms }}: - ${{ config }}_${{ platform }}: - BuildConfiguration: ${{ config }} - BuildPlatform: ${{ platform }} - displayName: Build - timeoutInMinutes: 240 - cancelTimeoutInMinutes: 1 - steps: - - checkout: self - clean: true - submodules: true - persistCredentials: True - - task: PkgESSetupBuild@12 - displayName: Package ES - Setup Build - inputs: - disableOutputRedirect: true - - task: PowerShell@2 - displayName: Rationalize Build Platform - inputs: - targetType: inline - script: >- - $Arch = "$(BuildPlatform)" - - If ($Arch -Eq "x86") { $Arch = "Win32" } - - Write-Host "##vso[task.setvariable variable=RationalizedBuildPlatform]${Arch}" - - template: .\templates\restore-nuget-steps.yml - - task: UniversalPackages@0 - displayName: Download terminal-internal Universal Package - inputs: - feedListDownload: 2b3f8893-a6e8-411f-b197-a9e05576da48 - packageListDownload: e82d490c-af86-4733-9dc4-07b772033204 - versionListDownload: $(TerminalInternalPackageVersion) - - task: TouchdownBuildTask@1 - displayName: Download Localization Files - inputs: - teamId: 7105 - authId: $(TouchdownAppId) - authKey: $(TouchdownAppKey) - resourceFilePath: >- - src\cascadia\TerminalApp\Resources\en-US\Resources.resw - - src\cascadia\TerminalApp\Resources\en-US\ContextMenu.resw - - src\cascadia\TerminalControl\Resources\en-US\Resources.resw - - src\cascadia\TerminalConnection\Resources\en-US\Resources.resw - - src\cascadia\TerminalSettingsModel\Resources\en-US\Resources.resw - - src\cascadia\TerminalSettingsEditor\Resources\en-US\Resources.resw - - src\cascadia\CascadiaPackage\Resources\en-US\Resources.resw - appendRelativeDir: true - localizationTarget: false - pseudoSetting: Included - - task: PowerShell@2 - displayName: Move Loc files one level up - inputs: - targetType: inline - script: >- - $Files = Get-ChildItem . -R -Filter 'Resources.resw' | ? FullName -Like '*en-US\*\Resources.resw' - - $Files | % { Move-Item -Verbose $_.Directory $_.Directory.Parent.Parent -EA:Ignore } - pwsh: true - - task: PowerShell@2 - displayName: Copy the Context Menu Loc Resources to CascadiaPackage - inputs: - filePath: ./build/scripts/Copy-ContextMenuResourcesToCascadiaPackage.ps1 - pwsh: true - - task: PowerShell@2 - displayName: Generate NOTICE.html from NOTICE.md - inputs: - filePath: .\build\scripts\Generate-ThirdPartyNotices.ps1 - arguments: -MarkdownNoticePath .\NOTICE.md -OutputPath .\src\cascadia\CascadiaPackage\NOTICE.html - pwsh: true - - ${{ if eq(parameters.buildTerminal, true) }}: - - task: VSBuild@1 - displayName: Build solution **\OpenConsole.sln - condition: true - inputs: - solution: '**\OpenConsole.sln' - msbuildArgs: /p:WindowsTerminalOfficialBuild=true /p:WindowsTerminalBranding=${{ parameters.branding }};PGOBuildMode=${{ parameters.pgoBuildMode }} /t:Terminal\CascadiaPackage /p:WindowsTerminalReleaseBuild=true /bl:$(Build.SourcesDirectory)\msbuild.binlog - platform: $(BuildPlatform) - configuration: $(BuildConfiguration) - clean: true - maximumCpuCount: true - - task: PublishBuildArtifacts@1 - displayName: 'Publish Artifact: binlog' - condition: failed() - continueOnError: True - inputs: - PathtoPublish: $(Build.SourcesDirectory)\msbuild.binlog - ArtifactName: binlog-$(BuildPlatform) - - task: PowerShell@2 - displayName: Check MSIX for common regressions - inputs: - targetType: inline - script: >- - $Package = Get-ChildItem -Recurse -Filter "CascadiaPackage_*.msix" - - .\build\scripts\Test-WindowsTerminalPackage.ps1 -Verbose -Path $Package.FullName - pwsh: true - - ${{ if eq(parameters.buildWPF, true) }}: - - task: VSBuild@1 - displayName: Build solution **\OpenConsole.sln for PublicTerminalCore - inputs: - solution: '**\OpenConsole.sln' - msbuildArgs: /p:WindowsTerminalOfficialBuild=true /p:WindowsTerminalBranding=${{ parameters.branding }};PGOBuildMode=${{ parameters.pgoBuildMode }} /p:WindowsTerminalReleaseBuild=true /t:Terminal\wpf\PublicTerminalCore - platform: $(BuildPlatform) - configuration: $(BuildConfiguration) - - ${{ if eq(parameters.buildConPTY, true) }}: - - task: VSBuild@1 - displayName: Build solution **\OpenConsole.sln for ConPTY - inputs: - solution: '**\OpenConsole.sln' - msbuildArgs: /p:WindowsTerminalOfficialBuild=true /p:WindowsTerminalBranding=${{ parameters.branding }};PGOBuildMode=${{ parameters.pgoBuildMode }} /p:WindowsTerminalReleaseBuild=true /t:Conhost\Host_EXE;Conhost\winconpty_DLL - platform: $(BuildPlatform) - configuration: $(BuildConfiguration) - - task: PowerShell@2 - displayName: Source Index PDBs - inputs: - filePath: build\scripts\Index-Pdbs.ps1 - arguments: -SearchDir '$(Build.SourcesDirectory)' -SourceRoot '$(Build.SourcesDirectory)' -recursive -Verbose -CommitId $(Build.SourceVersion) - errorActionPreference: silentlyContinue - pwsh: true - - task: PowerShell@2 - displayName: Run Unit Tests - condition: and(succeeded(), or(eq(variables['BuildPlatform'], 'x64'), eq(variables['BuildPlatform'], 'x86'))) - enabled: False - inputs: - filePath: build\scripts\Run-Tests.ps1 - arguments: -MatchPattern '*unit.test*.dll' -Platform '$(RationalizedBuildPlatform)' -Configuration '$(BuildConfiguration)' - - task: PowerShell@2 - displayName: Run Feature Tests - condition: and(succeeded(), eq(variables['BuildPlatform'], 'x64')) - enabled: False - inputs: - filePath: build\scripts\Run-Tests.ps1 - arguments: -MatchPattern '*feature.test*.dll' -Platform '$(RationalizedBuildPlatform)' -Configuration '$(BuildConfiguration)' - - ${{ if eq(parameters.buildTerminal, true) }}: - - task: CopyFiles@2 - displayName: Copy *.msix and symbols to Artifacts - inputs: - Contents: >- - **/*.msix - - **/*.appxsym - TargetFolder: $(Build.ArtifactStagingDirectory)/appx - OverWrite: true - flattenFolders: true - - - pwsh: |- - $Package = (Get-ChildItem "$(Build.ArtifactStagingDirectory)/appx" -Recurse -Filter "Cascadia*.msix" | Select -First 1) - $PackageFilename = $Package.FullName - Write-Host "##vso[task.setvariable variable=WindowsTerminalPackagePath]${PackageFilename}" - & "$(MakeAppxPath)" unpack /p $PackageFilename /d "$(Build.SourcesDirectory)\UnpackedTerminalPackage" - displayName: Unpack the new Terminal package for signing - - - task: EsrpCodeSigning@1 - displayName: Submit Terminal's binaries for signing - inputs: - ConnectedServiceName: 9d6d2960-0793-4d59-943e-78dcb434840a - FolderPath: '$(Build.SourcesDirectory)\UnpackedTerminalPackage' - signType: batchSigning - batchSignPolicyFile: '$(Build.SourcesDirectory)\build\config\ESRPSigning_Terminal.json' - - - pwsh: |- - $PackageFilename = "$(WindowsTerminalPackagePath)" - Remove-Item "$(Build.SourcesDirectory)\UnpackedTerminalPackage\CodeSignSummary*" - & "$(MakeAppxPath)" pack /h SHA256 /o /p $PackageFilename /d "$(Build.SourcesDirectory)\UnpackedTerminalPackage" - displayName: Re-pack the new Terminal package after signing - - - pwsh: |- - $XamlAppxPath = (Get-Item "src\cascadia\CascadiaPackage\AppPackages\*\Dependencies\$(BuildPlatform)\Microsoft.UI.Xaml*.appx").FullName - & .\build\scripts\New-UnpackagedTerminalDistribution.ps1 -TerminalAppX $(WindowsTerminalPackagePath) -XamlAppX $XamlAppxPath -Destination "$(Build.ArtifactStagingDirectory)/appx" - displayName: Build Unpackaged Distribution - - - task: AzureArtifacts.manifest-generator-task.manifest-generator-task.ManifestGeneratorTask@0 - displayName: 'Generate SBOM manifest (application)' - inputs: - BuildDropPath: '$(System.ArtifactsDirectory)/appx' - - - task: DropValidatorTask@0 - displayName: 'Validate application SBOM manifest' - inputs: - BuildDropPath: '$(System.ArtifactsDirectory)/appx' - OutputPath: 'output.json' - ValidateSignature: true - Verbosity: 'Verbose' - - - task: PublishBuildArtifacts@1 - displayName: Publish Artifact (Terminal app) - inputs: - PathtoPublish: $(Build.ArtifactStagingDirectory)/appx - ArtifactName: terminal-$(BuildPlatform)-$(BuildConfiguration) - - - ${{ if eq(parameters.buildConPTY, true) }}: - - task: CopyFiles@2 - displayName: Copy ConPTY to Artifacts - inputs: - Contents: |- - $(Build.SourcesDirectory)/bin/**/conpty.dll - $(Build.SourcesDirectory)/bin/**/conpty.lib - $(Build.SourcesDirectory)/bin/**/conpty.pdb - $(Build.SourcesDirectory)/bin/**/OpenConsole.exe - $(Build.SourcesDirectory)/bin/**/OpenConsole.pdb - TargetFolder: $(Build.ArtifactStagingDirectory)/conpty - OverWrite: true - flattenFolders: true - - task: PublishBuildArtifacts@1 - displayName: Publish Artifact (ConPTY) - inputs: - PathtoPublish: $(Build.ArtifactStagingDirectory)/conpty - ArtifactName: conpty-dll-$(BuildPlatform)-$(BuildConfiguration) - - ${{ if eq(parameters.buildWPF, true) }}: - - task: CopyFiles@2 - displayName: Copy PublicTerminalCore.dll to Artifacts - inputs: - Contents: >- - **/PublicTerminalCore.dll - TargetFolder: $(Build.ArtifactStagingDirectory)/wpf - OverWrite: true - flattenFolders: true - - task: PublishBuildArtifacts@1 - displayName: Publish Artifact (PublicTerminalCore) - inputs: - PathtoPublish: $(Build.ArtifactStagingDirectory)/wpf - ArtifactName: wpf-dll-$(BuildPlatform)-$(BuildConfiguration) - - - task: PublishSymbols@2 - displayName: Publish symbols path - continueOnError: True - inputs: - SearchPattern: | - $(Build.SourcesDirectory)/bin/**/*.pdb - $(Build.SourcesDirectory)/bin/**/*.exe - $(Build.SourcesDirectory)/bin/**/*.dll - IndexSources: false - SymbolServerType: TeamServices - -- ${{ if eq(parameters.runCompliance, true) }}: - - template: ./templates/build-console-compliance-job.yml - -- ${{ if eq(parameters.buildTerminal, true) }}: - - job: BundleAndSign - displayName: Create and sign AppX/MSIX bundles - variables: - ${{ if eq(parameters.branding, 'Release') }}: - BundleStemName: Microsoft.WindowsTerminal - ${{ elseif eq(parameters.branding, 'Preview') }}: - BundleStemName: Microsoft.WindowsTerminalPreview - ${{ else }}: - BundleStemName: WindowsTerminalDev - dependsOn: Build - steps: - - checkout: self - clean: true - fetchDepth: 1 - submodules: true - persistCredentials: True - - task: PkgESSetupBuild@12 - displayName: Package ES - Setup Build - inputs: - disableOutputRedirect: true - - ${{ each platform in parameters.buildPlatforms }}: - - task: DownloadBuildArtifacts@1 - displayName: Download Artifacts ${{ platform }} - inputs: - # Make sure to download the entire artifact, because it includes the SPDX SBOM - artifactName: terminal-${{ platform }}-Release - # Downloading to the source directory should ensure that the later SBOM generator can see the earlier SBOMs. - downloadPath: '$(Build.SourcesDirectory)/appx-artifacts' - # Add 3000 to the major version component, but only for the bundle. - # This is to ensure that it is newer than "2022.xx.yy.zz" or whatever the original bundle versions were before - # we switched to uniform naming. - - pwsh: |- - $VersionEpoch = 3000 - $Components = "$(XES_APPXMANIFESTVERSION)" -Split "\." - $Components[0] = ([int]$Components[0] + $VersionEpoch) - $BundleVersion = $Components -Join "." - New-Item -Type Directory "$(System.ArtifactsDirectory)\bundle" - .\build\scripts\Create-AppxBundle.ps1 -InputPath "$(Build.SourcesDirectory)/appx-artifacts" -ProjectName CascadiaPackage -BundleVersion $BundleVersion -OutputPath "$(System.ArtifactsDirectory)\bundle\$(BundleStemName)_$(XES_APPXMANIFESTVERSION)_8wekyb3d8bbwe.msixbundle" - displayName: Create WindowsTerminal*.msixbundle - - task: EsrpCodeSigning@1 - displayName: Submit *.msixbundle to ESRP for code signing - inputs: - ConnectedServiceName: 9d6d2960-0793-4d59-943e-78dcb434840a - FolderPath: $(System.ArtifactsDirectory)\bundle - Pattern: $(BundleStemName)*.msixbundle - UseMinimatch: true - signConfigType: inlineSignParams - inlineOperation: >- - [ - { - "KeyCode": "Dynamic", - "CertTemplateName": "WINMSAPP1ST", - "CertSubjectName": "CN=Microsoft Corporation, O=Microsoft Corporation, L=Redmond, S=Washington, C=US", - "OperationCode": "SigntoolSign", - "Parameters": { - "OpusName": "Microsoft", - "OpusInfo": "http://www.microsoft.com", - "FileDigest": "/fd \"SHA256\"", - "TimeStamp": "/tr \"http://rfc3161.gtm.corp.microsoft.com/TSS/HttpTspServer\" /td sha256" - }, - "ToolName": "sign", - "ToolVersion": "1.0" - }, - { - "KeyCode": "Dynamic", - "CertTemplateName": "WINMSAPP1ST", - "CertSubjectName": "CN=Microsoft Corporation, O=Microsoft Corporation, L=Redmond, S=Washington, C=US", - "OperationCode": "SigntoolVerify", - "Parameters": {}, - "ToolName": "sign", - "ToolVersion": "1.0" - } - ] - - - task: AzureArtifacts.manifest-generator-task.manifest-generator-task.ManifestGeneratorTask@0 - displayName: 'Generate SBOM manifest (bundle)' - inputs: - BuildDropPath: '$(System.ArtifactsDirectory)/bundle' - BuildComponentPath: '$(Build.SourcesDirectory)/appx-artifacts' - - - task: DropValidatorTask@0 - displayName: 'Validate bundle SBOM manifest' - inputs: - BuildDropPath: '$(System.ArtifactsDirectory)/bundle' - OutputPath: 'output.json' - ValidateSignature: true - Verbosity: 'Verbose' - - - task: PublishBuildArtifacts@1 - displayName: 'Publish Artifact: appxbundle-signed' - inputs: - PathtoPublish: $(System.ArtifactsDirectory)\bundle - ArtifactName: appxbundle-signed - -- ${{ if eq(parameters.buildConPTY, true) }}: - - job: PackageAndSignConPTY - strategy: - matrix: - ${{ each config in parameters.buildConfigurations }}: - ${{ config }}: - BuildConfiguration: ${{ config }} - displayName: Create NuGet Package (ConPTY) - dependsOn: Build - steps: - - checkout: self - clean: true - fetchDepth: 1 - submodules: true - persistCredentials: True - - task: PkgESSetupBuild@12 - displayName: Package ES - Setup Build - inputs: - disableOutputRedirect: true - - ${{ each platform in parameters.buildPlatforms }}: - - task: DownloadBuildArtifacts@1 - displayName: Download ${{ platform }} ConPTY binaries - inputs: - artifactName: conpty-dll-${{ platform }}-$(BuildConfiguration) - downloadPath: bin\${{ platform }}\$(BuildConfiguration)\ - extractTars: false - - task: PowerShell@2 - displayName: Move downloaded artifacts around - inputs: - targetType: inline - # Find all artifact files and move them up a directory. Ugh. - script: |- - Get-ChildItem bin -Recurse -Directory -Filter conpty-dll-* | % { - $_ | Get-ChildItem -Recurse -File | % { - Move-Item -Verbose $_.FullName $_.Directory.Parent.FullName - } - } - Move-Item bin\x86 bin\Win32 - - - task: EsrpCodeSigning@1 - displayName: Submit ConPTY libraries and OpenConsole for code signing - inputs: - ConnectedServiceName: 9d6d2960-0793-4d59-943e-78dcb434840a - FolderPath: '$(Build.SourcesDirectory)/bin' - signType: batchSigning - batchSignPolicyFile: '$(Build.SourcesDirectory)\build\config\ESRPSigning_ConPTY.json' - - - task: NuGetToolInstaller@1 - displayName: Use NuGet 5.10.0 - inputs: - versionSpec: 5.10.0 - - task: NuGetCommand@2 - displayName: NuGet pack - inputs: - command: pack - packagesToPack: $(Build.SourcesDirectory)\src\winconpty\package\winconpty.nuspec - packDestination: '$(Build.ArtifactStagingDirectory)/nupkg' - versioningScheme: byEnvVar - versionEnvVar: XES_PACKAGEVERSIONNUMBER - - task: EsrpCodeSigning@1 - displayName: Submit *.nupkg to ESRP for code signing - inputs: - ConnectedServiceName: 9d6d2960-0793-4d59-943e-78dcb434840a - FolderPath: $(Build.ArtifactStagingDirectory)/nupkg - Pattern: '*.nupkg' - UseMinimatch: true - signConfigType: inlineSignParams - inlineOperation: >- - [ - { - "KeyCode": "CP-401405", - "OperationCode": "NuGetSign", - "Parameters": {}, - "ToolName": "sign", - "ToolVersion": "1.0" - }, - { - "KeyCode": "CP-401405", - "OperationCode": "NuGetVerify", - "Parameters": {}, - "ToolName": "sign", - "ToolVersion": "1.0" - } - ] - - task: PublishBuildArtifacts@1 - displayName: Publish Artifact (nupkg) - inputs: - PathtoPublish: $(Build.ArtifactStagingDirectory)\nupkg - ArtifactName: conpty-nupkg-$(BuildConfiguration) - - -- ${{ if eq(parameters.buildWPF, true) }}: - - job: PackageAndSignWPF - strategy: - matrix: - ${{ each config in parameters.buildConfigurations }}: - ${{ config }}: - BuildConfiguration: ${{ config }} - displayName: Create NuGet Package (WPF Terminal Control) - dependsOn: Build - steps: - - checkout: self - clean: true - fetchDepth: 1 - submodules: true - persistCredentials: True - - task: PkgESSetupBuild@12 - displayName: Package ES - Setup Build - inputs: - disableOutputRedirect: true - - ${{ each platform in parameters.buildPlatforms }}: - - task: DownloadBuildArtifacts@1 - displayName: Download ${{ platform }} PublicTerminalCore - inputs: - artifactName: wpf-dll-${{ platform }}-$(BuildConfiguration) - itemPattern: '**/*.dll' - downloadPath: bin\${{ platform }}\$(BuildConfiguration)\ - extractTars: false - - task: PowerShell@2 - displayName: Move downloaded artifacts around - inputs: - targetType: inline - # Find all artifact files and move them up a directory. Ugh. - script: |- - Get-ChildItem bin -Recurse -Directory -Filter wpf-dll-* | % { - $_ | Get-ChildItem -Recurse -File | % { - Move-Item -Verbose $_.FullName $_.Directory.Parent.FullName - } - } - Move-Item bin\x86 bin\Win32 - - task: NuGetToolInstaller@1 - displayName: Use NuGet 5.10.0 - inputs: - versionSpec: 5.10.0 - - task: NuGetCommand@2 - displayName: NuGet restore copy - inputs: - selectOrConfig: config - nugetConfigPath: NuGet.Config - - task: VSBuild@1 - displayName: Build solution **\OpenConsole.sln for WPF Control - inputs: - solution: '**\OpenConsole.sln' - msbuildArgs: /p:WindowsTerminalReleaseBuild=$(UseReleaseBranding);Version=$(XES_PACKAGEVERSIONNUMBER) /t:Pack - platform: Any CPU - configuration: $(BuildConfiguration) - maximumCpuCount: true - - task: PublishSymbols@2 - displayName: Publish symbols path - continueOnError: True - inputs: - SearchPattern: | - $(Build.SourcesDirectory)/bin/**/*.pdb - $(Build.SourcesDirectory)/bin/**/*.exe - $(Build.SourcesDirectory)/bin/**/*.dll - IndexSources: false - SymbolServerType: TeamServices - SymbolsArtifactName: Symbols_WPF_$(BuildConfiguration) - - task: CopyFiles@2 - displayName: Copy *.nupkg to Artifacts - inputs: - Contents: '**/*Wpf*.nupkg' - TargetFolder: $(Build.ArtifactStagingDirectory)/nupkg - OverWrite: true - flattenFolders: true - - task: EsrpCodeSigning@1 - displayName: Submit *.nupkg to ESRP for code signing - inputs: - ConnectedServiceName: 9d6d2960-0793-4d59-943e-78dcb434840a - FolderPath: $(Build.ArtifactStagingDirectory)/nupkg - Pattern: '*.nupkg' - UseMinimatch: true - signConfigType: inlineSignParams - inlineOperation: >- - [ - { - "KeyCode": "CP-401405", - "OperationCode": "NuGetSign", - "Parameters": {}, - "ToolName": "sign", - "ToolVersion": "1.0" - }, - { - "KeyCode": "CP-401405", - "OperationCode": "NuGetVerify", - "Parameters": {}, - "ToolName": "sign", - "ToolVersion": "1.0" - } - ] - - task: PublishBuildArtifacts@1 - displayName: Publish Artifact (nupkg) - inputs: - PathtoPublish: $(Build.ArtifactStagingDirectory)\nupkg - ArtifactName: wpf-nupkg-$(BuildConfiguration) - -- ${{ if eq(parameters.publishSymbolsToPublic, true) }}: - - job: PublishSymbols - displayName: Publish Symbols - dependsOn: BundleAndSign - steps: - - checkout: self - clean: true - fetchDepth: 1 - submodules: true - - task: PkgESSetupBuild@12 - displayName: Package ES - Setup Build - - - template: .\templates\restore-nuget-steps.yml - - # Download the terminal-PLATFORM-CONFIG-VERSION artifact for every platform/version combo - - ${{ each platform in parameters.buildPlatforms }}: - - task: DownloadBuildArtifacts@1 - displayName: Download Symbols ${{ platform }} - inputs: - artifactName: terminal-${{ platform }}-Release - itemPattern: '**/*.appxsym' - - # It seems easier to do this -- download every appxsym -- then enumerate all the PDBs in the build directory for the - # public symbol push. Otherwise, we would have to list all of the PDB files one by one. - - pwsh: |- - mkdir $(Build.SourcesDirectory)/appxsym-temp - Get-ChildItem "$(System.ArtifactsDirectory)" -Filter *.appxsym -Recurse | % { - $src = $_.FullName - $dest = Join-Path "$(Build.SourcesDirectory)/appxsym-temp/" $_.Name - - mkdir $dest - Write-Host "Extracting $src to $dest..." - tar -x -v -f $src -C $dest - } - displayName: Extract symbols for public consumption - - - task: PowerShell@2 - displayName: Source Index PDBs (the public ones) - inputs: - filePath: build\scripts\Index-Pdbs.ps1 - arguments: -SearchDir '$(Build.SourcesDirectory)/appxsym-temp' -SourceRoot '$(Build.SourcesDirectory)' -recursive -Verbose -CommitId $(Build.SourceVersion) - pwsh: true - - # Publish the app symbols to the public MSDL symbol server - # accessible via https://msdl.microsoft.com/download/symbols - - task: PublishSymbols@2 - displayName: 'Publish app symbols to MSDL' - inputs: - symbolsFolder: '$(Build.SourcesDirectory)/appxsym-temp' - searchPattern: '**/*.pdb' - SymbolsMaximumWaitTime: 30 - SymbolServerType: 'TeamServices' - SymbolsProduct: 'Windows Terminal Application Binaries' - SymbolsVersion: '$(XES_APPXMANIFESTVERSION)' - # The ADO task does not support indexing of GitHub sources. - indexSources: false - detailedLog: true - # There is a bug which causes this task to fail if LIB includes an inaccessible path (even though it does not depend on it). - # To work around this issue, we just force LIB to be any dir that we know exists. - # Copied from https://github.com/microsoft/icu/blob/f869c214adc87415dfe751d81f42f1bca55dcf5f/build/azure-nuget.yml#L564-L583 - env: - LIB: $(Build.SourcesDirectory) - ArtifactServices_Symbol_AccountName: microsoftpublicsymbols - ArtifactServices_Symbol_PAT: $(ADO_microsoftpublicsymbols_PAT) - - -- ${{ if eq(parameters.buildTerminalVPack, true) }}: - - job: VPack - displayName: Create Windows vPack - dependsOn: BundleAndSign - steps: - - checkout: self - clean: true - fetchDepth: 1 - submodules: true - - task: PkgESSetupBuild@12 - displayName: Package ES - Setup Build - - task: DownloadBuildArtifacts@1 - displayName: Download Build Artifacts - inputs: - artifactName: appxbundle-signed - extractTars: false - - task: PowerShell@2 - displayName: Rename and stage packages for vpack - inputs: - targetType: inline - script: >- - # Rename to known/fixed name for Windows build system - - Get-ChildItem Microsoft.WindowsTerminal_*.msixbundle | Rename-Item -NewName { 'Microsoft.WindowsTerminal_8wekyb3d8bbwe.msixbundle' } - - - # Create vpack directory and place item inside - mkdir WindowsTerminal.app +stages: + - stage: Build + displayName: Build + dependsOn: [] + jobs: + - template: ./templates-v2/job-build-project.yml + parameters: + pool: + name: SHINE-INT-L # Run the compilation on the large agent pool, rather than the default small one. + demands: ImageOverride -equals SHINE-VS17-Latest + branding: ${{ parameters.branding }} + buildTerminal: ${{ parameters.buildTerminal }} + buildConPTY: ${{ parameters.buildConPTY }} + buildWPF: ${{ parameters.buildWPF }} + pgoBuildMode: ${{ parameters.pgoBuildMode }} + buildConfigurations: ${{ parameters.buildConfigurations }} + buildPlatforms: ${{ parameters.buildPlatforms }} + generateSbom: ${{ parameters.generateSbom }} + codeSign: ${{ parameters.codeSign }} + beforeBuildSteps: # Right before we build, lay down the universal package and localizations + - task: PkgESSetupBuild@12 + displayName: Package ES - Setup Build + inputs: + disableOutputRedirect: true + + - task: UniversalPackages@0 + displayName: Download terminal-internal Universal Package + inputs: + feedListDownload: 2b3f8893-a6e8-411f-b197-a9e05576da48 + packageListDownload: e82d490c-af86-4733-9dc4-07b772033204 + versionListDownload: ${{ parameters.terminalInternalPackageVersion }} + + - template: ./templates-v2/steps-fetch-and-prepare-localizations.yml + parameters: + includePseudoLoc: true + + - ${{ if eq(parameters.buildWPF, true) }}: + # Add an Any CPU build flavor for the WPF control bits + - template: ./templates-v2/job-build-project.yml + parameters: + # This job is allowed to run on the default small pool. + jobName: BuildWPF + branding: ${{ parameters.branding }} + buildTerminal: false + buildWPFDotNetComponents: true + buildConfigurations: ${{ parameters.buildConfigurations }} + buildPlatforms: + - Any CPU + generateSbom: ${{ parameters.generateSbom }} + codeSign: ${{ parameters.codeSign }} + beforeBuildSteps: + - task: PkgESSetupBuild@12 + displayName: Package ES - Setup Build + inputs: + disableOutputRedirect: true + # WPF doesn't need the localizations or the universal package, but if it does... put them here. + + - stage: Package + displayName: Package + dependsOn: [Build] + jobs: + - ${{ if eq(parameters.buildTerminal, true) }}: + - template: ./templates-v2/job-merge-msix-into-bundle.yml + parameters: + jobName: Bundle + branding: ${{ parameters.branding }} + buildConfigurations: ${{ parameters.buildConfigurations }} + buildPlatforms: ${{ parameters.buildPlatforms }} + generateSbom: ${{ parameters.generateSbom }} + codeSign: ${{ parameters.codeSign }} + + - ${{ if eq(parameters.buildConPTY, true) }}: + - template: ./templates-v2/job-package-conpty.yml + parameters: + buildConfigurations: ${{ parameters.buildConfigurations }} + buildPlatforms: ${{ parameters.buildPlatforms }} + generateSbom: ${{ parameters.generateSbom }} + codeSign: ${{ parameters.codeSign }} + + - ${{ if eq(parameters.buildWPF, true) }}: + - template: ./templates-v2/job-build-package-wpf.yml + parameters: + buildConfigurations: ${{ parameters.buildConfigurations }} + buildPlatforms: ${{ parameters.buildPlatforms }} + generateSbom: ${{ parameters.generateSbom }} + codeSign: ${{ parameters.codeSign }} + + - stage: Publish + displayName: Publish + dependsOn: [Build, Package] + jobs: + # We only support the vpack for Release builds that include Terminal + - ${{ if and(containsValue(parameters.buildConfigurations, 'Release'), parameters.buildTerminal, parameters.publishVpackToWindows) }}: + - template: ./templates-v2/job-submit-windows-vpack.yml + parameters: + buildConfiguration: Release + generateSbom: ${{ parameters.generateSbom }} + + - template: ./templates-v2/job-publish-symbols.yml + parameters: + includePublicSymbolServer: ${{ parameters.publishSymbolsToPublic }} - mv Microsoft.WindowsTerminal_8wekyb3d8bbwe.msixbundle .\WindowsTerminal.app\ - workingDirectory: $(System.ArtifactsDirectory)\appxbundle-signed - - task: PkgESVPack@12 - displayName: 'Package ES - VPack' - env: - SYSTEM_ACCESSTOKEN: $(System.AccessToken) - inputs: - sourceDirectory: $(System.ArtifactsDirectory)\appxbundle-signed\WindowsTerminal.app - description: VPack for the Windows Terminal Application - pushPkgName: WindowsTerminal.app - owner: conhost - githubToken: $(GitHubTokenForVpackProvenance) - - task: PublishPipelineArtifact@1 - displayName: 'Copy VPack Manifest to Drop' - inputs: - targetPath: $(XES_VPACKMANIFESTDIRECTORY) - artifactName: VPackManifest - - task: PkgESFCIBGit@12 - displayName: 'Submit VPack Manifest to Windows' - inputs: - configPath: '$(Build.SourcesDirectory)\build\config\GitCheckin.json' - artifactsDirectory: $(XES_VPACKMANIFESTDIRECTORY) - prTimeOut: 5 ... diff --git a/build/pipelines/templates-v2/job-build-package-wpf.yml b/build/pipelines/templates-v2/job-build-package-wpf.yml new file mode 100644 index 00000000000..fe668058b20 --- /dev/null +++ b/build/pipelines/templates-v2/job-build-package-wpf.yml @@ -0,0 +1,134 @@ +parameters: + - name: buildConfigurations + type: object + - name: buildPlatforms + type: object + - name: generateSbom + type: boolean + default: false + - name: codeSign + type: boolean + default: false + - name: pool + type: object + default: [] + - name: dependsOn + type: object + default: null + - name: artifactStem + type: string + default: '' + - name: jobName + type: string + default: PackWPF + +jobs: +- job: ${{ parameters.jobName }} + ${{ if ne(length(parameters.pool), 0) }}: + pool: ${{ parameters.pool }} + ${{ if eq(parameters.codeSign, true) }}: + displayName: Pack and Sign Microsoft.Terminal.Wpf + ${{ else }}: + displayName: Pack Microsoft.Terminal.Wpf + strategy: + matrix: + ${{ each config in parameters.buildConfigurations }}: + ${{ config }}: + BuildConfiguration: ${{ config }} + dependsOn: ${{ parameters.dependsOn }} + variables: + OutputBuildPlatform: AnyCPU + Terminal.BinDir: $(Build.SourcesDirectory)/bin/$(OutputBuildPlatform)/$(BuildConfiguration) + steps: + - checkout: self + clean: true + fetchDepth: 1 + fetchTags: false # Tags still result in depth > 1 fetch; we don't need them here + submodules: true + persistCredentials: True + + - task: PkgESSetupBuild@12 + displayName: Package ES - Setup Build + inputs: + disableOutputRedirect: true + + - template: steps-download-bin-dir-artifact.yml + parameters: + buildPlatforms: + - ${{ parameters.buildPlatforms }} + - Any CPU # Make sure we grab the precompiled WPF bits + # This build is already matrix'd on configuration, so + # just pass a single config into the download template. + buildConfigurations: + - $(BuildConfiguration) + artifactStem: ${{ parameters.artifactStem }} + + - template: .\steps-restore-nuget.yml + + - task: VSBuild@1 + displayName: Build solution OpenConsole.sln for WPF Control (Pack) + inputs: + solution: 'OpenConsole.sln' + msbuildArgs: >- + /p:WindowsTerminalReleaseBuild=true;Version=$(XES_PACKAGEVERSIONNUMBER) + /p:NoBuild=true + /p:IncludeSymbols=true + /t:Terminal\wpf\WpfTerminalControl:Pack + platform: Any CPU + configuration: $(BuildConfiguration) + maximumCpuCount: true + clean: false + + - task: CopyFiles@2 + displayName: Copy *.nupkg to Artifacts + inputs: + Contents: 'bin/**/*Wpf*.nupkg' + TargetFolder: $(Build.ArtifactStagingDirectory)/nupkg + OverWrite: true + flattenFolders: true + + - ${{ if eq(parameters.codeSign, true) }}: + - task: EsrpCodeSigning@1 + displayName: Submit *.nupkg to ESRP for code signing + inputs: + ConnectedServiceName: 9d6d2960-0793-4d59-943e-78dcb434840a + FolderPath: $(Build.ArtifactStagingDirectory)/nupkg + Pattern: '*.nupkg' + UseMinimatch: true + signConfigType: inlineSignParams + inlineOperation: >- + [ + { + "KeyCode": "CP-401405", + "OperationCode": "NuGetSign", + "Parameters": {}, + "ToolName": "sign", + "ToolVersion": "1.0" + }, + { + "KeyCode": "CP-401405", + "OperationCode": "NuGetVerify", + "Parameters": {}, + "ToolName": "sign", + "ToolVersion": "1.0" + } + ] + + - ${{ if eq(parameters.generateSbom, true) }}: + - task: AzureArtifacts.manifest-generator-task.manifest-generator-task.ManifestGeneratorTask@0 + displayName: 'Generate SBOM manifest (wpf)' + inputs: + BuildDropPath: '$(System.ArtifactsDirectory)/nupkg' + BuildComponentPath: '$(Build.SourcesDirectory)/bin' + + - task: DropValidatorTask@0 + displayName: 'Validate wpf SBOM manifest' + inputs: + BuildDropPath: '$(System.ArtifactsDirectory)/nupkg' + OutputPath: 'output.json' + ValidateSignature: true + Verbosity: 'Verbose' + + - publish: $(Build.ArtifactStagingDirectory)\nupkg + artifact: wpf-nupkg-$(BuildConfiguration)${{ parameters.artifactStem }} + displayName: Publish nupkg diff --git a/build/pipelines/templates-v2/job-build-project.yml b/build/pipelines/templates-v2/job-build-project.yml new file mode 100644 index 00000000000..1c07bb9008e --- /dev/null +++ b/build/pipelines/templates-v2/job-build-project.yml @@ -0,0 +1,263 @@ +parameters: + - name: branding + type: string + default: Dev + values: [Release, Preview, Dev] + - name: additionalBuildOptions + type: string + default: '' + - name: buildTerminal + type: boolean + default: true + - name: buildConPTY + type: boolean + default: false + - name: buildWPF + type: boolean + default: false + - name: buildWPFDotNetComponents # This weird hack is to make sure we sign and source index the .NET pieces + type: boolean + default: false + - name: buildEverything + displayName: "Build Everything (Overrides all other build options)" + type: boolean + default: false + - name: pgoBuildMode + type: string + default: None + values: [Optimize, Instrument, None] + - name: buildConfigurations + type: object + default: + - Release + - name: buildPlatforms + type: object + default: + - x64 + - x86 + - arm64 + - name: generateSbom + type: boolean + default: false + - name: codeSign + type: boolean + default: false + - name: keepAllExpensiveBuildOutputs + type: boolean + default: true + - name: artifactStem + type: string + default: '' + - name: jobName + type: string + default: 'Build' + - name: pool + type: object + default: [] + - name: beforeBuildSteps + type: stepList + default: [] + +jobs: +- job: ${{ parameters.jobName }} + ${{ if ne(length(parameters.pool), 0) }}: + pool: ${{ parameters.pool }} + strategy: + matrix: + ${{ each config in parameters.buildConfigurations }}: + ${{ each platform in parameters.buildPlatforms }}: + ${{ config }}_${{ platform }}: + BuildConfiguration: ${{ config }} + BuildPlatform: ${{ platform }} + ${{ if eq(platform, 'x86') }}: + OutputBuildPlatform: Win32 + ${{ elseif eq(platform, 'Any CPU') }}: + OutputBuildPlatform: AnyCPU + ${{ else }}: + OutputBuildPlatform: ${{ platform }} + variables: + MakeAppxPath: 'C:\Program Files (x86)\Windows Kits\10\bin\10.0.22621.0\x86\MakeAppx.exe' + Terminal.BinDir: $(Build.SourcesDirectory)/bin/$(OutputBuildPlatform)/$(BuildConfiguration) + # Azure DevOps abhors a vacuum + # If these are blank, expansion will fail later on... which will result in direct substitution of the variable *names* + # later on. We'll just... set them to a single space and if we need to, check IsNullOrWhiteSpace. + # Yup. + BuildTargetParameter: ' ' + SelectedSigningFragments: ' ' + displayName: Build + timeoutInMinutes: 240 + cancelTimeoutInMinutes: 1 + steps: + - checkout: self + clean: true + submodules: true + persistCredentials: True + # This generates either nothing for BuildTargetParameter, or /t:X;Y;Z, to control targets later. + - pwsh: |- + If (-Not [bool]::Parse("${{ parameters.buildEverything }}")) { + $BuildTargets = @() + $SignFragments = @() + If ([bool]::Parse("${{ parameters.buildTerminal }}")) { + $BuildTargets += "Terminal\CascadiaPackage" + $SignFragments += "terminal_constituents" + } + If ([bool]::Parse("${{ parameters.buildWPFDotNetComponents }}")) { + $BuildTargets += "Terminal\wpf\WpfTerminalControl" + $SignFragments += "wpfdotnet" + } + If ([bool]::Parse("${{ parameters.buildWPF }}")) { + $BuildTargets += "Terminal\wpf\PublicTerminalCore" + $SignFragments += "wpf" + } + If ([bool]::Parse("${{ parameters.buildConPTY }}")) { + $BuildTargets += "Conhost\Host_EXE;Conhost\winconpty_DLL" + $SignFragments += "conpty" + } + Write-Host "Targets: $($BuildTargets -Join ";")" + Write-Host "Sign targets: $($SignFragments -Join ";")" + Write-Host "##vso[task.setvariable variable=BuildTargetParameter]/t:$($BuildTargets -Join ";")" + Write-Host "##vso[task.setvariable variable=SelectedSigningFragments]$($SignFragments -Join ";")" + } + displayName: Prepare Build and Sign Targets + + - pwsh: |- + .\build\scripts\Generate-ThirdPartyNotices.ps1 -MarkdownNoticePath .\NOTICE.md -OutputPath .\src\cascadia\CascadiaPackage\NOTICE.html + displayName: Generate NOTICE.html from NOTICE.md + + - template: .\steps-restore-nuget.yml + + - ${{ parameters.beforeBuildSteps }} + + - task: VSBuild@1 + displayName: Build OpenConsole.sln + inputs: + solution: 'OpenConsole.sln' + msbuildArgs: >- + /p:WindowsTerminalOfficialBuild=true;WindowsTerminalBranding=${{ parameters.branding }};PGOBuildMode=${{ parameters.pgoBuildMode }} + ${{ parameters.additionalBuildOptions }} + /bl:$(Build.SourcesDirectory)\msbuild.binlog + $(BuildTargetParameter) + platform: $(BuildPlatform) + configuration: $(BuildConfiguration) + maximumCpuCount: true + + - publish: $(Build.SourcesDirectory)/msbuild.binlog + artifact: logs-$(BuildPlatform)-$(BuildConfiguration)${{ parameters.artifactStem }} + condition: always() + displayName: Publish Build Log + + # This saves ~2GiB per architecture. We won't need these later. + # Removes: + # - All .lib that do not have an associated .exp (which would indicate that they are import libs) + # - All .pdbs from those .libs (which were only used during linking) + # - Directories ending in Lib (static lib projects that we fully linked into DLLs which may also contain unnecessary resources) + # - All LocalTests_ project outputs, as they were subsumed into TestHostApp + # - All PDB files inside the WindowsTerminal/ output, which do not belong there. + - pwsh: |- + $binDir = '$(Terminal.BinDir)' + $ImportLibs = Get-ChildItem $binDir -Recurse -File -Filter '*.exp' | ForEach-Object { $_.FullName -Replace "exp$","lib" } + $StaticLibs = Get-ChildItem $binDir -Recurse -File -Filter '*.lib' | Where-Object FullName -NotIn $ImportLibs + + $Items = @() + $Items += $StaticLibs + $Items += Get-Item ($StaticLibs.FullName -Replace "lib$","pdb") -ErrorAction:Ignore + $Items += Get-ChildItem $binDir -Directory -Filter '*Lib' + $Items += Get-ChildItem $binDir -Directory -Filter 'LocalTests_*' + $Items += Get-ChildItem "${$binDir}\WindowsTerminal" -Filter '*.pdb' -ErrorAction:Ignore + + If (-Not [bool]::Parse('${{ parameters.keepAllExpensiveBuildOutputs }}')) { + $Items += Get-ChildItem '$(Terminal.BinDir)' -Filter '*.pdb' -Recurse + } + + $Items | Remove-Item -Recurse -Force -Verbose -ErrorAction:Ignore + displayName: Clean up static libs and extra symbols + errorActionPreference: silentlyContinue # It's OK if this silently fails + + # We cannot index PDBs that we have deleted! + - ${{ if eq(parameters.keepAllExpensiveBuildOutputs, true) }}: + - pwsh: |- + build\scripts\Index-Pdbs.ps1 -SearchDir '$(Terminal.BinDir)' -SourceRoot '$(Build.SourcesDirectory)' -recursive -Verbose -CommitId $(Build.SourceVersion) + displayName: Source Index PDBs + errorActionPreference: silentlyContinue + + - ${{ if or(parameters.buildTerminal, parameters.buildEverything) }}: + - pwsh: |- + $Package = (Get-ChildItem -Recurse -Filter "CascadiaPackage*.msix" | Select -First 1) + $PackageFilename = $Package.FullName + Write-Host "##vso[task.setvariable variable=WindowsTerminalPackagePath]${PackageFilename}" + displayName: Locate the MSIX + + # CHECK EXCEPTION + # PGO requires a desktop CRT + - ${{ if ne(parameters.pgoBuildMode, 'Instrument') }}: + - pwsh: |- + .\build\scripts\Test-WindowsTerminalPackage.ps1 -Verbose -Path "$(WindowsTerminalPackagePath)" + displayName: Check MSIX for common regressions + condition: and(succeeded(), ne(variables.WindowsTerminalPackagePath, '')) + + - ${{ if eq(parameters.codeSign, true) }}: + - pwsh: |- + & "$(MakeAppxPath)" unpack /p "$(WindowsTerminalPackagePath)" /d "$(Terminal.BinDir)/PackageContents" + displayName: Unpack the MSIX for signing + + - ${{ if eq(parameters.codeSign, true) }}: + - template: steps-create-signing-config.yml + parameters: + outFile: '$(Build.SourcesDirectory)/ESRPSigningConfig.json' + stage: build + fragments: $(SelectedSigningFragments) + + # Code-sign everything we just put together. + # We run the signing in Terminal.BinDir, because all of the signing batches are relative to the final architecture/configuration output folder. + - task: EsrpCodeSigning@1 + displayName: Submit Signing Request + inputs: + ConnectedServiceName: 9d6d2960-0793-4d59-943e-78dcb434840a + FolderPath: '$(Terminal.BinDir)' + signType: batchSigning + batchSignPolicyFile: '$(Build.SourcesDirectory)/ESRPSigningConfig.json' + + # We only need to re-pack the MSIX if we actually signed, so this can stay in the codeSign conditional + - ${{ if or(parameters.buildTerminal, parameters.buildEverything) }}: + - pwsh: |- + $outDir = New-Item -Type Directory "$(Terminal.BinDir)/_appx" -ErrorAction:Ignore + $PackageFilename = Join-Path $outDir.FullName (Split-Path -Leaf "$(WindowsTerminalPackagePath)") + & "$(MakeAppxPath)" pack /h SHA256 /o /p $PackageFilename /d "$(Terminal.BinDir)/PackageContents" + Write-Host "##vso[task.setvariable variable=WindowsTerminalPackagePath]${PackageFilename}" + displayName: Re-pack the new Terminal package after signing + + - ${{ else }}: # No Signing + - ${{ if or(parameters.buildTerminal, parameters.buildEverything) }}: + - pwsh: |- + $outDir = New-Item -Type Directory "$(Terminal.BinDir)/_appx" -ErrorAction:Ignore + $PackageFilename = Join-Path $outDir.FullName (Split-Path -Leaf "$(WindowsTerminalPackagePath)") + Copy-Item "$(WindowsTerminalPackagePath)" $PackageFilename + Write-Host "##vso[task.setvariable variable=WindowsTerminalPackagePath]${PackageFilename}" + displayName: Stage the package (unsigned) + condition: and(succeeded(), ne(variables.WindowsTerminalPackagePath, '')) + + - ${{ if or(parameters.buildTerminal, parameters.buildEverything) }}: + - pwsh: |- + $XamlAppxPath = (Get-Item "src\cascadia\CascadiaPackage\AppPackages\*\Dependencies\$(BuildPlatform)\Microsoft.UI.Xaml*.appx").FullName + $outDir = New-Item -Type Directory "$(Terminal.BinDir)/_unpackaged" -ErrorAction:Ignore + & .\build\scripts\New-UnpackagedTerminalDistribution.ps1 -TerminalAppX $(WindowsTerminalPackagePath) -XamlAppX $XamlAppxPath -Destination $outDir.FullName + displayName: Build Unpackaged Distribution (from MSIX) + condition: and(succeeded(), ne(variables.WindowsTerminalPackagePath, '')) + + - ${{ if eq(parameters.generateSbom, true) }}: + - task: AzureArtifacts.manifest-generator-task.manifest-generator-task.ManifestGeneratorTask@0 + displayName: 'Generate SBOM manifest' + inputs: + BuildDropPath: '$(Terminal.BinDir)' + + - task: DropValidatorTask@0 + displayName: 'Validate SBOM manifest' + inputs: + BuildDropPath: '$(Terminal.BinDir)' + OutputPath: 'output.json' + ValidateSignature: true + Verbosity: 'Verbose' + + - publish: $(Terminal.BinDir) + artifact: build-$(BuildPlatform)-$(BuildConfiguration)${{ parameters.artifactStem }} + displayName: Publish All Outputs diff --git a/build/pipelines/templates-v2/job-check-code-format.yml b/build/pipelines/templates-v2/job-check-code-format.yml new file mode 100644 index 00000000000..e889fc8756d --- /dev/null +++ b/build/pipelines/templates-v2/job-check-code-format.yml @@ -0,0 +1,15 @@ +jobs: +- job: CodeFormatCheck + displayName: Check Code Format + pool: { vmImage: windows-2022 } + + steps: + - checkout: self + fetchDepth: 1 + fetchTags: false # Tags still result in depth > 1 fetch; we don't need them here + submodules: false + clean: true + + - powershell: |- + .\build\scripts\Invoke-FormattingCheck.ps1 + displayName: 'Run formatters' diff --git a/build/pipelines/templates/codenav-indexer.yml b/build/pipelines/templates-v2/job-index-github-codenav.yml similarity index 57% rename from build/pipelines/templates/codenav-indexer.yml rename to build/pipelines/templates-v2/job-index-github-codenav.yml index 04e018ed2a0..e2edf55e651 100644 --- a/build/pipelines/templates/codenav-indexer.yml +++ b/build/pipelines/templates-v2/job-index-github-codenav.yml @@ -1,21 +1,15 @@ -parameters: - artifactName: 'drop' - jobs: - job: CodeNavIndexer displayName: Run Github CodeNav Indexer - pool: { vmImage: windows-2019 } + pool: { vmImage: windows-2022 } steps: - checkout: self fetchDepth: 1 + fetchTags: false # Tags still result in depth > 1 fetch; we don't need them here submodules: false clean: true - - task: DownloadBuildArtifacts@0 - inputs: - artifactName: ${{ parameters.artifactName }} - - task: RichCodeNavIndexer@0 inputs: languages: 'cpp,csharp' diff --git a/build/pipelines/templates-v2/job-merge-msix-into-bundle.yml b/build/pipelines/templates-v2/job-merge-msix-into-bundle.yml new file mode 100644 index 00000000000..318d3b2673a --- /dev/null +++ b/build/pipelines/templates-v2/job-merge-msix-into-bundle.yml @@ -0,0 +1,133 @@ +parameters: + - name: branding + type: string + - name: buildConfigurations + type: object + - name: buildPlatforms + type: object + - name: generateSbom + type: boolean + default: false + - name: codeSign + type: boolean + default: false + - name: pool + type: object + default: [] + - name: dependsOn + type: object + default: null + - name: artifactStem + type: string + default: '' + - name: jobName + type: string + default: Bundle + +jobs: +- job: ${{ parameters.jobName }} + ${{ if ne(length(parameters.pool), 0) }}: + pool: ${{ parameters.pool }} + ${{ if eq(parameters.codeSign, true) }}: + displayName: Pack and Sign Terminal MSIXBundle + ${{ else }}: + displayName: Pack Terminal MSIXBundle + strategy: + matrix: + ${{ each config in parameters.buildConfigurations }}: + ${{ config }}: + BuildConfiguration: ${{ config }} + variables: + ${{ if eq(parameters.branding, 'Release') }}: + BundleStemName: Microsoft.WindowsTerminal + ${{ elseif eq(parameters.branding, 'Preview') }}: + BundleStemName: Microsoft.WindowsTerminalPreview + ${{ else }}: + BundleStemName: WindowsTerminalDev + dependsOn: ${{ parameters.dependsOn }} + steps: + - checkout: self + clean: true + fetchDepth: 1 + fetchTags: false # Tags still result in depth > 1 fetch; we don't need them here + submodules: true + persistCredentials: True + - task: PkgESSetupBuild@12 + displayName: Package ES - Setup Build + inputs: + disableOutputRedirect: true + - template: steps-download-bin-dir-artifact.yml + parameters: + buildPlatforms: ${{ parameters.buildPlatforms }} + # This build is already matrix'd on configuration, so + # just pass a single config into the download template. + buildConfigurations: + - $(BuildConfiguration) + artifactStem: ${{ parameters.artifactStem }} + + # Add 3000 to the major version component, but only for the bundle. + # This is to ensure that it is newer than "2022.xx.yy.zz" or whatever the original bundle versions were before + # we switched to uniform naming. + - pwsh: |- + $VersionEpoch = 3000 + $Components = "$(XES_APPXMANIFESTVERSION)" -Split "\." + $Components[0] = ([int]$Components[0] + $VersionEpoch) + $BundleVersion = $Components -Join "." + New-Item -Type Directory "$(System.ArtifactsDirectory)/bundle" + .\build\scripts\Create-AppxBundle.ps1 -InputPath 'bin/' -ProjectName CascadiaPackage -BundleVersion $BundleVersion -OutputPath "$(System.ArtifactsDirectory)\bundle\$(BundleStemName)_$(XES_APPXMANIFESTVERSION)_8wekyb3d8bbwe.msixbundle" + displayName: Create msixbundle + + - ${{ if eq(parameters.codeSign, true) }}: + - task: EsrpCodeSigning@1 + displayName: Submit *.msixbundle to ESRP for code signing + inputs: + ConnectedServiceName: 9d6d2960-0793-4d59-943e-78dcb434840a + FolderPath: $(System.ArtifactsDirectory)\bundle + Pattern: $(BundleStemName)*.msixbundle + UseMinimatch: true + signConfigType: inlineSignParams + inlineOperation: >- + [ + { + "KeyCode": "Dynamic", + "CertTemplateName": "WINMSAPP1ST", + "CertSubjectName": "CN=Microsoft Corporation, O=Microsoft Corporation, L=Redmond, S=Washington, C=US", + "OperationCode": "SigntoolSign", + "Parameters": { + "OpusName": "Microsoft", + "OpusInfo": "http://www.microsoft.com", + "FileDigest": "/fd \"SHA256\"", + "TimeStamp": "/tr \"http://rfc3161.gtm.corp.microsoft.com/TSS/HttpTspServer\" /td sha256" + }, + "ToolName": "sign", + "ToolVersion": "1.0" + }, + { + "KeyCode": "Dynamic", + "CertTemplateName": "WINMSAPP1ST", + "CertSubjectName": "CN=Microsoft Corporation, O=Microsoft Corporation, L=Redmond, S=Washington, C=US", + "OperationCode": "SigntoolVerify", + "Parameters": {}, + "ToolName": "sign", + "ToolVersion": "1.0" + } + ] + + - ${{ if eq(parameters.generateSbom, true) }}: + - task: AzureArtifacts.manifest-generator-task.manifest-generator-task.ManifestGeneratorTask@0 + displayName: 'Generate SBOM manifest (bundle)' + inputs: + BuildDropPath: '$(System.ArtifactsDirectory)/bundle' + BuildComponentPath: '$(Build.SourcesDirectory)/bin' + + - task: DropValidatorTask@0 + displayName: 'Validate bundle SBOM manifest' + inputs: + BuildDropPath: '$(System.ArtifactsDirectory)/bundle' + OutputPath: 'output.json' + ValidateSignature: true + Verbosity: 'Verbose' + + - publish: '$(System.ArtifactsDirectory)/bundle' + artifact: appxbundle-$(BuildConfiguration)${{ parameters.artifactStem }} + displayName: Publish msixbundle diff --git a/build/pipelines/templates-v2/job-package-conpty.yml b/build/pipelines/templates-v2/job-package-conpty.yml new file mode 100644 index 00000000000..db648d905f2 --- /dev/null +++ b/build/pipelines/templates-v2/job-package-conpty.yml @@ -0,0 +1,118 @@ +parameters: + - name: buildConfigurations + type: object + - name: buildPlatforms + type: object + - name: generateSbom + type: boolean + default: false + - name: codeSign + type: boolean + default: false + - name: pool + type: object + default: [] + - name: dependsOn + type: object + default: null + - name: artifactStem + type: string + default: '' + - name: jobName + type: string + default: PackConPTY + +jobs: +- job: ${{ parameters.jobName }} + ${{ if ne(length(parameters.pool), 0) }}: + pool: ${{ parameters.pool }} + ${{ if eq(parameters.codeSign, true) }}: + displayName: Pack and Sign Microsoft.Windows.Console.ConPTY + ${{ else }}: + displayName: Pack Microsoft.Windows.Console.ConPTY + strategy: + matrix: + ${{ each config in parameters.buildConfigurations }}: + ${{ config }}: + BuildConfiguration: ${{ config }} + dependsOn: ${{ parameters.dependsOn }} + steps: + - checkout: self + clean: true + fetchDepth: 1 + fetchTags: false # Tags still result in depth > 1 fetch; we don't need them here + submodules: true + persistCredentials: True + + - task: PkgESSetupBuild@12 + displayName: Package ES - Setup Build + inputs: + disableOutputRedirect: true + + - template: steps-download-bin-dir-artifact.yml + parameters: + buildPlatforms: ${{ parameters.buildPlatforms }} + # This build is already matrix'd on configuration, so + # just pass a single config into the download template. + buildConfigurations: + - $(BuildConfiguration) + artifactStem: ${{ parameters.artifactStem }} + + - template: steps-ensure-nuget-version.yml + +# In the Microsoft Azure DevOps tenant, NuGetCommand is ambiguous. +# This should be `task: NuGetCommand@2` + - task: 333b11bd-d341-40d9-afcf-b32d5ce6f23b@2 + displayName: NuGet pack + inputs: + command: pack + packagesToPack: $(Build.SourcesDirectory)\src\winconpty\package\winconpty.nuspec + packDestination: '$(Build.ArtifactStagingDirectory)/nupkg' + versioningScheme: byEnvVar + versionEnvVar: XES_PACKAGEVERSIONNUMBER + + - ${{ if eq(parameters.codeSign, true) }}: + - task: EsrpCodeSigning@1 + displayName: Submit *.nupkg to ESRP for code signing + inputs: + ConnectedServiceName: 9d6d2960-0793-4d59-943e-78dcb434840a + FolderPath: $(Build.ArtifactStagingDirectory)/nupkg + Pattern: '*.nupkg' + UseMinimatch: true + signConfigType: inlineSignParams + inlineOperation: >- + [ + { + "KeyCode": "CP-401405", + "OperationCode": "NuGetSign", + "Parameters": {}, + "ToolName": "sign", + "ToolVersion": "1.0" + }, + { + "KeyCode": "CP-401405", + "OperationCode": "NuGetVerify", + "Parameters": {}, + "ToolName": "sign", + "ToolVersion": "1.0" + } + ] + + - ${{ if eq(parameters.generateSbom, true) }}: + - task: AzureArtifacts.manifest-generator-task.manifest-generator-task.ManifestGeneratorTask@0 + displayName: 'Generate SBOM manifest (conpty)' + inputs: + BuildDropPath: '$(System.ArtifactsDirectory)/nupkg' + BuildComponentPath: '$(Build.SourcesDirectory)/bin' + + - task: DropValidatorTask@0 + displayName: 'Validate conpty SBOM manifest' + inputs: + BuildDropPath: '$(System.ArtifactsDirectory)/nupkg' + OutputPath: 'output.json' + ValidateSignature: true + Verbosity: 'Verbose' + + - publish: $(Build.ArtifactStagingDirectory)\nupkg + artifact: conpty-nupkg-$(BuildConfiguration)${{ parameters.artifactStem }} + displayName: Publish nupkg diff --git a/build/pipelines/templates/pgo-build-and-publish-nuget-job.yml b/build/pipelines/templates-v2/job-pgo-build-nuget-and-publish.yml similarity index 57% rename from build/pipelines/templates/pgo-build-and-publish-nuget-job.yml rename to build/pipelines/templates-v2/job-pgo-build-nuget-and-publish.yml index 2c36f7cf158..b61a22c4fc1 100644 --- a/build/pipelines/templates/pgo-build-and-publish-nuget-job.yml +++ b/build/pipelines/templates-v2/job-pgo-build-nuget-and-publish.yml @@ -1,14 +1,26 @@ -# From our friends at MUX: https://github.com/microsoft/microsoft-ui-xaml/blob/main/build/AzurePipelinesTemplates/MUX-BuildAndPublishPGONuGet-Job.yml - parameters: - dependsOn: '' - pgoArtifact: PGO + - name: buildConfiguration + type: string + - name: pool + type: object + default: [] + - name: dependsOn + type: object + default: null + - name: artifactStem + type: string + default: '' + - name: jobName + type: string + default: BuildAndPublishPGONuget jobs: -- job: BuildAndPublishPGONuGet +- job: ${{ parameters.jobName }} + ${{ if ne(length(parameters.pool), 0) }}: + pool: ${{ parameters.pool }} dependsOn: ${{ parameters.dependsOn }} - pool: - vmImage: 'windows-2019' + displayName: Package and Publish PGO Databases + variables: artifactsPath: $(Build.SourcesDirectory)\Artifacts pgoToolsPath: $(Build.SourcesDirectory)\build\PGO @@ -16,20 +28,25 @@ jobs: nuspecFilename: PGO.nuspec steps: - - task: DownloadBuildArtifacts@0 + - checkout: self + clean: true + # It is important that this be 0, otherwise git will not fetch the branch ref names that the PGO rules require. + fetchDepth: 0 + submodules: false + persistCredentials: false + + - task: DownloadPipelineArtifact@2 + displayName: Download Final PGO Databases inputs: - artifactName: ${{ parameters.pgoArtifact }} + artifact: pgd-merged-${{ parameters.buildConfiguration }}${{ parameters.artifactStem }} downloadPath: $(artifactsPath) + - template: steps-ensure-nuget-version.yml + - task: NuGetAuthenticate@0 inputs: nuGetServiceConnections: 'Terminal Public Artifact Feed' - - task: NuGetToolInstaller@0 - displayName: 'Use NuGet 5.8.0' - inputs: - versionSpec: 5.8.0 - # In the Microsoft Azure DevOps tenant, NuGetCommand is ambiguous. # This should be `task: NuGetCommand@2` - task: 333b11bd-d341-40d9-afcf-b32d5ce6f23b@2 @@ -45,12 +62,11 @@ jobs: displayName: 'Create PGO Nuget' inputs: solution: $(pgoToolsPath)\PGO.DB.proj - msbuildArguments: '/t:CreatePGONuGet /p:PGOBuildMode=Instrument /p:PGDPathForAllArch=$(artifactsPath)\${{ parameters.pgoArtifact }} /p:PGOOutputPath=$(Build.ArtifactStagingDirectory)' + msbuildArguments: '/t:CreatePGONuGet /p:PGOBuildMode=Instrument /p:PGDPathForAllArch=$(artifactsPath) /p:PGOOutputPath=$(Build.ArtifactStagingDirectory)' - - task: PublishBuildArtifacts@1 - inputs: - pathToPublish: $(Build.ArtifactStagingDirectory) - artifactName: ${{ parameters.pgoArtifact }} + - publish: $(Build.ArtifactStagingDirectory) + artifact: pgo-nupkg-${{ parameters.buildConfiguration }}${{ parameters.artifactStem }} + displayName: "Publish Pipeline Artifact" - task: 333b11bd-d341-40d9-afcf-b32d5ce6f23b@2 displayName: 'NuGet push' diff --git a/build/pipelines/templates-v2/job-pgo-merge-pgd.yml b/build/pipelines/templates-v2/job-pgo-merge-pgd.yml new file mode 100644 index 00000000000..8b341b463eb --- /dev/null +++ b/build/pipelines/templates-v2/job-pgo-merge-pgd.yml @@ -0,0 +1,75 @@ +parameters: + - name: buildConfiguration + type: string + - name: buildPlatforms + type: object + - name: pool + type: object + default: [] + - name: dependsOn + type: object + default: null + - name: artifactStem + type: string + default: '' + - name: jobName + type: string + default: MergePGD + +jobs: +- job: ${{ parameters.jobName }} + ${{ if ne(length(parameters.pool), 0) }}: + pool: ${{ parameters.pool }} + dependsOn: ${{ parameters.dependsOn }} + displayName: Merge PGO Counts for ${{ parameters.buildConfiguration }} + + steps: + # The environment variable VCToolsInstallDir isn't defined on lab machines, so we need to retrieve it ourselves. + - script: | + "%ProgramFiles(x86)%\Microsoft Visual Studio\Installer\vswhere.exe" -Latest -requires Microsoft.Component.MSBuild -property InstallationPath > %TEMP%\vsinstalldir.txt + set /p _VSINSTALLDIR15=<%TEMP%\vsinstalldir.txt + del %TEMP%\vsinstalldir.txt + call "%_VSINSTALLDIR15%\Common7\Tools\VsDevCmd.bat" + echo VCToolsInstallDir = %VCToolsInstallDir% + echo ##vso[task.setvariable variable=VCToolsInstallDir]%VCToolsInstallDir% + displayName: 'Retrieve VC tools directory' + + - ${{ each platform in parameters.buildPlatforms }}: + - task: DownloadPipelineArtifact@2 + displayName: Download PGO Databases for ${{ platform }} + inputs: + artifactName: build-${{ platform }}-${{ parameters.buildConfiguration }}${{ parameters.artifactStem }} + itemPattern: '**/*.pgd' + downloadPath: '$(Build.SourcesDirectory)/pgd/${{ platform }}/${{ parameters.buildConfiguration }}' + - task: DownloadPipelineArtifact@2 + displayName: Download PGO Counts for ${{ platform }} + inputs: + artifactName: pgc-intermediates-${{ platform }}-${{ parameters.buildConfiguration }}${{ parameters.artifactStem }} + downloadPath: '$(Build.SourcesDirectory)/pgc/${{ platform }}/${{ parameters.buildConfiguration }}' + - pwsh: |- + $Arch = '${{ platform }}' + $Conf = '${{ parameters.buildConfiguration }}' + $PGCDir = '$(Build.SourcesDirectory)/pgc/${{ platform }}/${{ parameters.buildConfiguration }}' + $PGDDir = '$(Build.SourcesDirectory)/pgd/${{ platform }}/${{ parameters.buildConfiguration }}' + # Flatten the PGD directory + Get-ChildItem $PGDDir -Recurse -Filter *.pgd | Move-Item -Destination $PGDDir -Verbose + Get-ChildItem $PGCDir -Filter *.pgc | + ForEach-Object { + $Parts = $_.Name -Split "!"; + $_ | Add-Member Module $Parts[0] -PassThru + } | + Group-Object Module | + ForEach-Object { + & "$(VCToolsInstallDir)\bin\Hostx64\${{ platform }}\pgomgr.exe" /merge $_.Group.FullName "$PGDDir\$($_.Name).pgd" + } + displayName: Merge PGO Counts for ${{ platform }} + - task: CopyFiles@2 + displayName: 'Copy merged pgds to artifact staging' + inputs: + sourceFolder: '$(Build.SourcesDirectory)/pgd/${{ platform }}/${{ parameters.buildConfiguration }}' + contents: '**\*.pgd' + targetFolder: '$(Build.ArtifactStagingDirectory)\out-pgd\${{ platform }}' + + - publish: $(Build.ArtifactStagingDirectory)\out-pgd + artifact: pgd-merged-${{ parameters.buildConfiguration }}${{ parameters.artifactStem }} + displayName: "Publish merged PGDs" diff --git a/build/pipelines/templates-v2/job-publish-symbols.yml b/build/pipelines/templates-v2/job-publish-symbols.yml new file mode 100644 index 00000000000..6c37f0f4cbd --- /dev/null +++ b/build/pipelines/templates-v2/job-publish-symbols.yml @@ -0,0 +1,81 @@ +parameters: + - name: includePublicSymbolServer + type: boolean + default: false + - name: pool + type: object + default: [] + - name: dependsOn + type: object + default: null + - name: artifactStem + type: string + default: '' + - name: jobName + type: string + default: PublishSymbols + +jobs: +- job: ${{ parameters.jobName }} + ${{ if ne(length(parameters.pool), 0) }}: + pool: ${{ parameters.pool }} + ${{ if eq(parameters.includePublicSymbolServer, true) }}: + displayName: Publish Symbols to Internal and MSDL + ${{ else }}: + displayName: Publish Symbols Internally + dependsOn: ${{ parameters.dependsOn }} + steps: + - checkout: self + clean: true + fetchDepth: 1 + fetchTags: false # Tags still result in depth > 1 fetch; we don't need them here + submodules: true + persistCredentials: True + + - task: PkgESSetupBuild@12 + displayName: Package ES - Setup Build + inputs: + disableOutputRedirect: true + + - task: DownloadPipelineArtifact@2 + displayName: Download all PDBs from all prior build phases + inputs: + itemPattern: '**/*.pdb' + targetPath: '$(Build.SourcesDirectory)/bin' + + - task: PublishSymbols@2 + displayName: Publish Symbols (to current Azure DevOps tenant) + continueOnError: True + inputs: + SymbolsFolder: '$(Build.SourcesDirectory)/bin' + SearchPattern: '**/*.pdb' + IndexSources: false + DetailedLog: true + SymbolsMaximumWaitTime: 30 + SymbolServerType: 'TeamServices' + SymbolsProduct: 'Windows Terminal Converged Symbols' + SymbolsVersion: '$(XES_APPXMANIFESTVERSION)' + env: + LIB: $(Build.SourcesDirectory) + + - ${{ if eq(parameters.includePublicSymbolServer, true) }}: + - task: PublishSymbols@2 + displayName: 'Publish symbols to MSDL' + continueOnError: True + inputs: + SymbolsFolder: '$(Build.SourcesDirectory)/bin' + SearchPattern: '**/*.pdb' + IndexSources: false + DetailedLog: true + SymbolsMaximumWaitTime: 30 + SymbolServerType: 'TeamServices' + SymbolsProduct: 'Windows Terminal Converged Symbols' + SymbolsVersion: '$(XES_APPXMANIFESTVERSION)' + # The ADO task does not support indexing of GitHub sources. + # There is a bug which causes this task to fail if LIB includes an inaccessible path (even though it does not depend on it). + # To work around this issue, we just force LIB to be any dir that we know exists. + # Copied from https://github.com/microsoft/icu/blob/f869c214adc87415dfe751d81f42f1bca55dcf5f/build/azure-nuget.yml#L564-L583 + env: + LIB: $(Build.SourcesDirectory) + ArtifactServices_Symbol_AccountName: microsoftpublicsymbols + ArtifactServices_Symbol_PAT: $(ADO_microsoftpublicsymbols_PAT) diff --git a/build/pipelines/templates-v2/job-run-pgo-tests.yml b/build/pipelines/templates-v2/job-run-pgo-tests.yml new file mode 100644 index 00000000000..5270ca1bf85 --- /dev/null +++ b/build/pipelines/templates-v2/job-run-pgo-tests.yml @@ -0,0 +1,83 @@ +parameters: + buildConfiguration: 'Release' + buildPlatform: '' + artifactStem: '' + testLogPath: '$(Build.BinariesDirectory)\$(BuildPlatform)\$(BuildConfiguration)\testsOnBuildMachine.wtl' + +jobs: +- job: PGO${{ parameters.buildPlatform }}${{ parameters.buildConfiguration }} + displayName: PGO ${{ parameters.buildPlatform }} ${{ parameters.buildConfiguration }} + variables: + BuildConfiguration: ${{ parameters.buildConfiguration }} + BuildPlatform: ${{ parameters.buildPlatform }} + OutputBuildPlatform: ${{ parameters.buildPlatform }} + Terminal.BinDir: $(Build.SourcesDirectory)/bin/$(OutputBuildPlatform)/$(BuildConfiguration) + pool: + ${{ if eq(variables['System.CollectionUri'], 'https://dev.azure.com/ms/') }}: + ${{ if ne(parameters.buildPlatform, 'ARM64') }}: + name: SHINE-OSS-Testing-x64 + ${{ else }}: + name: SHINE-OSS-Testing-arm64 + ${{ if ne(variables['System.CollectionUri'], 'https://dev.azure.com/ms/') }}: + ${{ if ne(parameters.buildPlatform, 'ARM64') }}: + name: SHINE-INT-Testing-x64 + ${{ else }}: + name: SHINE-INT-Testing-arm64 + + steps: + - checkout: self + submodules: false + clean: true + fetchDepth: 1 + fetchTags: false # Tags still result in depth > 1 fetch; we don't need them here + + - task: DownloadPipelineArtifact@2 + displayName: Download artifacts + inputs: + artifactName: build-${{ parameters.buildPlatform }}-$(BuildConfiguration)${{ parameters.artifactStem }} + downloadPath: $(Terminal.BinDir) + + # The tests expect Terminal to be an unpackaged distribution named terminal-0.0.1.0 (after the dev build version scheme) + # Extract to that folder explicitly and strip the embedded folder name from the unpackaged archive. + - powershell: |- + $TargetDirectory = New-Item -Type Directory (Join-Path "$(Terminal.BinDir)" "terminal-0.0.1.0") + & tar.exe -x -v -f (Get-Item "$(Terminal.BinDir)/_unpackaged/*.zip") -C $TargetDirectory.FullName --strip-components=1 + displayName: Extract the unpackaged build for PGO + + - template: steps-ensure-nuget-version.yml + + - powershell: |- + $Package = 'Microsoft.Internal.Windows.Terminal.TestContent' + $Version = '1.0.1' + & nuget.exe install $Package -Version $Version + Write-Host "##vso[task.setvariable variable=TerminalTestContentPath]$(Build.SourcesDirectory)\packages\${Package}.${Version}\content" + displayName: Install Test Content + + - task: PowerShell@2 + displayName: 'Run PGO Tests' + inputs: + targetType: filePath + filePath: build\scripts\Run-Tests.ps1 + arguments: >- + -MatchPattern '*UIA.Tests.dll' + -Platform '$(OutputBuildPlatform)' + -Configuration '$(BuildConfiguration)' + -LogPath '${{ parameters.testLogPath }}' + -Root "$(Terminal.BinDir)" + -AdditionalTaefArguments '/select:(@IsPGO=true)','/p:WTTestContent=$(TerminalTestContentPath)' + condition: and(succeeded(), ne(variables['PGOBuildMode'], 'Instrument')) + + - task: CopyFiles@2 + displayName: 'Copy PGO outputs to Artifacts' + condition: always() + inputs: + Contents: | + **/*.pgc + ${{ parameters.testLogPath }} + TargetFolder: '$(Build.ArtifactStagingDirectory)/$(BuildConfiguration)/$(BuildPlatform)/pgc' + OverWrite: true + flattenFolders: true + + - publish: '$(Build.ArtifactStagingDirectory)/$(BuildConfiguration)/$(BuildPlatform)/pgc' + artifact: pgc-intermediates-$(BuildPlatform)-$(BuildConfiguration)${{ parameters.artifactStem }} + condition: always() diff --git a/build/pipelines/templates-v2/job-submit-windows-vpack.yml b/build/pipelines/templates-v2/job-submit-windows-vpack.yml new file mode 100644 index 00000000000..2bed2899cf0 --- /dev/null +++ b/build/pipelines/templates-v2/job-submit-windows-vpack.yml @@ -0,0 +1,70 @@ +parameters: + - name: buildConfiguration + type: string + - name: generateSbom + type: boolean + default: false + - name: pool + type: object + default: [] + - name: dependsOn + type: object + default: null + - name: artifactStem + type: string + default: '' + +jobs: +- job: VPack + ${{ if ne(length(parameters.pool), 0) }}: + pool: ${{ parameters.pool }} + displayName: Create and Submit Windows vPack + dependsOn: ${{ parameters.dependsOn }} + steps: + - checkout: self + clean: true + fetchDepth: 1 + fetchTags: false # Tags still result in depth > 1 fetch; we don't need them here + submodules: true + persistCredentials: True + + - task: PkgESSetupBuild@12 + displayName: Package ES - Setup Build + inputs: + disableOutputRedirect: true + + - task: DownloadPipelineArtifact@2 + displayName: Download MSIX Bundle Artifact + inputs: + artifactName: appxbundle-${{ parameters.buildConfiguration }}${{ parameters.artifactStem }} + downloadPath: '$(Build.SourcesDirectory)/bundle' + + # Rename to known/fixed name for Windows build system + - powershell: |- + # Create vpack directory and place item inside + $TargetFolder = New-Item -Type Directory '$(Build.SourcesDirectory)/WindowsTerminal.app' + Get-ChildItem bundle/Microsoft.WindowsTerminal_*.msixbundle | Move-Item (Join-Path $TargetFolder.FullName 'Microsoft.WindowsTerminal_8wekyb3d8bbwe.msixbundle') -Verbose + displayName: Stage packages for vpack + + - task: PkgESVPack@12 + displayName: 'Package ES - VPack' + env: + SYSTEM_ACCESSTOKEN: $(System.AccessToken) + inputs: + sourceDirectory: '$(Build.SourcesDirectory)/WindowsTerminal.app' + description: VPack for the Windows Terminal Application + pushPkgName: WindowsTerminal.app + owner: conhost + githubToken: $(GitHubTokenForVpackProvenance) + + - publish: $(XES_VPACKMANIFESTDIRECTORY) + artifact: vpack-manifest${{ parameters.artifactStem }} + displayName: 'Publish VPack Manifest to Drop' + + - task: PkgESFCIBGit@12 + displayName: 'Submit VPack Manifest to Windows' + inputs: + configPath: '$(Build.SourcesDirectory)\build\config\GitCheckin.json' + artifactsDirectory: $(XES_VPACKMANIFESTDIRECTORY) + prTimeOut: 5 + diff --git a/build/pipelines/templates/test-console-ci.yml b/build/pipelines/templates-v2/job-test-project.yml similarity index 72% rename from build/pipelines/templates/test-console-ci.yml rename to build/pipelines/templates-v2/job-test-project.yml index 5a7db6475bd..483a32893ce 100644 --- a/build/pipelines/templates/test-console-ci.yml +++ b/build/pipelines/templates-v2/job-test-project.yml @@ -1,9 +1,8 @@ parameters: configuration: 'Release' platform: '' - additionalBuildArguments: '' - artifactName: 'drop' testLogPath: '$(Build.BinariesDirectory)\$(BuildPlatform)\$(BuildConfiguration)\testsOnBuildMachine.wtl' + artifactStem: '' jobs: - job: Test${{ parameters.platform }}${{ parameters.configuration }} @@ -11,6 +10,11 @@ jobs: variables: BuildConfiguration: ${{ parameters.configuration }} BuildPlatform: ${{ parameters.platform }} + ${{ if eq(parameters.platform, 'x86') }}: + OutputBuildPlatform: Win32 + ${{ else }}: + OutputBuildPlatform: ${{ parameters.platform }} + Terminal.BinDir: $(Build.SourcesDirectory)/bin/$(OutputBuildPlatform)/$(BuildConfiguration) pool: ${{ if eq(variables['System.CollectionUri'], 'https://dev.azure.com/ms/') }}: ${{ if ne(parameters.platform, 'ARM64') }}: @@ -28,26 +32,20 @@ jobs: submodules: false clean: true fetchDepth: 1 + fetchTags: false # Tags still result in depth > 1 fetch; we don't need them here - - task: DownloadBuildArtifacts@0 + - task: DownloadPipelineArtifact@2 + displayName: Download artifacts inputs: - artifactName: ${{ parameters.artifactName }} - - - task: PowerShell@2 - displayName: 'Rationalize build platform' - inputs: - targetType: inline - script: | - $Arch = "$(BuildPlatform)" - If ($Arch -Eq "x86") { $Arch = "Win32" } - Write-Host "##vso[task.setvariable variable=RationalizedBuildPlatform]${Arch}" + artifactName: build-${{ parameters.platform }}-$(BuildConfiguration)${{ parameters.artifactStem }} + downloadPath: $(Terminal.BinDir) - task: PowerShell@2 displayName: 'Run Unit Tests' inputs: targetType: filePath filePath: build\scripts\Run-Tests.ps1 - arguments: -MatchPattern '*unit.test*.dll' -Platform '$(RationalizedBuildPlatform)' -Configuration '$(BuildConfiguration)' -LogPath '${{ parameters.testLogPath }}' -Root "$(System.ArtifactsDirectory)\\${{ parameters.artifactName }}\\$(BuildConfiguration)\\$(BuildPlatform)\\test" + arguments: -MatchPattern '*unit.test*.dll' -Platform '$(OutputBuildPlatform)' -Configuration '$(BuildConfiguration)' -LogPath '${{ parameters.testLogPath }}' -Root "$(Terminal.BinDir)" condition: and(succeeded(), ne(variables['PGOBuildMode'], 'Instrument')) - ${{ if or(eq(parameters.platform, 'x64'), eq(parameters.platform, 'arm64')) }}: @@ -56,7 +54,7 @@ jobs: inputs: targetType: filePath filePath: build\scripts\Run-Tests.ps1 - arguments: -MatchPattern '*feature.test*.dll' -Platform '$(RationalizedBuildPlatform)' -Configuration '$(BuildConfiguration)' -LogPath '${{ parameters.testLogPath }}' -Root "$(System.ArtifactsDirectory)\\${{ parameters.artifactName }}\\$(BuildConfiguration)\\$(BuildPlatform)\\test" + arguments: -MatchPattern '*feature.test*.dll' -Platform '$(OutputBuildPlatform)' -Configuration '$(BuildConfiguration)' -LogPath '${{ parameters.testLogPath }}' -Root "$(Terminal.BinDir)" condition: and(succeeded(), ne(variables['PGOBuildMode'], 'Instrument')) - task: PowerShell@2 @@ -89,4 +87,4 @@ jobs: flattenFolders: true - publish: '$(Build.ArtifactStagingDirectory)/$(BuildConfiguration)/$(BuildPlatform)/test-logs' - artifact: TestLogs$(BuildPlatform)$(BuildConfiguration) + artifact: test-logs-$(BuildPlatform)-$(BuildConfiguration)${{ parameters.artifactStem }} diff --git a/build/pipelines/templates-v2/steps-create-signing-config.yml b/build/pipelines/templates-v2/steps-create-signing-config.yml new file mode 100644 index 00000000000..790d7a3fa86 --- /dev/null +++ b/build/pipelines/templates-v2/steps-create-signing-config.yml @@ -0,0 +1,35 @@ +parameters: + - name: stage + type: string + - name: outFile + type: string + - name: fragments + type: string + +# This build step template takes all files named "esrp.STAGE.batch.*.json" +# and merges them into a single output signing config. +# +# We generate the batch signing config by sticking together multiple "batches". +# The filter below (with Fragments) works by splitting the filename, esrp.s.batch.x.json, +# to get 'x' and then checking whether x is in Fragments. +# We have to manually strip comments out of the batch fragments due to https://github.com/PowerShell/PowerShell/issues/14553 + +steps: + - pwsh: |- + $SignBatchFiles = (Get-Item build/config/esrp.${{ parameters.stage }}.batch.*.json) + $Fragments = "${{ parameters.fragments }}" + If (-Not [String]::IsNullOrWhiteSpace($Fragments)) { + $FragmentList = $Fragments -Split ";" + If ($FragmentList.Length -Gt 0) { + $SignBatchFiles = $SignBatchFiles | Where-Object { ($_.Name -Split '\.')[3] -In $FragmentList } + } + } + Write-Host "Found $(@($SignBatchFiles).Length) Signing Configs" + Write-Host ($SignBatchFiles.Name -Join ";") + $FinalSignConfig = @{ + Version = "1.0.0"; + UseMinimatch = $false; + SignBatches = @($SignBatchFiles | ForEach-Object { Get-Content $_ | Where-Object { $_ -NotMatch "^\s*\/\/" } | ConvertFrom-Json -Depth 10 }); + } + $FinalSignConfig | ConvertTo-Json -Depth 10 | Out-File -Encoding utf8 "${{ parameters.outFile }}" + displayName: Merge ${{ parameters.stage }} signing configs (${{ parameters.outFile }}) diff --git a/build/pipelines/templates-v2/steps-download-bin-dir-artifact.yml b/build/pipelines/templates-v2/steps-download-bin-dir-artifact.yml new file mode 100644 index 00000000000..e5e90cc8ea2 --- /dev/null +++ b/build/pipelines/templates-v2/steps-download-bin-dir-artifact.yml @@ -0,0 +1,24 @@ +parameters: + - name: buildConfigurations + type: object + - name: buildPlatforms + type: object + - name: artifactStem + type: string + default: '' + +steps: +- ${{ each configuration in parameters.buildConfigurations }}: + - ${{ each platform in parameters.buildPlatforms }}: + - task: DownloadPipelineArtifact@2 + displayName: Download artifacts for ${{ platform }} ${{ configuration }} + inputs: + # Make sure to download the entire artifact, because it includes the SPDX SBOM + artifactName: build-${{ platform }}-${{ configuration }}${{ parameters.artifactStem }} + # Downloading to the source directory should ensure that the later SBOM generator can see the earlier SBOMs. + ${{ if eq(platform, 'x86') }}: + downloadPath: '$(Build.SourcesDirectory)/bin/Win32/${{ configuration }}' + ${{ elseif eq(platform, 'Any CPU') }}: + downloadPath: '$(Build.SourcesDirectory)/bin/AnyCPU/${{ configuration }}' + ${{ else }}: + downloadPath: '$(Build.SourcesDirectory)/bin/${{ platform }}/${{ configuration }}' diff --git a/build/pipelines/templates-v2/steps-ensure-nuget-version.yml b/build/pipelines/templates-v2/steps-ensure-nuget-version.yml new file mode 100644 index 00000000000..fb5cd75e4c8 --- /dev/null +++ b/build/pipelines/templates-v2/steps-ensure-nuget-version.yml @@ -0,0 +1,5 @@ +steps: +- task: NuGetToolInstaller@1 + displayName: Use NuGet 6.6.1 + inputs: + versionSpec: 6.6.1 diff --git a/build/pipelines/templates-v2/steps-fetch-and-prepare-localizations.yml b/build/pipelines/templates-v2/steps-fetch-and-prepare-localizations.yml new file mode 100644 index 00000000000..8ace357aad3 --- /dev/null +++ b/build/pipelines/templates-v2/steps-fetch-and-prepare-localizations.yml @@ -0,0 +1,27 @@ +parameters: + - name: includePseudoLoc + type: boolean + default: true + +steps: + - task: TouchdownBuildTask@1 + displayName: Download Localization Files + inputs: + teamId: 7105 + authId: $(TouchdownAppId) + authKey: $(TouchdownAppKey) + resourceFilePath: | + src\cascadia\**\en-US\*.resw + appendRelativeDir: true + localizationTarget: false + ${{ if eq(parameters.includePseudoLoc, true) }}: + pseudoSetting: Included + + - pwsh: |- + $Files = Get-ChildItem . -R -Filter 'Resources.resw' | ? FullName -Like '*en-US\*\Resources.resw' + $Files | % { Move-Item -Verbose $_.Directory $_.Directory.Parent.Parent -EA:Ignore } + displayName: Move Loc files into final locations + + - pwsh: |- + ./build/scripts/Copy-ContextMenuResourcesToCascadiaPackage.ps1 + displayName: Copy the Context Menu Loc Resources to CascadiaPackage diff --git a/build/pipelines/templates/restore-nuget-steps.yml b/build/pipelines/templates-v2/steps-restore-nuget.yml similarity index 88% rename from build/pipelines/templates/restore-nuget-steps.yml rename to build/pipelines/templates-v2/steps-restore-nuget.yml index aabae84504c..bd0c067531c 100644 --- a/build/pipelines/templates/restore-nuget-steps.yml +++ b/build/pipelines/templates-v2/steps-restore-nuget.yml @@ -1,14 +1,12 @@ steps: -- task: NuGetToolInstaller@0 - displayName: 'Use NuGet 6.3.0' - inputs: - versionSpec: 6.3.0 +- template: steps-ensure-nuget-version.yml - task: NuGetAuthenticate@0 - script: |- echo ##vso[task.setvariable variable=NUGET_RESTORE_MSBUILD_ARGS]/p:Platform=$(BuildPlatform) displayName: Ensure NuGet restores for $(BuildPlatform) + condition: and(succeeded(), ne(variables['BuildPlatform'], 'Any CPU')) # In the Microsoft Azure DevOps tenant, NuGetCommand is ambiguous. # This should be `task: NuGetCommand@2` diff --git a/build/pipelines/templates/build-console-audit-job.yml b/build/pipelines/templates/build-console-audit-job.yml deleted file mode 100644 index 69b0b05ce31..00000000000 --- a/build/pipelines/templates/build-console-audit-job.yml +++ /dev/null @@ -1,34 +0,0 @@ -parameters: - platform: '' - additionalBuildArguments: '' - -jobs: -- job: Build${{ parameters.platform }}AuditMode - displayName: Static Analysis Build ${{ parameters.platform }} - variables: - BuildConfiguration: AuditMode - BuildPlatform: ${{ parameters.platform }} - pool: - ${{ if eq(variables['System.CollectionUri'], 'https://dev.azure.com/ms/') }}: - name: SHINE-OSS-L - ${{ if ne(variables['System.CollectionUri'], 'https://dev.azure.com/ms/') }}: - name: SHINE-INT-L - demands: ImageOverride -equals SHINE-VS17-Latest - - steps: - - checkout: self - submodules: true - clean: true - fetchDepth: 1 - - - template: restore-nuget-steps.yml - - - task: VSBuild@1 - displayName: 'Build solution **\OpenConsole.sln' - inputs: - solution: '**\OpenConsole.sln' - platform: '$(BuildPlatform)' - configuration: '$(BuildConfiguration)' - msbuildArgs: ${{ parameters.additionalBuildArguments }} - clean: true - maximumCpuCount: true diff --git a/build/pipelines/templates/build-console-ci.yml b/build/pipelines/templates/build-console-ci.yml deleted file mode 100644 index 48c27052e29..00000000000 --- a/build/pipelines/templates/build-console-ci.yml +++ /dev/null @@ -1,31 +0,0 @@ -parameters: - configuration: 'Release' - branding: 'Dev' - platform: '' - additionalBuildArguments: '' - -jobs: -- job: Build${{ parameters.platform }}${{ parameters.configuration }}${{ parameters.branding }} - displayName: Build ${{ parameters.platform }} ${{ parameters.configuration }} ${{ parameters.branding }} - variables: - BuildConfiguration: ${{ parameters.configuration }} - BuildPlatform: ${{ parameters.platform }} - WindowsTerminalBranding: ${{ parameters.branding }} - EnableRichCodeNavigation: true - pool: - ${{ if eq(variables['System.CollectionUri'], 'https://dev.azure.com/ms/') }}: - name: SHINE-OSS-L - ${{ if ne(variables['System.CollectionUri'], 'https://dev.azure.com/ms/') }}: - name: SHINE-INT-L - demands: ImageOverride -equals SHINE-VS17-Latest - - steps: - - template: build-console-steps.yml - parameters: - additionalBuildArguments: ${{ parameters.additionalBuildArguments }} - - # It appears that the Component Governance build task that gets automatically injected stopped working - # when we renamed our main branch. - - task: ms.vss-governance-buildtask.governance-build-task-component-detection.ComponentGovernanceComponentDetection@0 - displayName: 'Component Detection' - condition: and(succeededOrFailed(), not(eq(variables['Build.Reason'], 'PullRequest'))) diff --git a/build/pipelines/templates/build-console-compliance-job.yml b/build/pipelines/templates/build-console-compliance-job.yml deleted file mode 100644 index 0f885cfcce7..00000000000 --- a/build/pipelines/templates/build-console-compliance-job.yml +++ /dev/null @@ -1,203 +0,0 @@ -jobs: -- job: Compliance - # We don't *need* a matrix but there's no other way to set parameters on a "job" - # in the AzDO YAML syntax. It would have to be a "stage" or a "template". - # Doesn't matter. We're going to do compliance on Release x64 because - # that's the one all the tooling works against for sure. - strategy: - matrix: - Release_x64: - BuildConfiguration: Release - BuildPlatform: x64 - displayName: Validate Security and Compliance - timeoutInMinutes: 240 - steps: - - checkout: self - clean: true - submodules: true - persistCredentials: True - - task: PkgESSetupBuild@12 - displayName: Package ES - Setup Build - inputs: - disableOutputRedirect: true - - task: PowerShell@2 - displayName: Rationalize Build Platform - inputs: - targetType: inline - script: >- - $Arch = "$(BuildPlatform)" - - If ($Arch -Eq "x86") { $Arch = "Win32" } - - Write-Host "##vso[task.setvariable variable=RationalizedBuildPlatform]${Arch}" - - template: restore-nuget-steps.yml - - task: UniversalPackages@0 - displayName: Download terminal-internal Universal Package - inputs: - feedListDownload: 2b3f8893-a6e8-411f-b197-a9e05576da48 - packageListDownload: e82d490c-af86-4733-9dc4-07b772033204 - versionListDownload: $(TerminalInternalPackageVersion) - - task: TouchdownBuildTask@1 - displayName: Download Localization Files - inputs: - teamId: 7105 - authId: $(TouchdownAppId) - authKey: $(TouchdownAppKey) - resourceFilePath: >- - src\cascadia\TerminalApp\Resources\en-US\Resources.resw - - src\cascadia\TerminalControl\Resources\en-US\Resources.resw - - src\cascadia\TerminalConnection\Resources\en-US\Resources.resw - - src\cascadia\TerminalSettingsModel\Resources\en-US\Resources.resw - - src\cascadia\TerminalSettingsEditor\Resources\en-US\Resources.resw - - src\cascadia\CascadiaPackage\Resources\en-US\Resources.resw - appendRelativeDir: true - localizationTarget: false - pseudoSetting: Included - - task: PowerShell@2 - displayName: Move Loc files one level up - inputs: - targetType: inline - script: >- - $Files = Get-ChildItem . -R -Filter 'Resources.resw' | ? FullName -Like '*en-US\*\Resources.resw' - - $Files | % { Move-Item -Verbose $_.Directory $_.Directory.Parent.Parent -EA:Ignore } - pwsh: true - - # 1ES Component Governance onboarding (Detects open source components). See https://docs.opensource.microsoft.com/tools/cg.html - - task: ms.vss-governance-buildtask.governance-build-task-component-detection.ComponentGovernanceComponentDetection@0 - displayName: Component Detection - - # # PREfast and PoliCheck need Node. Install that first. - - task: NodeTool@0 - - # !!! NOTE !!! Run PREfast first. Some of the other tasks are going to run on a completed build. - # PREfast is going to build the code as a part of its analysis and the generated sources - # and output binaries will be sufficient for the rest of the analysis. - # If you disable this, the other tasks won't likely work. You would have to add a build - # step instead that builds the code normally before calling them. - # Also... PREfast will rebuild anyway so that's why we're not running a normal build first. - # Waste of time to build twice. - # PREfast. See https://www.1eswiki.com/wiki/SDL_Native_Rules_Build_Task - - # The following 1ES tasks all operate completely differently and have a different syntax for usage. - # Most notable is every one of them has a different way of excluding things. - # Go see their 1eswiki.com pages to figure out how to exclude things. - # When writing exclusions, try to make them narrow so when new projects/binaries are added, they - # cause an error here and have to be explicitly pulled out. Don't write an exclusion so broad - # that it will catch other new stuff. - - # https://www.1eswiki.com/wiki/PREfast_Build_Task - # Builds the project with C/C++ static analysis tools to find coding flaws and vulnerabilities - # !!! WARNING !!! It doesn't work with WAPPROJ packaging projects. Build the sub-projects instead. - - task: securedevelopmentteam.vss-secure-development-tools.build-task-prefast.SDLNativeRules@3 - displayName: 'Run the PREfast SDL Native Rules for MSBuild' - condition: succeededOrFailed() - inputs: - setupCommandlines: '"C:\Program Files\Microsoft Visual Studio\2022\Enterprise\Common7\Tools\VsMSBuildCmd.bat"' - msBuildCommandline: msbuild.exe /nologo /m /p:WindowsTerminalOfficialBuild=true /p:WindowsTerminalBranding=${{ parameters.branding }} /p:WindowsTerminalReleaseBuild=true /p:platform=$(BuildPlatform) /p:configuration=$(BuildConfiguration) /t:Terminal\Window\WindowsTerminal /p:VisualStudioVersion=17.0 $(Build.SourcesDirectory)\OpenConsole.sln - msBuildVersion: "17.0" - - # Copies output from PREfast SDL Native Rules task to expected location for consumption by PkgESSecComp - - task: CopyFiles@1 - displayName: 'Copy PREfast xml files to SDLNativeRulesDir' - inputs: - SourceFolder: '$(Agent.BuildDirectory)' - Contents: | - **\*.nativecodeanalysis.xml - TargetFolder: '$(Agent.BuildDirectory)\_sdt\logs\SDLNativeRules' - - # https://www.1eswiki.com/index.php?title=PoliCheck_Build_Task - # Scans the text of source code, comments, and content for terminology that could be sensitive for legal, cultural, or geopolitical reasons. - # (Also finds vulgarities... takes all the fun out of everything.) - - task: securedevelopmentteam.vss-secure-development-tools.build-task-policheck.PoliCheck@2 - displayName: 'Run PoliCheck' - inputs: - targetType: F - targetArgument: $(Build.SourcesDirectory) - result: PoliCheck.xml - optionsFC: 1 - optionsXS: 1 - optionsUEPath: $(Build.SourcesDirectory)\build\config\PolicheckExclusions.xml - optionsHMENABLE: 0 - continueOnError: true - - # https://www.1eswiki.com/wiki/CredScan_Azure_DevOps_Build_Task - # Searches through source code and build outputs for a credential left behind in the open - - task: securedevelopmentteam.vss-secure-development-tools.build-task-credscan.CredScan@3 - displayName: 'Run CredScan' - inputs: - outputFormat: pre - # suppressionsFile: LocalSuppressions.json - batchSize: 20 - debugMode: false - continueOnError: true - - # https://www.1eswiki.com/wiki/BinSkim_Build_Task - # Searches managed and unmanaged binaries for known security vulnerabilities. - - task: securedevelopmentteam.vss-secure-development-tools.build-task-binskim.BinSkim@4 - displayName: 'Run BinSkim' - inputs: - TargetPattern: guardianGlob - # See https://aka.ms/gdn-globs for how to do match patterns - AnalyzeTargetGlob: $(Build.SourcesDirectory)\bin\**\*.dll;$(Build.SourcesDirectory)\bin\**\*.exe;-:file|**\Microsoft.UI.Xaml.dll;-:file|**\Microsoft.Toolkit.Win32.UI.XamlHost.dll;-:file|**\vcruntime*.dll;-:file|**\vcomp*.dll;-:file|**\vccorlib*.dll;-:file|**\vcamp*.dll;-:file|**\msvcp*.dll;-:file|**\concrt*.dll;-:file|**\TerminalThemeHelpers*.dll - continueOnError: true - - # Set XES_SERIALPOSTBUILDREADY to run Security and Compliance task once per build - - powershell: Write-Host "##vso[task.setvariable variable=XES_SERIALPOSTBUILDREADY;]true" - displayName: 'Set XES_SERIALPOSTBUILDREADY Vars' - - # https://www.osgwiki.com/wiki/Package_ES_Security_and_Compliance - # Does a few things: - # - Ensures that Windows-required compliance tasks are run either inside this task - # or were run as a previous step prior to this one - # (PREfast, PoliCheck, Credscan) - # - Runs Windows-specific compliance tasks inside the task - # + CheckCFlags - ensures that compiler and linker flags meet Windows standards - # + CFGCheck/XFGCheck - ensures that Control Flow Guard (CFG) or - # eXtended Flow Guard (XFG) are enabled on binaries - # NOTE: CFG is deprecated and XFG isn't fully ready yet. - # NOTE2: CFG fails on an XFG'd binary - # - Brokers all security/compliance task logs to "Trust Services Automation (TSA)" (https://aka.ms/tsa) - # which is a system that maps all errors into the appropriate bug database - # template for each organization since they all vary. It should also suppress - # new bugs when one already exists for the product. - # This one is set up to go to the OS repository and use the given parameters - # to file bugs to our AzDO product path. - # If we don't use PkgESSecComp to do this for us, we need to install the TSA task - # ourselves in this pipeline to finalize data upload and bug creation. - # !!! NOTE !!! This task goes *LAST* after any other compliance tasks so it catches their logs - - task: PkgESSecComp@10 - displayName: 'Security and Compliance tasks' - inputs: - fileNewBugs: false - areaPath: 'OS\WDX\DXP\WinDev\Terminal' - teamProject: 'OS' - iterationPath: 'OS\Future' - bugTags: 'TerminalReleaseCompliance' - scanAll: true - errOnBugs: false - failOnStdErr: true - taskLogVerbosity: Diagnostic - secCompConfigFromTask: | - # Overrides default build sources directory - sourceTargetOverrideAll: $(Build.SourcesDirectory) - # Overrides default build binaries directory when "Scan all" option is specified - binariesTargetOverrideAll: $(Build.SourcesDirectory)\bin - - # Set the tools to false if they should not run in the build - tools: - - toolName: CheckCFlags - enable: true - - toolName: CFGCheck - enable: true - - toolName: Policheck - enable: false - - toolName: CredScan - enable: false - - toolName: XFGCheck - enable: false diff --git a/build/pipelines/templates/build-console-fuzzing.yml b/build/pipelines/templates/build-console-fuzzing.yml deleted file mode 100644 index 8277ab9dc3a..00000000000 --- a/build/pipelines/templates/build-console-fuzzing.yml +++ /dev/null @@ -1,90 +0,0 @@ -parameters: - configuration: 'Fuzzing' - platform: '' - additionalBuildArguments: '' - -jobs: -- job: Build${{ parameters.platform }}${{ parameters.configuration }} - displayName: Build ${{ parameters.platform }} ${{ parameters.configuration }} - variables: - BuildConfiguration: ${{ parameters.configuration }} - BuildPlatform: ${{ parameters.platform }} - pool: - ${{ if eq(variables['System.CollectionUri'], 'https://dev.azure.com/ms/') }}: - name: SHINE-OSS-L - ${{ if ne(variables['System.CollectionUri'], 'https://dev.azure.com/ms/') }}: - name: SHINE-INT-L - demands: ImageOverride -equals SHINE-VS17-Latest - - steps: - - checkout: self - submodules: true - clean: true - - - template: restore-nuget-steps.yml - - # The environment variable VCToolsInstallDir isn't defined on lab machines, so we need to retrieve it ourselves. - - script: | - "%ProgramFiles(x86)%\Microsoft Visual Studio\Installer\vswhere.exe" -Latest -requires Microsoft.Component.MSBuild -property InstallationPath > %TEMP%\vsinstalldir.txt - set /p _VSINSTALLDIR15=<%TEMP%\vsinstalldir.txt - del %TEMP%\vsinstalldir.txt - call "%_VSINSTALLDIR15%\Common7\Tools\VsDevCmd.bat" - echo VCToolsInstallDir = %VCToolsInstallDir% - echo ##vso[task.setvariable variable=VCToolsInstallDir]%VCToolsInstallDir% - displayName: 'Retrieve VC tools directory' - - - task: VSBuild@1 - displayName: 'Build solution **\OpenConsole.sln' - inputs: - solution: '**\OpenConsole.sln' - platform: '$(BuildPlatform)' - configuration: '$(BuildConfiguration)' - msbuildArgs: "${{ parameters.additionalBuildArguments }}" - clean: true - maximumCpuCount: true - - - task: PowerShell@2 - displayName: 'Rationalize build platform' - inputs: - targetType: inline - script: | - $Arch = "$(BuildPlatform)" - If ($Arch -Eq "x86") { $Arch = "Win32" } - Write-Host "##vso[task.setvariable variable=RationalizedBuildPlatform]${Arch}" - - - task: CopyFiles@2 - displayName: 'Copy result logs to Artifacts' - inputs: - Contents: | - **/*.wtl - **/*onBuildMachineResults.xml - ${{ parameters.testLogPath }} - TargetFolder: '$(Build.ArtifactStagingDirectory)/$(BuildConfiguration)/$(BuildPlatform)/test' - OverWrite: true - flattenFolders: true - - - task: CopyFiles@2 - displayName: 'Copy outputs needed for test runs to Artifacts' - inputs: - Contents: | - $(Build.SourcesDirectory)/bin/$(RationalizedBuildPlatform)/$(BuildConfiguration)/*.exe - $(Build.SourcesDirectory)/bin/$(RationalizedBuildPlatform)/$(BuildConfiguration)/*.dll - $(Build.SourcesDirectory)/bin/$(RationalizedBuildPlatform)/$(BuildConfiguration)/*.xml - **/Microsoft.VCLibs.*.appx - **/TestHostApp/*.exe - **/TestHostApp/*.dll - **/TestHostApp/*.xml - !**/*.pdb - !**/*.ipdb - !**/*.obj - !**/*.pch - TargetFolder: '$(Build.ArtifactStagingDirectory)/$(BuildConfiguration)/$(BuildPlatform)/test' - OverWrite: true - flattenFolders: true - condition: succeeded() - - - task: PublishBuildArtifacts@1 - displayName: 'Publish All Build Artifacts' - inputs: - PathtoPublish: '$(Build.ArtifactStagingDirectory)' - ArtifactName: 'fuzzingBuildOutput' diff --git a/build/pipelines/templates/build-console-pgo.yml b/build/pipelines/templates/build-console-pgo.yml deleted file mode 100644 index b1b4c501633..00000000000 --- a/build/pipelines/templates/build-console-pgo.yml +++ /dev/null @@ -1,55 +0,0 @@ -parameters: - configuration: 'Release' - platform: '' - additionalBuildArguments: '' - minimumExpectedTestsExecutedCount: 1 # Sanity check for minimum expected tests to be reported - rerunPassesRequiredToAvoidFailure: 5 - -jobs: -- job: Build${{ parameters.platform }}${{ parameters.configuration }} - displayName: Build ${{ parameters.platform }} ${{ parameters.configuration }} - variables: - BuildConfiguration: ${{ parameters.configuration }} - BuildPlatform: ${{ parameters.platform }} - PGOBuildMode: 'Instrument' - pool: - ${{ if eq(variables['System.CollectionUri'], 'https://dev.azure.com/ms/') }}: - name: SHINE-OSS-L - ${{ if ne(variables['System.CollectionUri'], 'https://dev.azure.com/ms/') }}: - name: SHINE-INT-L - demands: ImageOverride -equals SHINE-VS17-Latest - - steps: - - template: build-console-steps.yml - parameters: - additionalBuildArguments: '${{ parameters.additionalBuildArguments }}' - -- template: helix-runtests-job.yml - parameters: - name: 'RunTestsInHelix' - dependsOn: Build${{ parameters.platform }}${{ parameters.configuration }} - condition: succeeded() - testSuite: 'PgoInstrumentationSuite' - taefQuery: '@IsPgo=true' - configuration: ${{ parameters.configuration }} - platform: ${{ parameters.platform }} - rerunPassesRequiredToAvoidFailure: ${{ parameters.rerunPassesRequiredToAvoidFailure }} - -- template: helix-processtestresults-job.yml - parameters: - name: 'ProcessTestResults' - pgoArtifact: 'PGO' - dependsOn: - - RunTestsInHelix - condition: succeededOrFailed() - rerunPassesRequiredToAvoidFailure: ${{ parameters.rerunPassesRequiredToAvoidFailure }} - minimumExpectedTestsExecutedCount: ${{ parameters.minimumExpectedTestsExecutedCount }} - -- template: pgo-merge-pgd-job.yml - parameters: - name: 'MergePGD' - dependsOn: - - ProcessTestResults - pgoArtifact: 'PGO' - platform: ${{ parameters.platform }} - configuration: ${{ parameters.configuration }} diff --git a/build/pipelines/templates/build-console-steps.yml b/build/pipelines/templates/build-console-steps.yml deleted file mode 100644 index 947dd353553..00000000000 --- a/build/pipelines/templates/build-console-steps.yml +++ /dev/null @@ -1,137 +0,0 @@ -parameters: - additionalBuildArguments: '' - -steps: -- checkout: self - submodules: true - clean: true - fetchDepth: 1 - -- template: restore-nuget-steps.yml - -# The environment variable VCToolsInstallDir isn't defined on lab machines, so we need to retrieve it ourselves. -- script: | - "%ProgramFiles(x86)%\Microsoft Visual Studio\Installer\vswhere.exe" -Latest -requires Microsoft.Component.MSBuild -property InstallationPath > %TEMP%\vsinstalldir.txt - set /p _VSINSTALLDIR15=<%TEMP%\vsinstalldir.txt - del %TEMP%\vsinstalldir.txt - call "%_VSINSTALLDIR15%\Common7\Tools\VsDevCmd.bat" - echo VCToolsInstallDir = %VCToolsInstallDir% - echo ##vso[task.setvariable variable=VCToolsInstallDir]%VCToolsInstallDir% - displayName: 'Retrieve VC tools directory' - -- task: CmdLine@1 - displayName: 'Display build machine environment variables' - inputs: - filename: 'set' - -- task: VSBuild@1 - displayName: 'Build solution **\OpenConsole.sln' - inputs: - solution: '**\OpenConsole.sln' - platform: '$(BuildPlatform)' - configuration: '$(BuildConfiguration)' - msbuildArgs: "${{ parameters.additionalBuildArguments }} /p:PGOBuildMode=$(PGOBuildMode) /bl:$(Build.SourcesDirectory)\\msbuild.binlog" - clean: true - maximumCpuCount: true - -- task: PowerShell@2 - displayName: 'Check MSIX for common regressions' - # PGO runtime needs its own CRT and it's in the package for convenience. - # That will make this script mad so skip since we're not shipping the PGO Instrumentation one anyway. - condition: ne(variables['PGOBuildMode'], 'Instrument') - inputs: - targetType: inline - script: | - $Package = Get-ChildItem -Recurse -Filter "CascadiaPackage_*.msix" - .\build\scripts\Test-WindowsTerminalPackage.ps1 -Verbose -Path $Package.FullName - -- task: powershell@2 - displayName: 'Source Index PDBs' - condition: ne(variables['PGOBuildMode'], 'Instrument') - inputs: - targetType: filePath - filePath: build\scripts\Index-Pdbs.ps1 - arguments: -SearchDir '$(Build.SourcesDirectory)' -SourceRoot '$(Build.SourcesDirectory)' -recursive -Verbose -CommitId $(Build.SourceVersion) - errorActionPreference: silentlyContinue - -- task: PowerShell@2 - displayName: 'Rationalize build platform' - inputs: - targetType: inline - script: | - $Arch = "$(BuildPlatform)" - If ($Arch -Eq "x86") { $Arch = "Win32" } - Write-Host "##vso[task.setvariable variable=RationalizedBuildPlatform]${Arch}" - -- task: CopyFiles@2 - displayName: 'Copy *.msix to Artifacts' - inputs: - Contents: | - **/*.msix - **/*.appxsym - TargetFolder: '$(Build.ArtifactStagingDirectory)/appx' - OverWrite: true - flattenFolders: true - -- pwsh: |- - $TerminalMsixPath = (Get-Item "$(Build.ArtifactStagingDirectory)\appx\Cascadia*.msix").FullName - $XamlAppxPath = (Get-Item "src\cascadia\CascadiaPackage\AppPackages\*\Dependencies\$(BuildPlatform)\Microsoft.UI.Xaml*.appx").FullName - & .\build\scripts\New-UnpackagedTerminalDistribution.ps1 -TerminalAppX $TerminalMsixPath -XamlAppX $XamlAppxPath -Destination "$(Build.ArtifactStagingDirectory)/unpackaged" - displayName: Build Unpackaged Distribution - -- publish: $(Build.ArtifactStagingDirectory)/unpackaged - artifact: unpackaged-$(BuildPlatform)-$(BuildConfiguration) - displayName: Publish Artifact (unpackaged) - -- task: CopyFiles@2 - displayName: 'Copy outputs needed for test runs to Artifacts' - inputs: - Contents: | - $(Build.SourcesDirectory)/bin/$(RationalizedBuildPlatform)/$(BuildConfiguration)/*.exe - $(Build.SourcesDirectory)/bin/$(RationalizedBuildPlatform)/$(BuildConfiguration)/*.dll - $(Build.SourcesDirectory)/bin/$(RationalizedBuildPlatform)/$(BuildConfiguration)/*.xml - **/Microsoft.VCLibs.*.appx - **/*unit.test*.dll - **/*unit.test*.manifest - **/TestHostApp/*.exe - **/TestHostApp/*.dll - **/TestHostApp/*.xml - !**/*.pdb - !**/*.ipdb - !**/*.obj - !**/*.pch - TargetFolder: '$(Build.ArtifactStagingDirectory)/$(BuildConfiguration)/$(BuildPlatform)/test' - OverWrite: true - flattenFolders: true - condition: succeeded() - -- task: PublishBuildArtifacts@1 - displayName: 'Publish All Build Artifacts' - inputs: - PathtoPublish: '$(Build.ArtifactStagingDirectory)' - ArtifactName: 'drop' - -- task: CopyFiles@2 - displayName: 'Copy PGO databases needed for PGO instrumentation run' - inputs: - Contents: | - **/*.pgd - TargetFolder: '$(Build.ArtifactStagingDirectory)/$(BuildConfiguration)/PGO/$(BuildPlatform)' - OverWrite: true - flattenFolders: true - condition: and(succeeded(), eq(variables['PGOBuildMode'], 'Instrument')) - -- task: PublishBuildArtifacts@1 - displayName: 'Publish All PGO Artifacts' - inputs: - PathtoPublish: '$(Build.ArtifactStagingDirectory)/$(BuildConfiguration)/PGO' - ArtifactName: 'PGO' - condition: and(succeeded(), eq(variables['PGOBuildMode'], 'Instrument')) - -- task: PublishBuildArtifacts@1 - displayName: 'Publish Artifact: binlog' - condition: always() - continueOnError: True - inputs: - PathtoPublish: $(Build.SourcesDirectory)\msbuild.binlog - ArtifactName: binlog-$(BuildPlatform) diff --git a/build/pipelines/templates/check-formatting.yml b/build/pipelines/templates/check-formatting.yml deleted file mode 100644 index 46abca03240..00000000000 --- a/build/pipelines/templates/check-formatting.yml +++ /dev/null @@ -1,17 +0,0 @@ - -jobs: -- job: CodeFormatCheck - displayName: Proper Code Formatting Check - pool: { vmImage: windows-2022 } - - steps: - - checkout: self - fetchDepth: 1 - submodules: false - clean: true - - - task: PowerShell@2 - displayName: 'Code Formatting Check' - inputs: - targetType: filePath - filePath: '.\build\scripts\Invoke-FormattingCheck.ps1' diff --git a/build/pipelines/templates/console-ci-helix-job.yml b/build/pipelines/templates/console-ci-helix-job.yml deleted file mode 100644 index 07476712a02..00000000000 --- a/build/pipelines/templates/console-ci-helix-job.yml +++ /dev/null @@ -1,25 +0,0 @@ -parameters: - configuration: 'Release' - platform: '' - minimumExpectedTestsExecutedCount: 10 # Sanity check for minimum expected tests to be reported - rerunPassesRequiredToAvoidFailure: 5 - -jobs: -- template: helix-runtests-job.yml - parameters: - name: 'RunTestsInHelix' - # We're not setting dependsOn as we want to rely on the "stage" dependency above us - testSuite: 'DevTestSuite' - platform: ${{ parameters.platform }} - configuration: ${{ parameters.configuration }} - rerunPassesRequiredToAvoidFailure: ${{ parameters.rerunPassesRequiredToAvoidFailure }} - -- template: helix-processtestresults-job.yml - parameters: - dependsOn: - - RunTestsInHelix - # the default condition is succeededOrFailed(), and the "stage" condition ensures we only run as needed - platform: ${{ parameters.platform }} - configuration: ${{ parameters.configuration }} - rerunPassesRequiredToAvoidFailure: ${{ parameters.rerunPassesRequiredToAvoidFailure }} - minimumExpectedTestsExecutedCount: ${{ parameters.minimumExpectedTestsExecutedCount }} diff --git a/build/pipelines/templates/helix-createprojfile-steps.yml b/build/pipelines/templates/helix-createprojfile-steps.yml deleted file mode 100644 index 3f7dc8c0977..00000000000 --- a/build/pipelines/templates/helix-createprojfile-steps.yml +++ /dev/null @@ -1,15 +0,0 @@ -parameters: - condition: '' - testFilePath: '' - outputProjFileName: '' - testSuite: '' - taefQuery: '' - -steps: - - task: powershell@2 - displayName: 'Create ${{ parameters.outputProjFileName }}' - condition: ${{ parameters.condition }} - inputs: - targetType: filePath - filePath: build\Helix\GenerateTestProjFile.ps1 - arguments: -TestFile '${{ parameters.testFilePath }}' -OutputProjFile '$(Build.ArtifactStagingDirectory)\$(BuildConfiguration)\$(BuildPlatform)\${{ parameters.outputProjFileName }}' -JobTestSuiteName '${{ parameters.testSuite }}' -TaefPath '$(Build.SourcesDirectory)\build\Helix\packages\Microsoft.Taef.10.60.210621002\build\Binaries\x86' -TaefQuery '${{ parameters.taefQuery }}' \ No newline at end of file diff --git a/build/pipelines/templates/helix-processtestresults-job.yml b/build/pipelines/templates/helix-processtestresults-job.yml deleted file mode 100644 index bd8e967de92..00000000000 --- a/build/pipelines/templates/helix-processtestresults-job.yml +++ /dev/null @@ -1,71 +0,0 @@ -parameters: - condition: 'succeededOrFailed()' - dependsOn: '' - rerunPassesRequiredToAvoidFailure: 5 - minimumExpectedTestsExecutedCount: 10 - checkJobAttempt: false - pgoArtifact: '' - -jobs: -- job: ProcessTestResults - displayName: Process Helix Results ${{ parameters.platform }} ${{ parameters.configuration }} - condition: ${{ parameters.condition }} - dependsOn: ${{ parameters.dependsOn }} - pool: - vmImage: 'windows-2019' - timeoutInMinutes: 120 - variables: - helixOutputFolder: $(Build.SourcesDirectory)\HelixOutput - - steps: - - task: powershell@2 - displayName: 'UpdateUnreliableTests.ps1' - condition: succeededOrFailed() - env: - SYSTEM_ACCESSTOKEN: $(System.AccessToken) - HelixAccessToken: $(HelixApiAccessToken) - inputs: - targetType: filePath - filePath: build\Helix\UpdateUnreliableTests.ps1 - arguments: -RerunPassesRequiredToAvoidFailure '${{ parameters.rerunPassesRequiredToAvoidFailure }}' - - - task: powershell@2 - displayName: 'OutputTestResults.ps1' - condition: succeededOrFailed() - env: - SYSTEM_ACCESSTOKEN: $(System.AccessToken) - HelixAccessToken: $(HelixApiAccessToken) - inputs: - targetType: filePath - filePath: build\Helix\OutputTestResults.ps1 - arguments: -MinimumExpectedTestsExecutedCount '${{ parameters.minimumExpectedTestsExecutedCount }}' -CheckJobAttempt $${{ parameters.checkJobAttempt }} - - - task: powershell@2 - displayName: 'ProcessHelixFiles.ps1' - condition: succeededOrFailed() - env: - SYSTEM_ACCESSTOKEN: $(System.AccessToken) - HelixAccessToken: $(HelixApiAccessToken) - inputs: - targetType: filePath - filePath: build\Helix\ProcessHelixFiles.ps1 - arguments: -OutputFolder '$(helixOutputFolder)' - - - ${{if ne(parameters.pgoArtifact, '') }}: - - script: move /y $(helixOutputFolder)\PGO $(Build.ArtifactStagingDirectory) - displayName: 'Move pgc files to PGO artifact' - - - task: PublishBuildArtifacts@1 - displayName: 'Publish Helix files' - condition: succeededOrFailed() - inputs: - PathtoPublish: $(helixOutputFolder) - artifactName: drop - - - ${{if ne(parameters.pgoArtifact, '') }}: - - task: PublishBuildArtifacts@1 - displayName: 'Publish pgc files' - condition: succeededOrFailed() - inputs: - PathtoPublish: $(Build.ArtifactStagingDirectory)\PGO\Release - artifactName: ${{ parameters.pgoArtifact }} diff --git a/build/pipelines/templates/helix-runtests-job.yml b/build/pipelines/templates/helix-runtests-job.yml deleted file mode 100644 index 0e7c22efff0..00000000000 --- a/build/pipelines/templates/helix-runtests-job.yml +++ /dev/null @@ -1,164 +0,0 @@ -parameters: - name: 'RunTestsInHelix' - dependsOn: '' - condition: '' - testSuite: '' - # If a Pipeline runs this template more than once, this parameter should be unique per build flavor to differentiate the - # the different test runs: - helixType: 'test/devtest' - artifactName: 'drop' - maxParallel: 4 - rerunPassesRequiredToAvoidFailure: 5 - taefQuery: '' - configuration: '' - platform: '' - # if 'useBuildOutputFromBuildId' is set, we will default to using a build from this pipeline: - useBuildOutputFromPipeline: $(System.DefinitionId) - openHelixTargetQueues: 'windows.11.amd64.client.open.reunion' - closedHelixTargetQueues: 'windows.11.amd64.client.reunion' - -jobs: -- job: ${{ parameters.name }} - displayName: Submit Helix ${{ parameters.platform }} ${{ parameters.configuration }} - dependsOn: ${{ parameters.dependsOn }} - condition: ${{ parameters.condition }} - pool: - vmImage: 'windows-2019' - timeoutInMinutes: 120 - strategy: - maxParallel: ${{ parameters.maxParallel }} - variables: - buildConfiguration: ${{ parameters.configuration }} - buildPlatform: ${{ parameters.platform }} - openHelixTargetQueues: ${{ parameters.openHelixTargetQueues }} - closedHelixTargetQueues: ${{ parameters.closedHelixTargetQueues }} - artifactsDir: $(Build.SourcesDirectory)\Artifacts - taefPath: $(Build.SourcesDirectory)\build\Helix\packages\Microsoft.Taef.10.60.210621002\build\Binaries\$(buildPlatform) - helixCommonArgs: '/binaryLogger:$(Build.SourcesDirectory)/${{parameters.name}}.$(buildPlatform).$(buildConfiguration).binlog /p:HelixBuild=$(Build.BuildId).$(buildPlatform).$(buildConfiguration) /p:Platform=$(buildPlatform) /p:Configuration=$(buildConfiguration) /p:HelixType=${{parameters.helixType}} /p:TestSuite=${{parameters.testSuite}} /p:ProjFilesPath=$(Build.ArtifactStagingDirectory) /p:rerunPassesRequiredToAvoidFailure=${{parameters.rerunPassesRequiredToAvoidFailure}}' - - steps: - - task: CmdLine@1 - displayName: 'Display build machine environment variables' - inputs: - filename: 'set' - - - task: NuGetToolInstaller@0 - displayName: 'Use NuGet 6.3.0' - inputs: - versionSpec: 6.3.0 - - - task: 333b11bd-d341-40d9-afcf-b32d5ce6f23b@2 - displayName: 'NuGet restore build/Helix/packages.config' - inputs: - restoreSolution: build/Helix/packages.config - feedsToUse: config - nugetConfigPath: nuget.config - restoreDirectory: packages - - - task: DownloadBuildArtifacts@0 - condition: - and(succeeded(),eq(variables['useBuildOutputFromBuildId'],'')) - inputs: - artifactName: ${{ parameters.artifactName }} - downloadPath: '$(artifactsDir)' - - - task: DownloadBuildArtifacts@0 - condition: - and(succeeded(),ne(variables['useBuildOutputFromBuildId'],'')) - inputs: - buildType: specific - buildVersionToDownload: specific - project: $(System.TeamProjectId) - pipeline: ${{ parameters.useBuildOutputFromPipeline }} - buildId: $(useBuildOutputFromBuildId) - artifactName: ${{ parameters.artifactName }} - downloadPath: '$(artifactsDir)' - - - task: CmdLine@1 - displayName: 'Display Artifact Directory payload contents' - inputs: - filename: 'dir' - arguments: '/s $(artifactsDir)' - - - task: powershell@2 - displayName: 'PrepareHelixPayload.ps1' - inputs: - targetType: filePath - filePath: build\Helix\PrepareHelixPayload.ps1 - arguments: -Platform '$(buildPlatform)' -Configuration '$(buildConfiguration)' -ArtifactName '${{ parameters.artifactName }}' - - - task: CmdLine@1 - displayName: 'Display Helix payload contents' - inputs: - filename: 'dir' - arguments: '/s $(Build.SourcesDirectory)\HelixPayload' - - - task: PowerShell@2 - displayName: 'Make artifact directories' - inputs: - targetType: inline - script: | - New-Item -ItemType Directory -Force -Path "$(Build.ArtifactStagingDirectory)\$(BuildConfiguration)\" - New-Item -ItemType Directory -Force -Path "$(Build.ArtifactStagingDirectory)\$(BuildConfiguration)\$(BuildPlatform)\" - - - template: helix-createprojfile-steps.yml - parameters: - condition: and(succeeded(),eq('${{ parameters.testSuite }}','DevTestSuite')) - testFilePath: '$(artifactsDir)\${{ parameters.artifactName }}\$(buildConfiguration)\$(buildPlatform)\Test\TerminalApp.LocalTests.dll' - outputProjFileName: 'RunTestsInHelix-TerminalAppLocalTests.proj' - testSuite: '${{ parameters.testSuite }}' - taefQuery: ${{ parameters.taefQuery }} - - - template: helix-createprojfile-steps.yml - parameters: - condition: and(succeeded(),eq('${{ parameters.testSuite }}','DevTestSuite')) - testFilePath: '$(artifactsDir)\${{ parameters.artifactName }}\$(buildConfiguration)\$(buildPlatform)\Test\SettingsModel.LocalTests.dll' - outputProjFileName: 'RunTestsInHelix-SettingsModelLocalTests.proj' - testSuite: '${{ parameters.testSuite }}' - taefQuery: ${{ parameters.taefQuery }} - - - - template: helix-createprojfile-steps.yml - parameters: - condition: and(succeeded(),eq('${{ parameters.testSuite }}','DevTestSuite')) - testFilePath: '$(artifactsDir)\${{ parameters.artifactName }}\$(buildConfiguration)\$(buildPlatform)\Test\Conhost.UIA.Tests.dll' - outputProjFileName: 'RunTestsInHelix-HostTestsUIA.proj' - testSuite: '${{ parameters.testSuite }}' - taefQuery: ${{ parameters.taefQuery }} - - - template: helix-createprojfile-steps.yml - parameters: - condition: and(succeeded(),or(eq('${{ parameters.testSuite }}','PgoInstrumentationSuite'),eq('${{ parameters.testSuite }}','DevTestSuite'))) - testFilePath: '$(artifactsDir)\${{ parameters.artifactName }}\$(buildConfiguration)\$(buildPlatform)\Test\WindowsTerminal.UIA.Tests.dll' - outputProjFileName: 'RunTestsInHelix-WindowsTerminalUIATests.proj' - testSuite: '${{ parameters.testSuite }}' - taefQuery: ${{ parameters.taefQuery }} - - - task: PublishBuildArtifacts@1 - displayName: 'Publish generated .proj files' - inputs: - PathtoPublish: $(Build.ArtifactStagingDirectory) - artifactName: ${{ parameters.artifactName }} - - - task: DotNetCoreCLI@2 - displayName: 'Run tests in Helix (open queues)' - condition: and(succeeded(),eq(variables['System.CollectionUri'],'https://dev.azure.com/ms/')) - env: - SYSTEM_ACCESSTOKEN: $(System.AccessToken) - inputs: - command: custom - projects: build\Helix\RunTestsInHelix.proj - custom: msbuild - arguments: '$(helixCommonArgs) /p:IsExternal=true /p:Creator=Terminal /p:HelixTargetQueues=$(openHelixTargetQueues)' - - - task: DotNetCoreCLI@2 - displayName: 'Run tests in Helix (closed queues)' - condition: and(succeeded(),ne(variables['System.CollectionUri'],'https://dev.azure.com/ms/')) - env: - SYSTEM_ACCESSTOKEN: $(System.AccessToken) - HelixAccessToken: $(HelixApiAccessToken) - inputs: - command: custom - projects: build\Helix\RunTestsInHelix.proj - custom: msbuild - arguments: '$(helixCommonArgs) /p:HelixTargetQueues=$(closedHelixTargetQueues)' diff --git a/build/pipelines/templates/pgo-merge-pgd-job.yml b/build/pipelines/templates/pgo-merge-pgd-job.yml deleted file mode 100644 index 89cf5319ccd..00000000000 --- a/build/pipelines/templates/pgo-merge-pgd-job.yml +++ /dev/null @@ -1,70 +0,0 @@ -parameters: - dependsOn: '' - pgoArtifact: PGO - platform: '' - configuration: '' - -jobs: -- job: MergePGD - dependsOn: ${{ parameters.dependsOn }} - pool: - vmImage: 'windows-2019' - variables: - artifactsPath: $(Build.SourcesDirectory)\Artifacts - pgoArtifactsPath: $(artifactsPath)\${{ parameters.pgoArtifact }} - buildPlatform: ${{ parameters.platform }} - buildConfiguration: ${{ parameters.configuration }} - - steps: - # The environment variable VCToolsInstallDir isn't defined on lab machines, so we need to retrieve it ourselves. - - script: | - "%ProgramFiles(x86)%\Microsoft Visual Studio\Installer\vswhere.exe" -Latest -requires Microsoft.Component.MSBuild -property InstallationPath > %TEMP%\vsinstalldir.txt - set /p _VSINSTALLDIR15=<%TEMP%\vsinstalldir.txt - del %TEMP%\vsinstalldir.txt - call "%_VSINSTALLDIR15%\Common7\Tools\VsDevCmd.bat" - echo VCToolsInstallDir = %VCToolsInstallDir% - echo ##vso[task.setvariable variable=VCToolsInstallDir]%VCToolsInstallDir% - displayName: 'Retrieve VC tools directory' - - - task: NuGetToolInstaller@0 - displayName: 'Use NuGet 6.3.0' - inputs: - versionSpec: 6.3.0 - - - task: NuGetAuthenticate@0 - - # In the Microsoft Azure DevOps tenant, NuGetCommand is ambiguous. - # This should be `task: NuGetCommand@2` - - task: 333b11bd-d341-40d9-afcf-b32d5ce6f23b@2 - displayName: Restore NuGet packages for extraneous build actions - inputs: - command: restore - feedsToUse: config - configPath: NuGet.config - restoreSolution: build/packages.config - restoreDirectory: '$(Build.SourcesDirectory)\packages' - - - task: DownloadBuildArtifacts@0 - inputs: - artifactName: ${{ parameters.pgoArtifact }} - downloadPath: $(artifactsPath) - - - task: MSBuild@1 - displayName: Merge counts into PGD - inputs: - solution: $(Build.SourcesDirectory)\OpenConsole.sln - platform: $(buildPlatform) - configuration: $(buildConfiguration) - msbuildArguments: '/t:MergePGOCounts /p:PGOBuildMode=Instrument /p:PGDPath=$(pgoArtifactsPath)\$(buildPlatform) /p:PGCRootPath=$(pgoArtifactsPath)\$(buildPlatform)' - - - task: CopyFiles@2 - displayName: 'Copy merged pgd to artifact staging' - inputs: - sourceFolder: $(pgoArtifactsPath) - contents: '**\$(buildPlatform)\*.pgd' - targetFolder: $(Build.ArtifactStagingDirectory) - - - task: PublishBuildArtifacts@1 - inputs: - pathToPublish: $(Build.ArtifactStagingDirectory) - artifactName: ${{ parameters.pgoArtifact }} diff --git a/build/scripts/Run-Tests.ps1 b/build/scripts/Run-Tests.ps1 index 1e52a8757e2..bd4cf28fc50 100644 --- a/build/scripts/Run-Tests.ps1 +++ b/build/scripts/Run-Tests.ps1 @@ -9,7 +9,8 @@ Param( [Parameter(Mandatory=$false, Position=3)] [string]$LogPath, [Parameter(Mandatory=$false)] - [string]$Root = ".\bin\$Platform\$Configuration" + [string]$Root = ".\bin\$Platform\$Configuration", + [string[]]$AdditionalTaefArguments ) # Find test DLLs based on the provided root, match pattern, and recursion @@ -26,7 +27,7 @@ if ($LogPath) { } # Invoke the te.exe executable with arguments and test DLLs -& "$Root\te.exe" $args $testDlls.FullName +& "$Root\te.exe" $args $testDlls.FullName $AdditionalTaefArguments # Check the exit code of the te.exe process and exit accordingly if ($LASTEXITCODE -ne 0) { diff --git a/src/cascadia/WpfTerminalControl/WpfTerminalControl.csproj b/src/cascadia/WpfTerminalControl/WpfTerminalControl.csproj index 661f676b59d..dc3eebe8749 100644 --- a/src/cascadia/WpfTerminalControl/WpfTerminalControl.csproj +++ b/src/cascadia/WpfTerminalControl/WpfTerminalControl.csproj @@ -46,6 +46,19 @@ true runtimes\win-arm64\native\ + + + true + runtimes\win-x86\native\ + + + true + runtimes\win-x64\native\ + + + true + runtimes\win-arm64\native\ + diff --git a/src/common.nugetversions.props b/src/common.nugetversions.props index 19d169a45a0..15e2fe774da 100644 --- a/src/common.nugetversions.props +++ b/src/common.nugetversions.props @@ -1,7 +1,7 @@ - + diff --git a/src/common.nugetversions.targets b/src/common.nugetversions.targets index 3113c124072..2a3d5cc062e 100644 --- a/src/common.nugetversions.targets +++ b/src/common.nugetversions.targets @@ -37,7 +37,8 @@ - + + @@ -71,8 +72,8 @@ - - + +