AzureDevOpsDsc: Running integration test in pipeline

[johlju]

The integration tests are failing due to a lack of an Integration environment (Azure DevOps Services instance) and related API key.

In order to resolve the integration tests, the following variables need adding to the build/pipeline:

• AzureDevOps.Integration.ApiUri
• AzureDevOps.Integration.Pat (set as sensitive variable)

... and...

• the ApiUri value needs setting to a URI of an Azure DevOps organization (e.g. https://dev.azure.com/someOrganizationName/_apis/) where the organization will be torn down and recreated each time (i.e. don't use this organization for anything you want to keep! 😁) ... there also has to be some consideration to ensure that multiple sets of integration tests can't run similtaniously against the same instance (not sure how that would be handled, initially - not sure if you want to create a new organization for every build? ... I think there is a limit).
• the ApiKey value needs setting to a key with permissions to effectively do anything within the organization used so the DSC resources can be applied into it.

Originally posted by @SphenicPaul in #7 (comment)

[johlju]

The problem is that the integration test cannot run for PR's since the secret pipeline values are not added to the build for PR's. That will make the build fail on main branch only, and then its "to late" to fix the build issue for the PR. So I'm thinking we should look at making the integration tests to run manually and each contributor can set there own "destructible" DevOps tenant.

[johlju]

The part of running the integration tests is commented out, so if there is not a good way to run the integration tests for PR's this entire job should be removed (if moving to running them manually).

AzureDevOpsDsc/azure-pipelines.yml

Lines 194 to 221 in 8d43d12

	# TODO: Current disabled due to integration tests throws the exception:
	# "Cannot validate argument on parameter 'Pat'. The " Test-AzDevOpsPat -Pat $_ -IsValid " validation script for the argument with value "$(AzureDevOps.Integration.Pat)" did not return a result of True. Determine why the validation script failed, and then try the command again."
	# - powershell: |
	# ./build.ps1 -Tasks test -CodeCoverageThreshold 0 -PesterScript @(
	# 'tests/Integration/'
	# )
	# name: test
	# displayName: 'Run Integration Tests'
	# env:
	# azureDevOpsIntegrationApiUri: $(AzureDevOps.Integration.ApiUri)
	# azureDevOpsIntegrationPat: $(AzureDevOps.Integration.Pat)
	# - task: PublishTestResults@2
	# displayName: 'Publish Test Results'
	# condition: succeededOrFailed()
	# inputs:
	# testResultsFormat: 'NUnit'
	# testResultsFiles: 'output/testResults/NUnit*.xml'
	# testRunTitle: 'Windows Server Core (PowerShell Core) - Integration'
	# - task: PublishCodeCoverageResults@1
	# displayName: 'Publish Code Coverage'
	# condition: succeededOrFailed()
	# inputs:
	# codeCoverageTool: 'JaCoCo'
	# summaryFileLocation: 'output/testResults/CodeCov*.xml'
	# pathToSources: '$(Build.SourcesDirectory)/output/$(dscBuildVariable.RepositoryName)'

[SphenicPaul]

Some initial thoughts:

• I think we need integration tests as part of this module's test in some form... ideally automatic.
• Not sure if there are options to either/or:
  • Pull the API key and URI of a PR-author-specific environment (or a as part of the PR in some other way?)
  • Trigger a PR-author-specific-integration build (in their own, public organization) using the same commit (i.e. code and pipeline definition)? - Would need a little more setup to contribute initially and would need something to tie the PR build into the contributor to determine which to execute. Could get a little difficult to manage/control/support?
  • If there is a resource (or simple script for now) for installiing the OnPrem/Server version of Azure DevOps, these integration tests could be run directly on the build server, but requires additional SQL Install (possibly using SqlServerDsc module), significant amount of work with the initial resource to get to that point and there might be memory lmiitations on the build agents.

[SphenicPaul]

Additionally, this is relevant to documentation here which will need amending as appropriate.

[johlju]

Agree we need integration test.

Pull the API key and URI of a PR-author-specific environment (or a as part of the PR in some other way?)

I don't see any secure way of getting the API key for a PR that couldn't leak to other contributors/maintainers. 🤔

Trigger a PR-author-specific-integration build (in their own, public organization)

There might be the only way. Do not run integration tests in the PR, but instead the contributor must configure Azure Pipelines against there fork. We add documentation how a contributor creates a "destructible" Azure DevOps tenant on top of the one the fork is connected to so integration tests can run for the working branches in the fork. Then we add an entry to the PR template that clearly says that a PR must include a link to a test run that passes.

We run the integration tests on merge to main. We verify the PR in the contributors pipeline. It might be a slim risk that it will fail in main. It is a lot of work to contribute though, and more work to review.

If there is a resource (or simple script for now) for installing the OnPrem/Server version of Azure DevOps, these integration tests could be run directly on the build server

This would be the best way to go. Looking at the software requirements it could be feasible looking at the Microsoft hosted-agents hardware it is using a Standard_DS2_v2 that has 7GB of memory. So if we can limit the SQL Server to 2GB (or 1GB) then run Azure DevOps server can run on (the not recommended) 2GB memory, then it could work.

[johlju]

I have create a new Azure DevOps organization (https://dev.azure.com/azuredevopsdsc/) that is destructible, and added a PAT for that which is available when running builds on main. The organization is entirely separate with its own accounts so we could potentially add maintainers to it if needed.

This does not solve this issue, but at least the tests that exist and new that is created can run. There will be a problem if merging a PR and the build fails on integration tests.

[SphenicPaul]

Useful SQL Server installation information from SqlServerDsc project:

• SQL2019 ISO/EXE Information
• SQL2019 Installation Usage - See/Search DownloadExeName, DownloadExePath, DownloadIsoName, DownloadIsoPath

Useful links for Azure DevOps Server 2020 RTW (from here):

• ISO
• Web Install
• Requirements/Compatibilty Documentation
• Installation Documentation

[SphenicPaul]

Just as an update... I've been taking a look at this and am now at a point where Azure DevOps installer/EXE is downloaded, then Azure DevOps Server is installed and configured automatically/programatically.

So far, it takes about 25 minutes on a hosted, build server at present (the majority of that time is the installation itself but there might be ways of installing non-used components to speed this up at some point).

Currently, I am trying to (and need to) determine:

• How to connect into Azure DevOps Server once installed - It's not clear from the documentation which users initially have access initially (still looking though).

• How to generate a Personal Access Token (PAT) - There might be a function for doing this via the API, but it's unclear if this API call will work with a user's password and not a PAT on the intial call, or can the user has to log in using a different mechanism? - Can't really use the API to generate a PAT if we need a PAT to use it - Not sure if use of Selenium WebDriver or similar could be used to work through the steps via the GUI in some way.

  I'm not sure on how to proceed with this one at present. Any thoughts welcome.

Also note that Azure DevOps will seemingly install SQL Server Express as part of the configuration step so I've been able to perform a successful (according to the logs, but I've not been able to connect into it yet) installation and configuration of 'BasicInstall' Azure DevOps Server without having to perform a specific, SQL Server installation.

I'm unclear what features using this default, SQL Server Express will include/omit from the Azure DevOps Server instance at present but I've been trying to focus on getting a basic, Azure DevOps Services instance running before considering anything else.

[SphenicPaul]

I might park the above for a bit and have a think on it, and possibly look at alternative options... specifically around running (or confirming the run of) a contributor-specific pipeline/build (in their own Azure DevOps Server Organization) as part of the PR build based on the same, PR commit from the same repository.

Some notes:

• We could typically have multiple contributors, and multiple PRs per contributor (all may want/have different 'Organizations' to run against.
• Not initially sure where the contributor-specific (or PR-specific) information would have to be defined in order to use in builds both local (this, AzureDevOpsDsc organization) and remote (contributor organization) - Possibly in a config file and make use of .gitattributes file to omit this file from final merge into 'main':
  • https://stackoverflow.com/questions/15232000/git-ignore-files-during-merge
  • https://git-scm.com/book/en/v2/Customizing-Git-Git-Attributes#_merge_strategies
• Could also be some conditional logic in the azure-pipelines.yml if needed (e.g. if PR build or non-PR build, or if in DSCCommuinity AzureDevOpsDsc build/org etc.) to determine steps/jobs to be run
• Ideally want to remove jobs/tasks running twice (i.e. both local and remote) - Can PR use solely a remote build?
• Might be able to make use of nested/referenced/composite build templates to seperate any contributor/PR-specific information from local/DscCommunity one.
• Not intiially sure how any pipelines and/or organisations referenced would have to have some way of being accessed/called/referenced/checked. Guessing they would have to be public?
• Make use of something similar to this to retrieve remote, build statuses? or the API?
• Would need to work through the PR and pipeline (local and remote) workflow/order relating to both PR build and final merge into main
• Can we (and do we we want to) make use of build triggers (i.e. when a pipeline has completed)?
• Want to avoid PAT being in text/config anywhere (each contributor would maintain PAT for own Organization)

I'm slightly steering towards this being my preferred solution for this (if I can get to something that is workable) as this avoids the 25+ minute, pre-integration test, setup time (see my previous comment/post) and allows the integration tests to be completely within a few minutes (at present) - Nice, fast feedback from the integration tests 😄.

We'd also be testing against an up-to-date version/instance - It would remove a little maintenance relating to upgrading integration test target instances etc. and give us relatively quick visability of changes to the API breaking functionality in the module.

The setup of the builds for new contributors is still likely to be more time-consuming using this approach though.

[SphenicPaul]

And also, just putting it out there (even though it's bad practice, and potentially, higher risk)...

What are the downsides/risks of making the variable (in the build pipeline) that holds the PAT into a non- sensitive variable (so the PR builds can use/see it), and ensuring the scope is limited to the resources that can be managed by it.

I'd guess the PAT would no-longer be protected (and effectively public to those that would want to create a PR/change/whatever to uncover it as it wouldn't be suppressed in the logs - although could potentially create a second variable (that was sensitive) with the same value/PAT to obsfucate this?).

The PAT would only be providing access to teardown 'organization' and any resources in it (assuming it is scoped and created correctly) so the likely problems resulting from this would be where people/public want to deliberately mess up resources within the instance to hinder/impact the build, integration environment (which would/could be "reset" during a build anyway).

This would also mean that the PRs couldn't run in parallel as they would all be running against the same instance (and there would have to be some lock mechanism in the build to prevent this).

This seems like it may be less work than the other options? Making a PAT, deliberately visable (even though it's a little work to get hold of and if it grants access to little of any use/value) does seem non-prefered.

Thoughts?

[SphenicPaul]

Also, I've just found the following Microsoft answer which might suggest that obtaining an API key via the API is not going to be an option (with reference to the option of installing a build server, copy of Azure DevOps Server)...

https://docs.microsoft.com/en-us/azure/devops/organizations/accounts/use-personal-access-tokens-to-authenticate?view=azure-devops&tabs=preview-page#q-is-there-a-way-to-renew-a-pat-via-rest-api

... which might suggest some form of automation via the GUI would be required to obtain an API key.