backend/azurerm: upgrading the SDK / support for proxies #19414

tombuildsstuff · 2018-11-20T15:51:27Z

This PR upgrades to v21.3.0 of github.com/Azure/azure-sdk-for-go, v10.15.4 of git.luolix.topAzure/go-autorest and vendors github.com/hashicorp/go-azure-helpers @ 0.1.1

This is the first in a couple of PR's to the Backend which allows us to support additional authentication modes for the backend, which will allow us to fix #18425 and #16763.

This PR intentionally doesn't fix the locking issue tracked in #17046 (see more details below) - however it should be possible to fix that once this is merged (since this changeset's already quite big)

Tasks:

Upgrade Azure/go-autorest / Azure/azure-sdk-for-go
Threading through the new auth package
Proxy Support
Switching to use the Resources Profile
Switching to use the Storage Profile
Setting the User Agent
Additional Acceptance Tests coverage for Service Principal Client Secret / Access Key
Manual Testing

tombuildsstuff · 2018-11-20T20:47:54Z

This test currently fails in master:

$ TF_ACC=1 envchain azurerm go test -v ./backend/remote-state/azure/ -run="TestBackendLocked"
=== RUN   TestBackendLocked
2018/11/20 21:37:58 [DEBUG] Could not lock as state blob did not exist, creating with empty state
--- FAIL: TestBackendLocked (136.20s)
    backend_test.go:120: creating resource group terraform-backend-testing-1587647403404359918
    backend_test.go:126: creating storage account tfbackendtesting4ahk
    backend_test.go:140: fetching access key for storage account
    backend_test.go:157: creating container terraform
    backend_test.go:77: TestBackendConfig on *azure.Backend with configs.synthBody{Filename:"<TestWrapConfig>", Values:map[string]cty.Value{"access_key":cty.StringVal("/nAdA86fOV+5XQZPSyBW/GuXG4OSTamTbKneOw8ucg3pg3w8JYgRxPGcOxrJ/2IUgTTdekiJviJ65jAc25i3LQ=="), "container_name":cty.StringVal("terraform"), "key":cty.StringVal("testState"), "storage_account_name":cty.StringVal("tfbackendtesting4ahk")}}
    backend_test.go:84: TestBackendConfig on *azure.Backend with configs.synthBody{Filename:"<TestWrapConfig>", Values:map[string]cty.Value{"access_key":cty.StringVal("/nAdA86fOV+5XQZPSyBW/GuXG4OSTamTbKneOw8ucg3pg3w8JYgRxPGcOxrJ/2IUgTTdekiJviJ65jAc25i3LQ=="), "container_name":cty.StringVal("terraform"), "key":cty.StringVal("testState"), "storage_account_name":cty.StringVal("tfbackendtesting4ahk")}}
    backend_test.go:91: TestBackend: testing state locking for *azure.Backend
    backend_test.go:92: TestBackend: testing state locking for *azure.Backend
    backend_test.go:92: could not unlock with the reported ID "d4dcc229-3c31-8f1c-3bce-341391bd10f4": failed to delete lock info from metadata: storage: service returned error: StatusCode=412, ErrorCode=LeaseIdMissing, ErrorMessage=There is currently a lease on the blob and no lease ID was specified in the request.
        RequestId:7763c56f-901e-0022-3f10-811362000000
        Time:2018-11-20T20:38:02.3230462Z, RequestInitiated=Tue, 20 Nov 2018 20:38:02 GMT, RequestId=7763c56f-901e-0022-3f10-811362000000, API Version=2016-05-31, QueryParameterName=, QueryParameterValue=
        Lock Info:
          ID:        d4dcc229-3c31-8f1c-3bce-341391bd10f4
          Path:      terraform/testState
          Operation: test
          Who:       clientA
          Version:   0.12.0
          Created:   2018-11-20 20:38:00.379197 &#43;0000 UTC
          Info:
    backend_test.go:174: destroying created resources
    backend_test.go:184: Azure resources destroyed
FAIL
FAIL	github.com/hashicorp/terraform/backend/remote-state/azure	137.146s

And also fails in this branch:

$ TF_ACC=1 envchain azurerm go test -v ./backend/remote-state/azure/ -run="TestBackendLocked"
=== RUN   TestBackendLocked
2018/11/20 21:35:17 Testing if Service Principal / Client Certificate is applicable for Authentication..
2018/11/20 21:35:17 Testing if Service Principal / Client Secret is applicable for Authentication..
2018/11/20 21:35:17 Using Service Principal / Client Secret for Authentication
2018/11/20 21:35:17 Creating Resource Group "acctestrg-backend-9xc2"
2018/11/20 21:35:20 Creating Storage Account "acctestsa9xc2" in Resource Group "acctestrg-backend-9xc2"
2018/11/20 21:35:40 fetching access key for storage account
2018/11/20 21:35:40 Creating Container "acctestcont" in Storage Account "acctestsa9xc2" (Resource Group "acctestrg-backend-9xc2")
2018/11/20 21:35:41 [DEBUG] Could not lock as state blob did not exist, creating with empty state
2018/11/20 21:35:46 [DEBUG] Deleting Resource Group "acctestrg-backend-9xc2"..
2018/11/20 21:35:47 [DEBUG] Waiting for deletion of Resource Group "acctestrg-backend-9xc2"..
--- FAIL: TestBackendLocked (123.95s)
    backend_test.go:75: TestBackendConfig on *azure.Backend with configs.synthBody{Filename:"<TestWrapConfig>", Values:map[string]cty.Value{"key":cty.StringVal("testState"), "storage_account_name":cty.StringVal("acctestsa9xc2"), "access_key":cty.StringVal("S8wiuV5VHzlzWd36JpHZKeq1qkwh0cdOcbec+vmP/Z1ETV7e+Au4QI2jkyDPRiD53GWLAAKS6th8muxOT5F8mA=="), "container_name":cty.StringVal("acctestcont")}}
    backend_test.go:82: TestBackendConfig on *azure.Backend with configs.synthBody{Filename:"<TestWrapConfig>", Values:map[string]cty.Value{"access_key":cty.StringVal("S8wiuV5VHzlzWd36JpHZKeq1qkwh0cdOcbec+vmP/Z1ETV7e+Au4QI2jkyDPRiD53GWLAAKS6th8muxOT5F8mA=="), "container_name":cty.StringVal("acctestcont"), "key":cty.StringVal("testState"), "storage_account_name":cty.StringVal("acctestsa9xc2")}}
    backend_test.go:89: TestBackend: testing state locking for *azure.Backend
    backend_test.go:90: TestBackend: testing state locking for *azure.Backend
    backend_test.go:90: could not unlock with the reported ID "a1e0bd92-5967-e906-ccff-dd5a04e590f0": failed to delete lock info from metadata: storage: service returned error: StatusCode=412, ErrorCode=LeaseIdMissing, ErrorMessage=There is currently a lease on the blob and no lease ID was specified in the request.
        RequestId:e7446ee4-101e-0088-8010-81dd44000000
        Time:2018-11-20T20:35:45.6724620Z, RequestInitiated=Tue, 20 Nov 2018 20:35:45 GMT, RequestId=e7446ee4-101e-0088-8010-81dd44000000, API Version=2016-05-31, QueryParameterName=, QueryParameterValue=
        Lock Info:
          ID:        a1e0bd92-5967-e906-ccff-dd5a04e590f0
          Path:      acctestcont/testState
          Operation: test
          Who:       clientA
          Version:   0.12.0
          Created:   2018-11-20 20:35:43.604863 &#43;0000 UTC
          Info:
FAIL
FAIL	github.com/hashicorp/terraform/backend/remote-state/azure	124.944s

The underlying issue is being tracked in #17046 - since this functionality is currently broken in master, I'm going to tackle this in the next PR, since this PR's already quite large.

backend/remote-state/azure/arm_client.go

backend/remote-state/azure/backend.go

tombuildsstuff · 2018-11-21T15:05:09Z

Tests prior to updating the names:

backend/remote-state/azure/arm_client.go

backend/remote-state/azure/helpers_test.go

- updating to v21.3.0 of github.com/Azure/azure-sdk-for-go - updating to v10.15.4 of github.com/Azure/go-autorest - vendoring github.com/hashicorp/go-azure-helpers @ 0.1.1

- refactoring the backend to use a shared client via the new auth package - adding tests covering both Service Principal and Access Key auth - support for authenticating using a proxy - rewriting the backend documentation to include examples of both authentication types

tombuildsstuff · 2018-11-21T17:17:20Z

Now that we're testing both Access Key and Service Principal authentication explicitly - the failing test has been duplicated (which is intentional) - but this otherwise passes on Azure Public & Azure Germany:

The failing tests in question (which will be fixed in the follow up PR):

radeksimko

Overall LGTM except the log level thingy.

backend/remote-state/azure/sender.go

radeksimko · 2018-11-21T18:05:00Z

Also this certainly needs review from someone more familiar with Azure than me.

katbyte

LGTM @tombuildsstuff 👍

… var

ghost · 2020-03-31T02:01:36Z

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

tombuildsstuff added enhancement dependencies Auto-pinning backend/azure labels Nov 20, 2018

tombuildsstuff commented Nov 20, 2018

View reviewed changes

backend/remote-state/azure/arm_client.go Outdated Show resolved Hide resolved

tombuildsstuff commented Nov 20, 2018

View reviewed changes

backend/remote-state/azure/backend.go Show resolved Hide resolved

tombuildsstuff changed the title ~~[WIP] AzureRM Backend: SDK upgrades~~ AzureRM Backend: upgrading the SDK / support for proxies Nov 21, 2018

tombuildsstuff force-pushed the f/backend/azurerm-upgrade branch 2 times, most recently from 9dce791 to 2bf839d Compare November 21, 2018 16:20

tombuildsstuff commented Nov 21, 2018

View reviewed changes

backend/remote-state/azure/arm_client.go Show resolved Hide resolved

tombuildsstuff commented Nov 21, 2018

View reviewed changes

backend/remote-state/azure/helpers_test.go Outdated Show resolved Hide resolved

tombuildsstuff requested review from katbyte and a team November 21, 2018 16:49

tombuildsstuff added 2 commits November 21, 2018 17:58

vendor updates

3a756da

- updating to v21.3.0 of github.com/Azure/azure-sdk-for-go - updating to v10.15.4 of github.com/Azure/go-autorest - vendoring github.com/hashicorp/go-azure-helpers @ 0.1.1

tombuildsstuff force-pushed the f/backend/azurerm-upgrade branch from 5833415 to 942007f Compare November 21, 2018 17:01

tombuildsstuff requested a review from radeksimko November 21, 2018 17:03

tombuildsstuff mentioned this pull request Nov 21, 2018

Update vendor of Azure/azure-sdk-for-go to v21.1.0 #18968

Closed

tombuildsstuff changed the title ~~AzureRM Backend: upgrading the SDK / support for proxies~~ backend/azurerm: upgrading the SDK / support for proxies Nov 21, 2018

radeksimko reviewed Nov 21, 2018

View reviewed changes

backend/remote-state/azure/sender.go Outdated Show resolved Hide resolved

katbyte approved these changes Nov 21, 2018

View reviewed changes

tombuildsstuff added 2 commits November 21, 2018 21:43

switching to use the build-in logging function

92988a7

documenting it's also possible to retrieve the access key from an env…

639ac3f

… var

tombuildsstuff removed the request for review from a team November 21, 2018 20:47

tombuildsstuff merged commit 0ec109b into master Nov 21, 2018

tombuildsstuff deleted the f/backend/azurerm-upgrade branch November 21, 2018 21:06

tombuildsstuff added a commit that referenced this pull request Nov 21, 2018

Updating to include #19414

9f15381

tombuildsstuff mentioned this pull request Nov 21, 2018

Support MSI (like EC2 roles) authentication when using a remote backend? #18425

Closed

ghost locked and limited conversation to collaborators Mar 31, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

backend/azurerm: upgrading the SDK / support for proxies #19414

backend/azurerm: upgrading the SDK / support for proxies #19414

tombuildsstuff commented Nov 20, 2018 •

edited

Loading

tombuildsstuff commented Nov 20, 2018

tombuildsstuff commented Nov 21, 2018

tombuildsstuff commented Nov 21, 2018

radeksimko left a comment

radeksimko commented Nov 21, 2018

katbyte left a comment

ghost commented Mar 31, 2020

backend/azurerm: upgrading the SDK / support for proxies #19414

backend/azurerm: upgrading the SDK / support for proxies #19414

Conversation

tombuildsstuff commented Nov 20, 2018 • edited Loading

tombuildsstuff commented Nov 20, 2018

tombuildsstuff commented Nov 21, 2018

tombuildsstuff commented Nov 21, 2018

radeksimko left a comment

Choose a reason for hiding this comment

radeksimko commented Nov 21, 2018

katbyte left a comment

Choose a reason for hiding this comment

ghost commented Mar 31, 2020

tombuildsstuff commented Nov 20, 2018 •

edited

Loading