Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

v3.2.0 No Longer Supports Proxy Settings #198

Merged
merged 12 commits into from
Nov 7, 2022
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@ module github.com/terraform-providers/terraform-provider-http
go 1.18

require (
github.com/elazarl/goproxy v0.0.0-20221015165544-a0805db90819
github.com/hashicorp/terraform-plugin-docs v0.13.0
github.com/hashicorp/terraform-plugin-framework v0.15.0
github.com/hashicorp/terraform-plugin-framework-validators v0.5.0
Expand Down
5 changes: 5 additions & 0 deletions go.sum
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,10 @@ github.com/creack/pty v1.1.9/go.mod h1:oKZEueFk5CKHvIhNR5MUki03XCEU+Q6VDXinZuGJ3
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/elazarl/goproxy v0.0.0-20221015165544-a0805db90819 h1:RIB4cRk+lBqKK3Oy0r2gRX4ui7tuhiZq2SuTtTCi0/0=
github.com/elazarl/goproxy v0.0.0-20221015165544-a0805db90819/go.mod h1:Ro8st/ElPeALwNFlcTpWmkr6IoMFfkjXAvTHpevnDsM=
github.com/elazarl/goproxy/ext v0.0.0-20190711103511-473e67f1d7d2 h1:dWB6v3RcOy03t/bUadywsbyrQwCqZeNIEX6M1OtSZOM=
github.com/elazarl/goproxy/ext v0.0.0-20190711103511-473e67f1d7d2/go.mod h1:gNh8nYJoAm43RfaxurUnxr+N1PwuFV3ZMl/efxlIlY8=
github.com/emirpasic/gods v1.12.0 h1:QAUIPSaCu4G+POclxeqb3F+WPpdKqFGlw36+yOzGlrg=
github.com/emirpasic/gods v1.12.0/go.mod h1:YfzfFFoVP/catgzJb4IKIqXjX78Ha8FMSDh3ymbK86o=
github.com/envoyproxy/go-control-plane v0.9.0/go.mod h1:YTl/9mNaCwkRvm6d1a2C3ymFceY/DCBVvsKhRF0iEA4=
Expand Down Expand Up @@ -216,6 +220,7 @@ github.com/posener/complete v1.2.3 h1:NP0eAhjcjImqslEwo/1hq7gpajME0fTLTezBKDqfXq
github.com/posener/complete v1.2.3/go.mod h1:WZIdtGGp+qx0sLrYKtIRAruyNpv6hFCicSgv7Sy7s/s=
github.com/prometheus/client_model v0.0.0-20190812154241-14fe0d1b01d4/go.mod h1:xMI15A0UPsDsEKsMN9yxemIoYk6Tm2C1GtYGdfGttqA=
github.com/rogpeppe/fastuuid v1.2.0/go.mod h1:jVj6XXZzXRy/MSR5jhDC/2q6DgLz+nrA6LYCDYWNEvQ=
github.com/rogpeppe/go-charset v0.0.0-20180617210344-2471d30d28b4/go.mod h1:qgYeAmZ5ZIpBWTGllZSQnw97Dj+woV0toclVaRGI8pc=
github.com/rogpeppe/go-internal v1.6.1 h1:/FiVV8dS/e+YqF2JvO3yXRFbBLTIuSDkuC7aBOAvL+k=
github.com/russross/blackfriday v1.6.0 h1:KqfZb0pUVN2lYqZUYRddxF4OR8ZMURnJIG5Y3VRLtww=
github.com/russross/blackfriday v1.6.0/go.mod h1:ti0ldHuxg49ri4ksnFxlkCfN+hvslNlmVHqNRXXJNAY=
Expand Down
1 change: 1 addition & 0 deletions internal/provider/data_source_http.go
Original file line number Diff line number Diff line change
Expand Up @@ -163,6 +163,7 @@ func (d *httpDataSource) Read(ctx context.Context, req datasource.ReadRequest, r
caCertificate := model.CaCertificate

tr := &http.Transport{
Proxy: http.ProxyFromEnvironment,
Copy link

@dcaswell-square dcaswell-square Nov 1, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure this is the most complete way to handle behavior differences that were introduced in v3.2.0, as the Proxy is only one of several properties that are set on the DefaultTransport (https://github.com/golang/go/blob/f3c39a83a3076eb560c7f687cbb35eef9b506e7d/src/net/http/transport.go#L38-L54) that are no longer going to be consistent with the previous version behavior. Ignoring the proxy values was the most immediately noticeable to users, but what about all of the timeouts, MaxIdleConns, and ForceAttemptHTTP2 which will also default to different values than those from older provider versions?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would also recommend using http.DefaultTransport to prevent other potential behavior changes. There could also be consideration for using https://pkg.go.dev/github.com/hashicorp/go-cleanhttp similar to other HashiCorp maintained Go modules to prevent potential issues with multiple host connections across data sources or other regressions involving a mutated http.DefaultTransport.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Have switched to using http.DefaultTransport. I took a look at https://pkg.go.dev/github.com/hashicorp/go-cleanhttp but it seems that there are differences in the configuration of the Transport so in order to avoid changes in behaviour this has not been used at this time.

TLSClientConfig: &tls.Config{},
}

Expand Down
31 changes: 31 additions & 0 deletions internal/provider/data_source_http_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ import (
"strings"
"testing"

"github.com/elazarl/goproxy"
"github.com/hashicorp/terraform-plugin-sdk/v2/helper/resource"
)

Expand Down Expand Up @@ -471,6 +472,36 @@ func TestDataSource_UnsupportedInsecureCaCert(t *testing.T) {
})
}

func TestDataSource_HTTPViaProxyWithEnv(t *testing.T) {
svr := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(http.StatusOK)
}))
defer svr.Close()

p := goproxy.NewProxyHttpServer()

proxy := httptest.NewServer(p)
defer proxy.Close()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Over in the terraform-provider-tls acceptance testing, all the testing HTTP servers, including the proxies, have connection counting:

https://github.com/hashicorp/terraform-provider-tls/blob/b9cbb62bc2a8b90f6af91e86047082a5357e8c2f/internal/provider/testutils/local_server.go#L38-L46

Then there is a TestCheckFunc which ensures that the "real" HTTP server and proxy HTTP server have matching connection counts:

https://github.com/hashicorp/terraform-provider-tls/blob/b9cbb62bc2a8b90f6af91e86047082a5357e8c2f/internal/provider/testutils/local_server.go#L117-L127

I think we'll need something similar here to verify that the proxy is correctly being used, unless there is a better testing strategy.

Copy link
Contributor Author

@bendbennett bendbennett Nov 2, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm looking into the current test failures (e.g., https://github.com/hashicorp/terraform-provider-http/actions/runs/3376234811/jobs/5603729486).

Check 2/2 error: expected server and proxy active connection count to match: server was 1, while proxy was 0

I can't reproduce this failure locally, I see a different failure.

Check 2/2 error: expected server and proxy active connection count to match: server was 3, while proxy was 1

Copy link
Contributor Author

@bendbennett bendbennett Nov 2, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see the same failures in CI (e.g., https://github.com/hashicorp/terraform-provider-http/actions/runs/3376368850/jobs/5604018894) when using

	tr := &http.Transport{
		Proxy: http.ProxyFromEnvironment,
	}
Check 2/2 error: expected server and proxy active connection count to match: server was 1, while proxy was 0

But all tests pass when running locally with this configuration.


t.Setenv("HTTP_PROXY", proxy.URL)
t.Setenv("HTTPS_PROXY", proxy.URL)

resource.UnitTest(t, resource.TestCase{
ProtoV5ProviderFactories: protoV5ProviderFactories(),

Steps: []resource.TestStep{
{
Config: fmt.Sprintf(`
data "http" "http_test" {
url = "%s"
}
`, svr.URL),
Check: resource.TestCheckResourceAttr("data.http.http_test", "status_code", "200"),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This test looks correct to me, but for some reason is unhappy on Terraform version 0.14:

=== RUN   TestDataSource_HTTPViaProxyWithEnv
    data_source_http_test.go:489: Step 1/1 error: Check failed: Not found: data.http.http_test in [root]
--- FAIL: TestDataSource_HTTPViaProxyWithEnv (0.40s)

I'll try to do some investigative work with my end of day.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The quirk with Terraform version 0.14.x is that it will not set the Terraform state for a data source which returns a warning diagnostic. The warning being generated here is:

2022-11-01T16:50:56.310-0400 [TRACE] sdk.proto: Received downstream response: diagnostic_error_count=0 diagnostic_warning_count=1 tf_req_duration_ms=0 tf_data_source_type=http tf_req_id=5887d459-0e1a-5aa1-0241-13d21f8698ca tf_provider_addr=registry.terraform.io/hashicorp/http tf_proto_version=5.3 tf_rpc=ReadDataSource
2022-11-01T16:50:56.310-0400 [WARN]  sdk.proto: Response contains warning diagnostic: tf_provider_addr=registry.terraform.io/hashicorp/http tf_proto_version=5.3 tf_rpc=ReadDataSource tf_data_source_type=http diagnostic_severity=WARNING diagnostic_summary="Content-Type is not recognized as a text type, got \"\"" diagnostic_detail="If the content is binary data, Terraform may not properly handle the contents of the response." tf_req_id=5887d459-0e1a-5aa1-0241-13d21f8698ca
2022-11-01T16:50:56.310-0400 [TRACE] sdk.proto: Served request: tf_provider_addr=registry.terraform.io/hashicorp/http tf_proto_version=5.3 tf_rpc=ReadDataSource tf_data_source_type=http tf_req_id=5887d459-0e1a-5aa1-0241-13d21f8698ca

The data source correctly does not exit early in this case. The testing fix for this is to ensure the HTTP response also includes a valid Content-Type header, e.g.

svr := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
	w.Header().Set("Content-Type", "text/plain")
	w.WriteHeader(http.StatusOK)
}))

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the investigation. Have added the fix you suggested.

},
},
})
}

type TestHttpMock struct {
server *httptest.Server
}
Expand Down