Restructure tests to take advantage of kitchen-terraform #20

Jberlinsky · 2018-10-23T15:42:02Z

Restructures tests to take advantage of kitchen-terraform, and eliminate the use of bash scripting for test fixture generation.

wyardley · 2018-10-26T17:56:50Z

👍
I'm curious also, why does this module use gcloud CLI vs. inspec-gcp to do all the checks? It feels very "Googly" to me 😉

adrienthebo · 2018-10-26T19:55:55Z

@wyardley that's a good question! A lot of the Terraform modules in this organization originally used BATS, so it was straightforward to migrate the existing tests to Test Kitchen and InSpec by shelling out and using the command resources. Changing test frameworks is a lot of work as it stands, so moving to test-kitchen while shelling out to gcloud provides a lot of value by itself. Switching to inspec-gcp would be cleaner and more elegant, but in terms of validating correctness shelling out to gcloud provides around the same level of verification as would using inspec-gcp - but with less up front work

In some situations and modules we could use inspec-gcp, but in some cases we need test coverage for features that are exposed by gcloud but aren't yet supported by inspec-gcp. For instance GCP subnet IAM bindings/policies can be fetched by gcloud but inspec-gcp doesn't yet support them, and gcloud can enumerate activated APIs but there's no matching inspec-gcp resource. (I've been meaning to submit PRs for these features but haven't found the time to do so.) In this scenario it would be confusing to use inspec-gcp resources for some tests and shell out to gcloud for other tests.

In the long term I expect/hope that we'll move to inspec-gcp resources - it's a great project and I've had a lot of luck using it in smaller test environments with less surface area. However in the short run we aren't able to overhaul all of our tests to use test-kitchen, implement all of the inspec-gcp resources that we need, and do so while shipping new features for the modules themselves.

Does that answer your question?

adrienthebo

This code looks functionally correct and reads clearly. There are some test details leaking in that I've noted, they're not blockers but I think we'd benefit from comments clarifying their purpose.

I'll pull down this code in the next day or to and perform functional review on this PR; if that works out well I'll 👍.

adrienthebo · 2018-10-26T20:04:28Z

examples/deploy_service/outputs.tf

@@ -14,23 +14,61 @@
 * limitations under the License.
 */

-output "name_example" {
+output "project_id" {


Since we're basing our test cases off of the examples, I think we should document why we're using these passthru outputs. People just looking at these examples might be confused/surprised otherwise.

adrienthebo · 2018-10-26T20:11:21Z

examples/node_pool/outputs.tf

@@ -14,31 +14,56 @@
 * limitations under the License.
 */

-output "name_example" {
+output "project_id" {


Document passthrough outputs in example code.

Alternately, we could extract this into a secondary variables file - perhaps outputs-test.tf or somesuch? Making that change may be out of scope for this PR but an idea to think about in the future.

adrienthebo · 2018-10-26T20:11:57Z

examples/node_pool/outputs.tf

-  description = "Cluster location"
-  value       = "${module.gke.location}"
+output "ca_certificate" {
+  sensitive = true


Certificates shouldn't need to be sensitive, only private keys - does this output contain any key material?

adrienthebo · 2018-10-26T20:13:42Z

examples/stub_domains/outputs.tf

-  sensitive   = true
-  description = "Cluster endpoint"
-  value       = "${module.gke.endpoint}"
+output "network" {


Are these passthrough variables consistent across all examples?

test/fixtures/deploy_service/terraform.tfvars.sample

wyardley · 2018-10-26T21:45:54Z

@adrienthebo Absolutely! Super helpful... thanks. I was just confused when I saw the project was using inspec in the README but then noticed all those shell calls in the actual tests.

I had the same thought, that inspec-gcp might not have all of those resources available yet. I tested out for something I was working on (using some of the plumbing from here as an example), and it definitely seems really fast.

I do wish there were some better frameworks for doing expectation based unit tests of tf, vs only being able to do integration tests, but so far, I haven't found much out there.

adrienthebo · 2018-10-30T17:27:53Z

I've run into #24 while testing this PR. Since this PR didn't introduce the issue we can merge anyways, but should address this soon.

adrienthebo · 2018-10-30T18:09:14Z

I'm running into this error with the deploy-service-local and simple-zonal-local test instances, @Jberlinsky have you run into this before?

Error:

       * google_container_node_pool.zonal_pools: error creating NodePool: googleapi: Error 400: The required IP ranges for pods can't be fulfilled by the given container secondary range., badRequest

Jberlinsky · 2018-10-30T18:20:58Z

@adrienthebo What network(s)/subnet(s) are you trying to deploy the test cases into? Can you share that part of your terraform.tfvars? What's the output of gcloud beta container subnets list-usable with respect to those networks/subnets?

In general, I think this ties in with a conversation @morgante and I were having about test fixtures. Personally, I think it would be best to spin up a fixture network, with appropriate subnets, before the tests are run, and pass those subnets in dynamically. This would reduce the amount of information that needs to be conveyed to test-runners (e.g. the CIDR size restrictions for GKE clusters, which would be codified in the fixture setup), and reduce the likelihood of issues when running tests.

adrienthebo · 2018-10-30T18:26:50Z

I've documented my config in #25, and as far as I can tell /20s should have been big enough. Manually reducing the max count in the node pool from 100 to 10 has unblocked me, but that's not a viable fix to this. I'm going to keep running through this PR.

adrienthebo · 2018-10-30T19:46:04Z

I've run into a number of issues while verifying this PR, most of which are due to the early state of testing for these modules. For the sake of posterity I'm going to run through

The requirements/fixtures for the modules aren't listed (networks, subnet names, secondary ranges, secondary range sizes)
The Docker images cannot be rebuilt (Alpine Linux dropped git=2.18.10-r0 in favor of git=2.18.1-r0
An overly small subnet size can cause the node pool resources to fail, but it's hard to deduce that the secondary network size is at fault.
There's no particularly good way (that I've found) to deduce how big the secondary networks need to be.
The min_master_version value is too tightly specified (`1.10.6-gke.2") and the default value is no longer accepted.
Credentials aren't passed from Terraform to InSpec; when running kitchen verify within Docker gcloud fails because it isn't authorized.

The noteworthy bit is that out of the above issues, only the last is really related to these changes. This puts us in an unfortunate state where the changes in the PR are reasonable, but the tests take an enormous amount of work and small tweaks to run.

I see two approaches forward:

Block this pull request until everything is fixed. This will cause this pull request to balloon in scope, prevent incremental progress in other areas, but it follows best practices of only merging PRs that are fully functional.
Merge this pull request to get the improvements it does have, and then work through a number of smaller pull requests to fix the issues that I've outlined above. This has the downside of merging code into what amounts to a broken build which is pretty uncomfortable, but if we're diligent we can use this as incremental progress and build on that.

My inclination is that we spring for the second case, accept that we have a lot to work on, and focus on getting changes submitted and merged. We've been working on variations of tests for this module for a while but we haven't landed much code. I'd like to propose lowering the threshold for merging tests that are non-functional (only affecting tests) so that we can iterate much faster and keep up with the issues that are popping up.

morgante · 2018-10-30T19:55:55Z

@adrienthebo Thanks for the detailed review. I sympathize with the danger of having long-running PRs but am also reticent to merge in tests which are functionally impossible to run.

I suggest that the immediate first step is simply to document the requirements for running tests (even if it's just posting a working Terraform config into an issue and linking to that). We can then do follow-on work to improve the tests.

Next steps:

Make clear README instructions for running the tests (clearly specifying upfront work you need to do)
Open issues for all other problems found to make sure we circle back and fix them

@Jberlinsky I'm fine with having the cluster auto-create networks as part of the test fixtures, since that's comparatively much less expensive than creating projects.

Jberlinsky · 2018-10-30T19:59:43Z

@adrienthebo Thanks for the detailed review, and thanks for your thoughts, @morgante.

My preference here is to establish precedent for having the tests bootstrap networking fixtures. I'll work on that, in the process making sure that enough of the identified issues are knocked out that the tests can functionally run, and circle back here shortly.

adrienthebo · 2018-10-31T15:32:51Z

test/fixtures/node_pool/terraform.tfvars.sample

@@ -0,0 +1,8 @@
+project_id=""
+credentials_path="../../../credentials.json"


This path appears to be relative to test/fixtures/node_pool, but the other examples use ../.. which I think is relative to examples/{example}. Can we fix this up?

morgante · 2018-10-31T17:43:25Z

I accidentally merged this too soon.

pratikmallya · 2018-11-06T13:36:19Z

Will this PR be reopened? @morgante

morgante · 2018-11-06T19:07:16Z

@pratikmallya Yes, need to merge it again soon.

…modules/feature/restructure-tests Restructure tests to take advantage of kitchen-terraform

Jberlinsky force-pushed the feature/restructure-tests branch from e537811 to 48d0cab Compare October 23, 2018 15:42

Restructure tests

6162310

Jberlinsky force-pushed the feature/restructure-tests branch from 48d0cab to 6162310 Compare October 23, 2018 15:43

Jberlinsky requested review from morgante and lilithmooncohen and removed request for morgante October 23, 2018 15:43

Jberlinsky mentioned this pull request Oct 23, 2018

Private Cluster Configuration #21

Closed

morgante requested a review from adrienthebo October 26, 2018 15:44

adrienthebo reviewed Oct 26, 2018

View reviewed changes

adrienthebo mentioned this pull request Oct 30, 2018

Relax kubernetes version #24

Closed

Jberlinsky mentioned this pull request Oct 30, 2018

Document requirements for running test-kitchen tests #25

Closed

adrienthebo reviewed Oct 31, 2018

View reviewed changes

adrienthebo mentioned this pull request Oct 31, 2018

Setup gcloud credentials when running interactively within docker #28

Closed

morgante merged commit 30059af into master Oct 31, 2018

morgante mentioned this pull request Oct 31, 2018

Revert "Restructure tests to take advantage of kitchen-terraform" #29

Merged

Jberlinsky mentioned this pull request Nov 20, 2018

Build Network Fixtures and Run Tests with Kitchen-Terraform #33

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Restructure tests to take advantage of kitchen-terraform #20

Restructure tests to take advantage of kitchen-terraform #20

Jberlinsky commented Oct 23, 2018

wyardley commented Oct 26, 2018

adrienthebo commented Oct 26, 2018

adrienthebo left a comment

adrienthebo Oct 26, 2018

adrienthebo Oct 26, 2018

adrienthebo Oct 26, 2018

adrienthebo Oct 26, 2018

wyardley commented Oct 26, 2018

adrienthebo commented Oct 30, 2018

adrienthebo commented Oct 30, 2018

Jberlinsky commented Oct 30, 2018

adrienthebo commented Oct 30, 2018

adrienthebo commented Oct 30, 2018

morgante commented Oct 30, 2018

Jberlinsky commented Oct 30, 2018

adrienthebo Oct 31, 2018

morgante commented Oct 31, 2018

pratikmallya commented Nov 6, 2018

morgante commented Nov 6, 2018

		@@ -0,0 +1,8 @@
		project_id=""
		credentials_path="../../../credentials.json"

Restructure tests to take advantage of kitchen-terraform #20

Restructure tests to take advantage of kitchen-terraform #20

Conversation

Jberlinsky commented Oct 23, 2018

wyardley commented Oct 26, 2018

adrienthebo commented Oct 26, 2018

adrienthebo left a comment

Choose a reason for hiding this comment

adrienthebo Oct 26, 2018

Choose a reason for hiding this comment

adrienthebo Oct 26, 2018

Choose a reason for hiding this comment

adrienthebo Oct 26, 2018

Choose a reason for hiding this comment

adrienthebo Oct 26, 2018

Choose a reason for hiding this comment

wyardley commented Oct 26, 2018

adrienthebo commented Oct 30, 2018

adrienthebo commented Oct 30, 2018

Jberlinsky commented Oct 30, 2018

adrienthebo commented Oct 30, 2018

adrienthebo commented Oct 30, 2018

morgante commented Oct 30, 2018

Jberlinsky commented Oct 30, 2018

adrienthebo Oct 31, 2018

Choose a reason for hiding this comment

morgante commented Oct 31, 2018

pratikmallya commented Nov 6, 2018

morgante commented Nov 6, 2018