*: fix new install core

[crawford]

This now brings up a functioning OpenTonic cluster.

[crawford]

Supersedes #280, #289, #301, and #309.

/cc @yifan-gu @abhinavdahiya @wking

[wking]

Supersedes #280, #289, #301, and #309.

Heh, this is pretty big ;). Would it be hard to land those and then rebase this on top? I don't know how intertwined these are, but I think #280 is ready for a lgtm now.

[crawford]

@wking If people are around today to /lgtm multiple times, that works for me. I just want to make sure we get this in before everyone else begins integrating. Otherwise, we'll have to integrate everything twice.

[wking]

What if there are multiple worker pools? Shouldn't these be sums?

Also, thi nil-replicas case counts as 1.

[crawford]

We should probably ensure that you can't have multiple worker pools with the same name.

[wking]

We should probably ensure that you can't have multiple worker pools with the same name.

How do you feel about the ideas in #264?

[crawford]

That makes sense. We could probably drop this method as well, since I only think we really need the master count. I'll go ahead and drop this method.

[wking]

I would much rather return []string here and only Join where that serialization is needed.

[wking]

If this is just for clients, can we update the variable names to match? And possibly update the doc dependency graph and other places using the old naming?

[crawford]

I believe that CA is for everything etcd-related now. It's probably more appropriate to change the filenames, but I didn't want to change that yet, since the old codebase assumes the CA is just for clients.

[wking]

It's probably more appropriate to change the filenames, but I didn't want to change that yet, since the old codebase assumes the CA is just for clients.

Is this the same situation as the kubeconfigs, with an external dependency relying on the specific path? Or are these internal references all we'd need to update?

$ git log -1 --oneline origin/master
c53ea04 Merge pull request #309 from crawford/tectonic
$ git grep 'etcd-client-ca\.' origin/master | cat
origin/master:installer/pkg/config-generator/tls.go:	if err := copy.Copy(filepath.Join(clusterDir, etcdCAKeyPath), filepath.Join(clusterDir, "generated/tls/etcd-client-ca.key")); err != nil {
origin/master:installer/pkg/config-generator/tls.go:	if err := copy.Copy(filepath.Join(clusterDir, etcdCACertPath), filepath.Join(clusterDir, "generated/tls/etcd-client-ca.crt")); err != nil {
origin/master:modules/bootkube/resources/bootkube.sh:			--etcd-ca=/assets/tls/etcd-client-ca.crt \
origin/master:modules/bootkube/resources/bootkube.sh:	--cacrt=/opt/tectonic/tls/etcd-client-ca.crt \
origin/master:modules/bootkube/resources/bootkube.sh:	--cakey=/opt/tectonic/tls/etcd-client-ca.key \
origin/master:modules/bootkube/resources/bootkube.sh:		--cacert=/opt/tectonic/tls/etcd-client-ca.crt \
origin/master:modules/bootkube/resources/manifests/kube-apiserver-secret.yaml:  etcd-client-ca.crt: ${etcd_ca_cert}
origin/master:modules/bootkube/resources/manifests/openshift-apiserver-secret.yaml:  etcd-client-ca.crt: ${etcd_ca_cert}
origin/master:pkg/asset/ignition/content/bootkube.go:			--etcd-ca=/assets/tls/etcd-client-ca.crt \
origin/master:pkg/asset/ignition/content/bootkube.go:	--cacrt=/opt/tectonic/tls/etcd-client-ca.crt \
origin/master:pkg/asset/ignition/content/bootkube.go:	--cakey=/opt/tectonic/tls/etcd-client-ca.key \
origin/master:pkg/asset/ignition/content/bootkube.go:		--cacert=/opt/tectonic/tls/etcd-client-ca.crt \
origin/master:pkg/asset/manifests/content/bootkube/kube-apiserver-secret.go:  etcd-client-ca.crt: {{.EtcdCaCert}}
origin/master:pkg/asset/manifests/content/bootkube/openshift-apiserver-secret.go:  etcd-client-ca.crt: {{.EtcdCaCert}}
origin/master:steps/assets/base/ignition-tls.tf:  path = "/opt/tectonic/tls/etcd-client-ca.key"
origin/master:steps/assets/base/ignition-tls.tf:  path = "/opt/tectonic/tls/etcd-client-ca.crt"
origin/master:steps/assets/base/inputs.tf:  etcd_ca_cert_pem                = "${file("${local.tls_path}/etcd-client-ca.crt")}"
origin/master:steps/assets/base/inputs.tf:  etcd_ca_key_pem                 = "${file("${local.tls_path}/etcd-client-ca.key")}"

[crawford]

As far as I know, these are all internal references, so we can change the filename as soon as we drop the old codebase (technically, we could do it now, but I'd rather not bother renaming if we are just going to delete it later).

[abhinavdahiya]

Some external dependencies:
etcd-client-ca.crt,key is expected by the static pods that run bootstrap control plane, (kube-core-operator)

kubeconfig (kubeconfig-admin) bootube (binary) expects this file to exist at auth/kubeconfig that is used by bootstrap control plane.

[crawford]

Looks like we ran out of EIPs.

* aws_eip.nat_eip.0: Error creating EIP: AddressLimitExceeded: The maximum number of addresses has been reached.

[wking]

Looks like we ran out of EIPs.

I'll run my cleanup script now, but there's a resource depdendency that keeps it from removing stale VPCs, so I still have to delete those by hand. And I'm still working my way back out of the hole from before openshift/release#1542 landed ;).

[crawford]

@wking This should be ready to go.

[crawford]

/retest

[abhinavdahiya]

/lgtm

[abhinavdahiya]

Forgot, should have approved. :P

/hold
@wking had some comments. /hold cancel

[wking]

This is lossy, and doesn't allow for an IP within the subnet. I think we want to keep this package how it was.

[crawford]

I think that's okay. IPNet is supposed to encode a network address, not a host address. Do we have any configuration options that allow a host address?

[wking]

IPNet is supposed to encode a network address, not a host address. Do we have any configuration options that allow a host address?

Maybe not yet, but recovering the initial IP seems straightforward, so I don't think we need to make project-specific assumptions in this package.

[crawford]

We should change the name of the package then, since the IPNet from the standard library is designed for the same task (the only functionality we are adding is the ability to marshal and unmarshal). I think that's probably out of scope for this PR, but I can follow up after this merges.

[wking]

We should change the name of the package then, since the IPNet from the standard library is designed for the same task (the only functionality we are adding is the ability to marshal and unmarshal).

We just add the pretty CIDR serialization. The stdlib's IPNet round-trips non-root IPs fine.

[abhinavdahiya]

@wking the round trip from pkg/ipnet
https://play.golang.org/p/4-DXPIkNHsi

2009/11/10 23:00:00 original: net.IPNet{IP:net.IP{0xc0, 0xa8, 0x0, 0xa}, Mask:net.IPMask{0xff, 0xff, 0xff, 0x0}}
2009/11/10 23:00:00 new: net.IPNet{IP:net.IP{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xff, 0xff, 0xc0, 0xa8, 0x0, 0xa}, Mask:net.IPMask{0xff, 0xff, 0xff, 0x0}}

We moved from IPv4 to IPv6.

[wking]

I havent traced this back to see if ServiceCIDR is inotialed with an IP that doesn't start the subnet. But if that's the concern you can use IP.Mask to get the initial IP. Something like:

svcCIDR := kco.installConfig.Networking.ServiceCIDR
ip, err := cidr.Host(&net.IPNet{IP: svcCIDR.IP.Mask(svcCIDR), Mask: svcCIDR.Mask}, 10)

[crawford]

/retest

[abhinavdahiya]

From previous run

Error: Error applying plan:

6 error(s) occurred:

* module.iam.aws_iam_role.worker_role: 1 error(s) occurred:

* aws_iam_role.worker_role: Error creating IAM Role ci-op-s43yrcpx-f3aac-worker-role: EntityAlreadyExists: Role with name ci-op-s43yrcpx-f3aac-worker-role already exists.
	status code: 409, request id: 1b6ddbfd-bfaa-11e8-bebb-e1016bb3b541
* module.vpc.aws_elb.console: 1 error(s) occurred:

* aws_elb.console: DuplicateLoadBalancerName: Load Balancer named ci-op-s43yrcpx-f3aac-con already exists and it is configured with different parameters.
	status code: 400, request id: 88f0ad6c-bfaa-11e8-96b2-f3018199edb6
* module.vpc.aws_elb.api_external: 1 error(s) occurred:

* aws_elb.api_external: DuplicateLoadBalancerName: Load Balancer named ci-op-s43yrcpx-f3aac-ext already exists and it is configured with different parameters.
	status code: 400, request id: 895f4e6b-bfaa-11e8-8362-83410e28af9e
* module.masters.aws_iam_role.master_role: 1 error(s) occurred:

* aws_iam_role.master_role: Error creating IAM Role ci-op-s43yrcpx-f3aac-master-role: EntityAlreadyExists: Role with name ci-op-s43yrcpx-f3aac-master-role already exists.
	status code: 409, request id: 1b6e2a09-bfaa-11e8-8095-4f644b01fa4d
* module.bootstrap.aws_iam_role.bootstrap: 1 error(s) occurred:

* aws_iam_role.bootstrap: Error creating IAM Role ci-op-s43yrcpx-f3aac-bootstrap-role: EntityAlreadyExists: Role with name ci-op-s43yrcpx-f3aac-bootstrap-role already exists.
	status code: 409, request id: 1b6e2a0a-bfaa-11e8-8095-4f644b01fa4d
* module.vpc.aws_elb.api_internal: 1 error(s) occurred:

* aws_elb.api_internal: DuplicateLoadBalancerName: Load Balancer named ci-op-s43yrcpx-f3aac-int already exists and it is configured with different parameters.
	status code: 400, request id: 899fd9b6-bfaa-11e8-8feb-8926d9d5a30d

Terraform does not automatically rollback in the face of errors.
Instead, your Terraform state file has been partially updated with
any resources that successfully completed. Please address the error
above and apply again to incrementally change your infrastructure.

@crawford looks like leaked resources.. :(
if you force push it should give a new namespace and cluster name..
Or we need to clean up the existing resources.

[crawford]

@abhinavdahiya interesting. I'd like to understand what resources were leaked.

EDIT: Looks like load balancers and IAM roles.

[wking]

/lgtm

[openshift-bot]

/retest

Please review the full test history for this PR and help us cut down flakes.

[crawford]

/retest

[wking]

EIP exhaustion.

/retest

[crawford]

EIP Exhaustion again.

[wking]

/retest

[wking]

/lgtm

[openshift-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: abhinavdahiya, crawford, wking

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [abhinavdahiya,crawford,wking]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[wking]

The smoke error was:

Invoking installer ...
time="2018-09-25T06:26:43Z" level=fatal msg="failed to get configuration from file \"inputs.yaml\": inputs.yaml is not a valid config file: failed to determine default AMI: RequestCanceled: request context canceled\ncaused by: context deadline exceeded" 
2018/09/25 06:26:44 Container setup in pod e2e-aws-smoke failed, exit code 1, reason Error

So we passed our 30-second limit on retrieving the AMI. Try again:

/retest

[wking]

The merge smoke test hit:

6 error(s) occurred:

* module.iam.aws_iam_role.worker_role: 1 error(s) occurred:

* aws_iam_role.worker_role: Error creating IAM Role ci-op-2f32p7ib-f3aac-worker-role: EntityAlreadyExists: Role with name ci-op-2f32p7ib-f3aac-worker-role already exists.
	status code: 409, request id: 81f20c43-c08f-11e8-93f5-3953970cf1ea
* module.vpc.aws_elb.api_internal: 1 error(s) occurred:

* aws_elb.api_internal: DuplicateLoadBalancerName: Load Balancer named ci-op-2f32p7ib-f3aac-int already exists and it is configured with different parameters.
	status code: 400, request id: 8f6a3a67-c08f-11e8-9ab3-a14e83e1af4f
* module.bootstrap.aws_iam_role.bootstrap: 1 error(s) occurred:

* aws_iam_role.bootstrap: Error creating IAM Role ci-op-2f32p7ib-f3aac-bootstrap-role: EntityAlreadyExists: Role with name ci-op-2f32p7ib-f3aac-bootstrap-role already exists.
	status code: 409, request id: 81f1701d-c08f-11e8-ac05-a114d28a5a6e
* module.vpc.aws_elb.console: 1 error(s) occurred:

* aws_elb.console: DuplicateLoadBalancerName: Load Balancer named ci-op-2f32p7ib-f3aac-con already exists and it is configured with different parameters.
	status code: 400, request id: 8fe86b69-c08f-11e8-9ab3-a14e83e1af4f
* module.masters.aws_iam_role.master_role: 1 error(s) occurred:

* aws_iam_role.master_role: Error creating IAM Role ci-op-2f32p7ib-f3aac-master-role: EntityAlreadyExists: Role with name ci-op-2f32p7ib-f3aac-master-role already exists.
	status code: 409, request id: 81f20c45-c08f-11e8-93f5-3953970cf1ea
* module.vpc.aws_elb.api_external: 1 error(s) occurred:

* aws_elb.api_external: DuplicateLoadBalancerName: Load Balancer named ci-op-2f32p7ib-f3aac-ext already exists and it is configured with different parameters.
	status code: 400, request id: 8f4e9c39-c08f-11e8-9f0a-bf36972b5a0a

because our destroy code is broken (possibly because of this). I removed the resources by hand, so kicking it off again:

/retest

[wking]

/retest