upi/vsphere: add vsphere to cloudproviderconfig #1516

staebler · 2019-04-02T02:13:50Z

Add the config for vsphere to cloudproviderconfig.
Add vsphere cloud creds to cloud-creds-secret.yaml.
Add fields to vsphere platform in installconfig to support cloudproviderconfig.
Set the enable_disk_uuid attribute to true for vsphere VMs as that is required to ensure the VMDK properly mounts disks.

See https://vmware.github.io/vsphere-storage-for-kubernetes/documentation/existing.html
for details on the vsphere cloud config file.

staebler · 2019-04-02T02:14:20Z

/hold

This builds on top of #1479.

staebler · 2019-04-02T03:20:05Z

$ cat > install-config.yaml << EOF
apiVersion: v1beta4
baseDomain: devcluster.openshift.com
metadata:
  name: mstaeble
networking:
  machineCIDR: "139.178.89.192/26"
platform:
  vsphere:
    virtualCenters:
    - name: vcsa.vmware.devcluster.openshift.com
      username: dummy@e2e.local
      password: MyPassword
      datacenters:
      - dc1
    workspace:
      defaultDatastore: nvme-ds1
    scsiControllerType: pvscsi
    publicNetwork: VM Network
pullSecret: redacted
sshKey: redacted
EOF

$ openshift-install create manifests
INFO Consuming "Install Config" from target directory

$ cat manifests/cloud-provider-config
apiVersion: v1
data:
  config: |-
    [Global]
    secret-name = "vsphere-creds"
    secret-namespace = "kube-system"

    [VirtualCenter "vcsa.vmware.devcluster.openshift.com"]
    datacenters = "dc1"

    [Workspace]
    server = "vcsa.vmware.devcluster.openshift.com"
    datacenter = "dc1"
    default-datastore = "nvme-ds1"
    resourcepool-path = "mstaeble"
    folder = "mstaeble"

    [Disk]
    scsicontrollertype = "pvscsi"

    [Network]
    public-network = "VM Network"
kind: ConfigMap
metadata:
  creationTimestamp: null
  name: cloud-provider-config
  namespace: openshift-config

$ cat openshift/99_cloud-creds-secret.yaml
kind: Secret
apiVersion: v1
metadata:
  namespace: kube-system
  name: vsphere-creds
data:
  vcsa.vmware.devcluster.openshift.com.username: ZHVtbXlAZTJlLmxvY2Fs
  vcsa.vmware.devcluster.openshift.com.password: TXlQYXNzd29yZA==

DanyC97 · 2019-04-02T13:57:51Z

@staebler quick question if you don't mind asking: have you considered/ is on the table for future work to obfuscate the below credentials ?

 vsphere:
    virtualCenters:
    - name: vcsa.vmware.devcluster.openshift.com
      username: dummy@e2e.local
      password: MyPassword

Some users might have use an admin user and i personally would not feel comfortable to put that in clear as everyone will then be able to use the infrastructure.

Same goes with the Secret as is a simple base64 decode.

kind: Secret
apiVersion: v1
metadata:
  namespace: kube-system
  name: vsphere-creds
data:
  vcsa.vmware.devcluster.openshift.com.username = ZHVtbXlAZTJlLmxvY2Fs
  vcsa.vmware.devcluster.openshift.com.password = TXlQYXNzd29yZA==

i know and understand that it might be a bigger discussion however i thought i should mention (although i'm sure is already known)

staebler · 2019-04-02T14:51:29Z

@staebler quick question if you don't mind asking: have you considered/ is on the table for future work to obfuscate the below credentials ?

@DanyC97 Obfuscation would be good. I'm not sure what options we have, though. We are limited by what the vsphere provider supports, which as far as I know is an un-obfuscated username and password. Do you have any suggestion for how we could add obfuscation?

staebler · 2019-04-02T18:52:32Z

/cc @gnufied How does this look to you with regards to setting up the vSphere cloud config?

pkg/types/vsphere/validation/platform.go

gnufied · 2019-04-02T19:29:16Z

@staebler this is looking good. I just have a minor comment about some of validation.

DanyC97 · 2019-04-02T21:44:58Z

@staebler quick question if you don't mind asking: have you considered/ is on the table for future work to obfuscate the below credentials ?

@DanyC97 Obfuscation would be good. I'm not sure what options we have, though. We are limited by what the vsphere provider supports, which as far as I know is an un-obfuscated username and password. Do you have any suggestion for how we could add obfuscation?

@staebler i'm not an terraform expert (still ramping up) however i'll dig a bit into and then open an issue to discuss further so i don't distract the attention on this PR.

pkg/asset/manifests/vsphere/cloudproviderconfig.go

abhinavdahiya · 2019-04-03T00:21:57Z

pkg/asset/manifests/vsphere/cloudproviderconfig.go

+public-network = "{{.PublicNetwork}}"`
+
+// CloudProviderConfig generates the cloud provider config for the vSphere platform.
+func CloudProviderConfig(installConfig *types.InstallConfig) (string, error) {


nit: start with Vpshere platform?

pkg/asset/manifests/openshift.go

pkg/types/vsphere/defaults/platform_test.go

abhinavdahiya · 2019-04-03T00:27:34Z

/hold

This builds on top of #1479.

/hold cancel

@staebler feel free to carry the change from #1479 as part of this PR. i'll close #1479 when this lands.

abhinavdahiya · 2019-04-03T00:32:25Z

@staebler quick question if you don't mind asking: have you considered/ is on the table for future work to obfuscate the below credentials ?
 vsphere:
    virtualCenters:
    - name: vcsa.vmware.devcluster.openshift.com
      username: dummy@e2e.local
      password: MyPassword
Some users might have use an admin user and i personally would not feel comfortable to put that in clear as everyone will then be able to use the infrastructure.

The install-config does end up on the cluster, as config map. it might be necessary to remove this password when we store the configmap here

Same goes with the Secret as is a simple base64 decode.

kind: Secret
apiVersion: v1
metadata:
  namespace: kube-system
  name: vsphere-creds
data:
  vcsa.vmware.devcluster.openshift.com.username = ZHVtbXlAZTJlLmxvY2Fs
  vcsa.vmware.devcluster.openshift.com.password = TXlQYXNzd29yZA==

Secrets are pushed to the api and then they are always kept encrypted in etcd like any other secret. Only valid users has access to secrets. So there is no requirement for doing anything here.

i know and understand that it might be a bigger discussion however i thought i should mention (although i'm sure is already known)

This configuration file is used to configure the Kubernetes cloud provider integration when using the built-in cloud provider integration or the external cloud controller manager. The configmap containing cloud provider config must be in `openshift-config` namespace as required by openshift/api PR 245 [1] [1]: https://github.com/openshift/api/blob/master/config/v1/types_infrastructure.go#L28

staebler · 2019-04-03T16:49:32Z

The install-config does end up on the cluster, as config map. it might be necessary to remove this password when we store the configmap here

The vSphere usernames and passwords are now redacted from the install-config.

Add the config for vsphere to cloudproviderconfig. Add vsphere cloud creds to cloud-creds-secret.yaml. Add fields to vsphere platform in installconfig to support cloudproviderconfig. Set the enable_disk_uuid attribute to true for vsphere VMs as that is required to ensure the VMDK properly mounts disks. See https://vmware.github.io/vsphere-storage-for-kubernetes/documentation/existing.html for details on the vsphere cloud config file.

staebler · 2019-04-03T18:05:33Z

Updated output

$ cat > install-config.yaml << EOF
apiVersion: v1beta4
baseDomain: devcluster.openshift.com
metadata:
  name: mstaeble
networking:
  machineCIDR: "139.178.89.192/26"
platform:
  vsphere:
    virtualCenters:
    - name: vcsa.vmware.devcluster.openshift.com
      username: dummy@e2e.local
      password: MyPassword
      datacenters:
      - dc1
    workspace:
      defaultDatastore: nvme-ds1
    scsiControllerType: pvscsi
    publicNetwork: VM Network
pullSecret: redacted
sshKey: redacted
EOF

$ openshift-install create manifests
INFO Consuming "Install Config" from target directory

$ cat manifests/cloud-provider-config.yaml
apiVersion: v1
data:
  config: |+
    [Global]
    secret-name      = vsphere-creds
    secret-namespace = kube-system

    [Workspace]
    server            = vcsa.vmware.devcluster.openshift.com
    datacenter        = dc1
    default-datastore = nvme-ds1
    folder            = mstaeble

    [Disk]
    scsicontrollertype = pvscsi

    [Network]
    public-network = VM Network

    [VirtualCenter "vcsa.vmware.devcluster.openshift.com"]
    datacenters = dc1

kind: ConfigMap
metadata:
  creationTimestamp: null
  name: cloud-provider-config
  namespace: openshift-config

$ cat openshift/99_cloud-creds-secret.yaml
kind: Secret
apiVersion: v1
metadata:
  namespace: kube-system
  name: vsphere-creds
data:
  vcsa.vmware.devcluster.openshift.com.username: ZHVtbXlAZTJlLmxvY2Fs
  vcsa.vmware.devcluster.openshift.com.password: TXlQYXNzd29yZA==

abhinavdahiya · 2019-04-03T18:27:00Z

/approve

abhinavdahiya · 2019-04-05T15:19:17Z

/lgtm

openshift-ci-robot · 2019-04-05T15:20:07Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: abhinavdahiya, staebler

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [abhinavdahiya,staebler]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

openshift-ci-robot added needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Apr 2, 2019

openshift-ci-robot added do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. approved Indicates a PR has been approved by an approver from all required OWNERS files. labels Apr 2, 2019

openshift-ci-robot requested review from crawford and smarterclayton April 2, 2019 02:15

gnufied reviewed Apr 2, 2019

View reviewed changes

pkg/types/vsphere/validation/platform.go Outdated Show resolved Hide resolved

abhinavdahiya reviewed Apr 3, 2019

View reviewed changes

pkg/asset/manifests/vsphere/cloudproviderconfig.go Outdated Show resolved Hide resolved

abhinavdahiya reviewed Apr 3, 2019

View reviewed changes

pkg/asset/manifests/openshift.go Outdated Show resolved Hide resolved

abhinavdahiya reviewed Apr 3, 2019

View reviewed changes

pkg/types/vsphere/defaults/platform_test.go Show resolved Hide resolved

openshift-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Apr 3, 2019

gnufied mentioned this pull request Apr 3, 2019

Sync cloudprovider configmap openshift/cluster-kube-controller-manager-operator#206

Merged

openshift-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Apr 3, 2019

openshift-ci-robot assigned abhinavdahiya Apr 5, 2019

openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Apr 5, 2019

openshift-merge-robot merged commit 976f49d into openshift:master Apr 5, 2019

vrutkovs mentioned this pull request Apr 8, 2019

Run vsphere UPI tests for installer repo openshift/release#3305

Merged

8 tasks

sdodson mentioned this pull request Apr 8, 2019

manifests: add cloudproviderconfig asset #1479

Closed

wking mentioned this pull request Apr 25, 2019

data/bootstrap: update the mco bootstrap to include the cloudconf file path #1679

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

upi/vsphere: add vsphere to cloudproviderconfig #1516

upi/vsphere: add vsphere to cloudproviderconfig #1516

staebler commented Apr 2, 2019

staebler commented Apr 2, 2019

staebler commented Apr 2, 2019 •

edited

Loading

DanyC97 commented Apr 2, 2019

staebler commented Apr 2, 2019

staebler commented Apr 2, 2019

gnufied commented Apr 2, 2019

DanyC97 commented Apr 2, 2019

abhinavdahiya Apr 3, 2019

abhinavdahiya commented Apr 3, 2019

abhinavdahiya commented Apr 3, 2019

staebler commented Apr 3, 2019

staebler commented Apr 3, 2019

abhinavdahiya commented Apr 3, 2019

abhinavdahiya commented Apr 5, 2019

openshift-ci-robot commented Apr 5, 2019

upi/vsphere: add vsphere to cloudproviderconfig #1516

upi/vsphere: add vsphere to cloudproviderconfig #1516

Conversation

staebler commented Apr 2, 2019

staebler commented Apr 2, 2019

staebler commented Apr 2, 2019 • edited Loading

DanyC97 commented Apr 2, 2019

staebler commented Apr 2, 2019

staebler commented Apr 2, 2019

gnufied commented Apr 2, 2019

DanyC97 commented Apr 2, 2019

abhinavdahiya Apr 3, 2019

Choose a reason for hiding this comment

abhinavdahiya commented Apr 3, 2019

abhinavdahiya commented Apr 3, 2019

staebler commented Apr 3, 2019

staebler commented Apr 3, 2019

abhinavdahiya commented Apr 3, 2019

abhinavdahiya commented Apr 5, 2019

openshift-ci-robot commented Apr 5, 2019

staebler commented Apr 2, 2019 •

edited

Loading