v1alpha2 clusterctl fails on pivot

[aaroniscode]

What steps did you take and what happened:

Using the following cluster.yaml

apiVersion: cluster.x-k8s.io/v1alpha2
kind: Cluster
metadata:
  name: management-cluster
spec:
  clusterNetwork:
    pods:
      cidrBlocks:
      - 10.0.0.0/16
  infrastructureRef:
    apiVersion: infrastructure.cluster.x-k8s.io/v1alpha2
    kind: AWSCluster
    name: management-cluster
---
apiVersion: infrastructure.cluster.x-k8s.io/v1alpha2
kind: AWSCluster
metadata:
  name: management-cluster
spec:
  networkSpec:
    subnets:
    - id: subnet-abcdef01
    - id: subnet-abcdef01
    - id: subnet-abcdef01
    - id: subnet-abcdef01
    - id: subnet-abcdef01
    - id: subnet-abcdef01
    vpc:
      id: vpc-abcdef01
  region: us-gov-west-1
  sshKeyName: default

Running clusterctl like this:

clusterctl create cluster -v 4 \
  --bootstrap-type kind \
  --bootstrap-cluster-cleanup=false \
  --provider aws \
  -m ./machines.yaml \
  -c ./cluster.yaml \
  -p ./provider-components.yaml \
  -a ./addons.yaml

Then worked around issue #1537 by deploying the AWSCluster and AWSMachine yamls directly to the kind cluster.

clusterctl fails with:

[user@host 00-management-cluster]$ ./create-management-cluster.sh
NOTICE: clusterctl has been deprecated in v1alpha2 and will be removed in a future version.
I1014 22:37:37.491340   29032 createbootstrapcluster.go:27] Preparing bootstrap cluster
I1014 22:37:37.491373   29032 kind.go:70] Running: kind [create cluster --name=clusterapi-0295f]
I1014 22:38:28.524409   29032 kind.go:73] Ran: kind [create cluster --name=clusterapi-0295f] Output: Creating cluster "clusterapi-0295f" ...
 • Ensuring node image (kindest/node:v1.15.3) 🖼  ...
 ✓ Ensuring node image (kindest/node:v1.15.3) 🖼
 • Preparing nodes 📦  ...
 ✓ Preparing nodes 📦
 • Creating kubeadm config 📜  ...
 ✓ Creating kubeadm config 📜
 • Starting control-plane 🕹️  ...
 ✓ Starting control-plane 🕹️
 • Installing CNI 🔌  ...
 ✓ Installing CNI 🔌
 • Installing StorageClass 💾  ...
 ✓ Installing StorageClass 💾
Cluster creation complete. You can now use the cluster with:

 

export KUBECONFIG="$(kind get kubeconfig-path --name="clusterapi-0295f")"
kubectl cluster-info
I1014 22:38:28.524453   29032 kind.go:70] Running: kind [get kubeconfig-path --name=clusterapi-0295f]
I1014 22:38:28.561767   29032 kind.go:73] Ran: kind [get kubeconfig-path --name=clusterapi-0295f] Output: /home/rupadhya/.kube/kind-config-clusterapi-0295f
I1014 22:38:28.582266   29032 clusterdeployer.go:83] Applying Cluster API stack to bootstrap cluster
I1014 22:38:28.582285   29032 applyclusterapicomponents.go:26] Applying Cluster API Provider Components
I1014 22:38:28.582299   29032 clusterclient.go:812] Waiting for kubectl apply...
I1014 22:38:30.107301   29032 clusterclient.go:842] Waiting for Cluster resources to be listable...
I1014 22:38:30.148431   29032 clusterdeployer.go:88] Provisioning target cluster via bootstrap cluster
I1014 22:38:30.157568   29032 applycluster.go:44] Creating cluster object management-cluster in namespace "default"
I1014 22:38:30.164908   29032 clusterdeployer.go:101] Creating control plane machine "management-cluster-controlplane-0" in namespace "default"
I1014 22:38:30.167533   29032 applymachines.go:44] Creating machines in namespace "default"
I1014 22:38:30.183590   29032 clusterclient.go:863] Waiting for Machine management-cluster-controlplane-0 to become ready...
 
<this repeated until the AWSCluster and AWSMachines were manually applied to the kind cluster>
.....
 
I1014 22:55:30.186321   29032 clusterclient.go:863] Waiting for Machine management-cluster-controlplane-0 to become ready...
I1014 22:55:30.189347   29032 clusterdeployer.go:110] Creating target cluster
I1014 22:55:30.189364   29032 getkubeconfig.go:40] Getting target cluster kubeconfig.
I1014 22:55:30.189382   29032 getkubeconfig.go:65] Waiting for kubeconfig from Secret "management-cluster-kubeconfig" in namespace "default" to become available...
I1014 22:55:30.227636   29032 applyaddons.go:25] Applying Addons
I1014 22:55:30.227655   29032 clusterclient.go:812] Waiting for kubectl apply...
I1014 22:55:31.043986   29032 clusterdeployer.go:128] Pivoting Cluster API stack to target cluster
I1014 22:55:31.044021   29032 pivot.go:76] Applying Cluster API Provider Components to Target Cluster
I1014 22:55:31.044035   29032 clusterclient.go:812] Waiting for kubectl apply...
I1014 22:55:33.772102   29032 pivot.go:81] Pivoting Cluster API objects from bootstrap to target cluster.
I1014 22:55:33.772125   29032 pivot.go:92] Ensuring cluster v1alpha2 resources are available on the source cluster
I1014 22:55:33.772137   29032 clusterclient.go:842] Waiting for Cluster resources to be listable...
I1014 22:55:33.774305   29032 pivot.go:97] Ensuring cluster v1alpha2 resources are available on the target cluster
I1014 22:55:33.774324   29032 clusterclient.go:842] Waiting for Cluster resources to be listable...
I1014 22:55:35.179500   29032 pivot.go:102] Parsing list of cluster-api controllers from provider components
I1014 22:55:35.204634   29032 pivot.go:110] Scaling down controller capi-system/capi-controller-manager
I1014 22:55:45.222222   29032 pivot.go:110] Scaling down controller cabpk-system/cabpk-controller-manager
I1014 22:55:55.236764   29032 pivot.go:110] Scaling down controller capa-system/capa-controller-manager
I1014 22:56:05.252425   29032 pivot.go:116] Retrieving list of Clusters to move
I1014 22:56:05.254224   29032 pivot.go:166] Preparing to move Clusters: [management-cluster]
I1014 22:56:05.254245   29032 pivot.go:177] Moving Cluster default/management-cluster
I1014 22:56:05.254253   29032 pivot.go:179] Ensuring namespace "default" exists on target cluster
F1014 22:56:05.268771   29032 create_cluster.go:61] unable to pivot cluster api stack to target cluster: unable to pivot cluster API objects: Failed to move cluster: default/management-cluster: error copying Cluster default/management-cluster infrastructure reference to target cluster: error fetching unstructured object "infrastructure.cluster.x-k8s.io/v1alpha2, Kind=AWSCluster" /management-cluster: error fetching unstructured object "infrastructure.cluster.x-k8s.io/v1alpha2, Kind=AWSCluster" /management-cluster: the server could not find the requested resource

What did you expect to happen:

Cluster to be created, and pivot to occur.

Anything else you would like to add:

This may be related to #1537 . Guessing the clusterctl is not properly moving the providerref, AWSCluster and only moving the Cluster object.

Environment:

• Cluster-api version: v0.2.3
• Minikube/KIND version: v0.5.1
• Kubernetes version: (use kubectl version): v1.15.4
• OS (e.g. from /etc/os-release): Amazon Linux 2

/kind bug

[vincepri]

Which clusterctl version are you using?

[aaroniscode]

v0.2.3 ... I didn't see any closed issues or fixes related to this in newer versions, but I can test tomorrow on latest if you'd like.

[aaroniscode]

@vincepri tested with clusterctl v0.2.5 and failed in the same manner

[aaroniscode]

@vincepri this was worked around by adding namespace: default to all the InfrastructureRef's and ConfigReg's!!

So this and #1537 are probably the same issue, just errors occurring in different spots

[vincepri]

@aaroniscode Can we merge the issues together?

[aaroniscode]

@vincepri you OK if I take a look at the code and see if I can put together a PR? Might take me until next week as I need to find time.

[vincepri]

Sure thing, make sure to assign and /lifecycle active once you start working on it :)

[aaroniscode]

/assign @aaroniscode
/lifecycle active