Temporarily decom MIT Brain hubs (bican, dandi, linc)

[yuvipanda]

Partnerships is working on reconfiguring this relationship, but it isn't going to look like 3 separate hubs on 3 separate clusters. Decom this to save our cloud costs, as nobody is currently using these.

Based on a meeting I just had with them (along with @colliand). Will have an associated issue soon.

Ref https://github.com/2i2c-org/leads/issues/344

[yuvipanda]

I've terraform destroyed everything for now.

[consideRatio]

These clusters still incurring cloud costs from k8s clusters (72 USD / month) a core node (~180 USD / month), as this was controlled via eksctl rather than terraform.

@yuvipanda if you think its OK to proceed i'll just delete the eksctl resources as well.

[yuvipanda]

@consideRatio ah, I missed that. yes, please do when you get a minute.

[consideRatio]

Done, bican, dandi, linc deleted.

jsonnet $CLUSTER_NAME.jsonnet > $CLUSTER_NAME.eksctl.yaml
# WARNING: eksctl delete cluster doesn't provide a confirmation with --approve, I've opened issue 7777 in eksctl about this https://github.com/eksctl-io/eksctl/issues/7777
eksctl delete cluster --config-file $CLUSTER_NAME.eksctl.yaml

# to help drain operation from another terminal without availability of deployer creds
eksctl utils write-kubeconfig --config-file=$CLUSTER_NAME.eksctl.yaml --auto-kubeconfig
export KUBECONFIG=/home/erik/.kube/eksctl/clusters/$CLUSTER_NAME

# eksctl reports: 6 pods are unevictable from node ip-192-168-8-78.us-east-2.compute.internal
# I think four of these referred to two coredns pods and two ebs-csi-controller
# pods with protection from PDB resources
# I think another pod was from the cryptnono daemonset, which as a daemonset
# shouldn't cause issues, but maybe it had a emptyDir volume or similar that maybe requires
# a flag that it was ok to delete also such pods?

[consideRatio]

• we'll now start seeing pagerduty warnings I think
• we'll also until this is merged risk causing additional maintenance work and confusion about bican/dandi/linc during maintenance

Due to this, I'll go for a merge - then the pagerduty healthchecks are deleted I think via automation scanning our cluster config files.

[github-actions]

🎉🎉🎉🎉

Monitor the deployment of the hubs here 👉 https://github.com/2i2c-org/infrastructure/actions/runs/9185897951

[consideRatio]

I applied the deployer:skip-deploy label as 166e53f incorrectly seemed to have caused the wrong conclusion that all hubs need to re-deploy and test.

I've opened #4116 about this

[sgibson91]

incorrectly seemed to have caused the wrong conclusion that all hubs need to re-deploy and test.

This can happen if the branch is not up-to-date with main #2766