[k8s] Support autoscaling TPU node pools

[romilbhardwaj]

Adds support for autoscaling TPU node pools on GKE.

Currently single-host TPU v5 is supported.

Users must configure autoscaler: gke and a large provision_timeout to let the cluster autoscale.

# ~/.sky/config.yaml
kubernetes:
  autoscaler: gke
  provision_timeout: 900

Tested on scale-from-zero nodepool withct5lp-hightpu-1t (tpu v5e, 1x1):

1. Script to create GKE cluster with an autoscaling TPU nodepool:

gcloud beta container --project "sky-dev-465" clusters create "tputest" --region "us-west1" --tier "standard" --no-enable-basic-auth --cluster-version "1.31.4-gke.1256000" --release-channel "regular" --machine-type "e2-standard-8" --image-type "COS_CONTAINERD" --disk-type "pd-balanced" --disk-size "100" --metadata disable-legacy-endpoints=true --scopes "https://www.googleapis.com/auth/devstorage.read_only","https://www.googleapis.com/auth/logging.write","https://www.googleapis.com/auth/monitoring","https://www.googleapis.com/auth/servicecontrol","https://www.googleapis.com/auth/service.management.readonly","https://www.googleapis.com/auth/trace.append" --num-nodes "1" --logging=SYSTEM,WORKLOAD --monitoring=SYSTEM,STORAGE,POD,DEPLOYMENT,STATEFULSET,DAEMONSET,HPA,CADVISOR,KUBELET --enable-ip-alias --network "projects/sky-dev-465/global/networks/default" --subnetwork "projects/sky-dev-465/regions/us-west1/subnetworks/default" --no-enable-intra-node-visibility --default-max-pods-per-node "110" --enable-ip-access --security-posture=standard --workload-vulnerability-scanning=disabled --no-enable-master-authorized-networks --no-enable-google-cloud-access --addons HorizontalPodAutoscaling,HttpLoadBalancing,GcePersistentDiskCsiDriver --enable-autoupgrade --enable-autorepair --max-surge-upgrade 1 --max-unavailable-upgrade 0 --binauthz-evaluation-mode=DISABLED --enable-managed-prometheus --enable-shielded-nodes && gcloud beta container --project "sky-dev-465" node-pools create "tpupool" --cluster "tputest" --region "us-west1" --machine-type "ct5lp-hightpu-1t" --image-type "COS_CONTAINERD" --disk-type "pd-balanced" --disk-size "100" --metadata disable-legacy-endpoints=true --scopes "https://www.googleapis.com/auth/devstorage.read_only","https://www.googleapis.com/auth/logging.write","https://www.googleapis.com/auth/monitoring","https://www.googleapis.com/auth/servicecontrol","https://www.googleapis.com/auth/service.management.readonly","https://www.googleapis.com/auth/trace.append" --num-nodes "0" --enable-autoscaling --total-min-nodes "0" --total-max-nodes "2" --location-policy "ANY" --enable-autoupgrade --enable-autorepair --max-surge-upgrade 1 --max-unavailable-upgrade 0

2. Run sky launch -c tpu --cloud kubernetes --gpus tpu-v5-lite-podslice

[Michaelvll]

Thanks @romilbhardwaj! LGTM.