[RayCluster] Make headpod name deterministic

[owenowenisme]

Why are these changes needed?

Related issue number

Closes #3013

Checks

( raycluster-autoscaler is the rayCluster name from a random yaml I used for testing, which is not related to the issue.)

❯ kubectl get pods -l=ray.io/is-ray-node=yes
NAME                         READY   STATUS    RESTARTS   AGE
raycluster-autoscaler-head   2/2     Running   0          9m34s

• I've made sure the tests are passing.
• Testing Strategy
  • Unit tests
  • Manual tests
  • This PR is not tested :(

[owenowenisme]

Tests that are failed by this changed will also be fixed in this PR.

[kevin85421]

1. please don't use fmt.Sprintf inside log function. See https://github.com/nginx/nginx-gateway-fabric/blob/main/docs/developer/logging-guidelines.md#message-guidelines for more details.

2. please don't print headPods.Items. Instead, print the name of head Pods.

[owenowenisme]

kuberay/ray-operator/controllers/ray/raycluster_controller.go

Lines 557 to 560 in 713ff51

	if len(services.Items) > 1 {
	logger.Info("reconcileHeadService", "Duplicate head service found", services.Items)
	return fmt.Errorf("%d head service found %v", len(services.Items), services.Items)
	}

I'm actually following the HeadService which also print the Items, should I change it as well?

[owenowenisme]

current err msg look like this

{"level":"info","ts":"2025-02-13T04:16:20.592Z","logger":"controllers.RayCluster",
"msg":"Multiple head pods found, it should only exist one head pod. Please delete extra head pods.",
"RayCluster":{"name":"rayservice-sample-raycluster-9wl9m","namespace":"test-ns-6zktp"},
"reconcileID":"6d93964f-00a4-4d78-8db6-0d40c53804e2",
"found pods":["mytest-head","rayservice-sample-raycluster-9wl9m-head"],
"should only leave":"rayservice-sample-raycluster-9wl9m-head"}

{"level":"info","ts":"2025-02-13T04:16:20.592Z",
"logger":"controllers.RayCluster",
"msg":"Reconciliation error",
"RayCluster":{"name":"rayservice-sample-raycluster-9wl9m","namespace":"test-ns-6zktp"},
"reconcileID":"6d93964f-00a4-4d78-8db6-0d40c53804e2",
"error":"2 head pods found [mytest-head rayservice-sample-raycluster-9wl9m-head]. Please delete extra head pods and leave only the head pod with name rayservice-sample-raycluster-9wl9m-head"}

[owenowenisme]

kuberay/ray-operator/controllers/ray/raycluster_controller.go

Lines 750 to 764 in fa0aeaa

	// Reconcile worker pods now
	for _, worker := range instance.Spec.WorkerGroupSpecs {
	if !r.rayClusterScaleExpectation.IsSatisfied(ctx, instance.Namespace, instance.Name, worker.GroupName) {
	logger.Info("reconcilePods", "worker group", worker.GroupName, "Expectation", "NotSatisfiedGroupExpectations, reconcile the group later")
	continue
	}
	// workerReplicas will store the target number of pods for this worker group.
	var workerReplicas int32 = utils.GetWorkerGroupDesiredReplicas(ctx, worker)
	logger.Info("reconcilePods", "desired workerReplicas (always adhering to minReplicas/maxReplica)", workerReplicas, "worker group", worker.GroupName, "maxReplicas", worker.MaxReplicas, "minReplicas", worker.MinReplicas, "replicas", worker.Replicas)
	workerPods := corev1.PodList{}
	if err := r.List(ctx, &workerPods, common.RayClusterGroupPodsAssociationOptions(instance, worker.GroupName).ToListOptions()...); err != nil {
	return err
	}

@kevin85421
I'm thinking that maybe we should change the var name from worker to something like workerGroup?
Which I think is more accurate.

BTW, I think the reconcile Worker for-loop in reconcilePod is pretty hard to read and realize for it is 150 lines and have to trace many function all over files, maybe we should refactor this part?