[Feature] Add an e2e test for Autoscaler to scale up by manually updating minReplicas

[LeoLiao123]

Why are these changes needed?

See #2576

Related issue number

Closes #2576

Test Log

This test follows these sequential steps:

• Create a RayCluster with the Autoscaler enabled, containing a worker group with replicas: 1 and minReplicas: 0.

{"msg": "reconcilePods", "desired workerReplicas": 1, "maxReplicas": 5, "minReplicas": 0, "replicas": 1}

• Update minReplicas from 0 to 2.
• KubeRay should create an additional Pod.

{"msg": "reconcilePods", "desired workerReplicas": 2, "maxReplicas": 5, "minReplicas": 2, "replicas": 1}
{"msg": "Number workers to add": 1, "Worker group": "test-group"}
{"msg": "Created worker Pod for RayCluster"}

• Submit some Ray detached actors to trigger autoscaling to 5 Pods.

"Executing command: [python /home/ray/test_scripts/create_detached_actor.py actor0]"
"Executing command: [python /home/ray/test_scripts/create_detached_actor.py actor1]"
"Executing command: [python /home/ray/test_scripts/create_detached_actor.py actor2]"
"Executing command: [python /home/ray/test_scripts/create_detached_actor.py actor3]"
"Executing command: [python /home/ray/test_scripts/create_detached_actor.py actor4]"

• Verify whether the Autoscaler successfully scales up to 5 Pods.

{"msg": "reconcilePods", "workerReplicas": 5, "NumOfHosts": 1, "runningPods": 2, "diff": 3}
{"msg": "Number workers to add": 3, "Worker group": "test-group"}

• Check whether replicas is set to 5.

"new status": {
    "desiredWorkerReplicas": 5,
    "minWorkerReplicas": 2,
    "maxWorkerReplicas": 5,
    "observedGeneration": 3
}

Screenshot

Checks

• I've made sure the tests are passing.
• Testing Strategy
  • Unit tests
  • Manual tests
  • This PR is not tested :(

[LeoLiao123]

@MortalHappiness @kevin85421 PTAL

[kevin85421]

cc @MortalHappiness can you review this PR?

[MortalHappiness]

Please rebase with the master branch. Thanks.