[SDK] Add more unit tests for TrainingClient APIs

[andreyvelich]

We need to add more unit tests for TrainingClient APIs by updating the training_client_test.py.
That should help us to detect bugs like this one: #2160.
Let's use this issue to track unit tests for various public APIs.

Feel free to submit PR to add unit tests for one of the following APIs, just comment on this issue on which unit tests you are working on:

• create_job Training operator SDK unit test #1938
• wait_for_job_conditions [SDK] Add UTs for wait_for_job_conditions #2196
• get_job [SDK] test: add unit test for get_job method of the training_client #2205
• list_jobs [SDK] test: add unit test for list_jobs method of the training_client #2267
• get_job_conditions [SDK] Training Client Conditions related unit tests #2253
• is_job_created [SDK] Training Client Conditions related unit tests #2253
• is_job_running [SDK] Training Client Conditions related unit tests #2253
• is_job_restarting [SDK] Training Client Conditions related unit tests #2253
• is_job_succeeded [SDK] Training Client Conditions related unit tests #2253
• is_job_failed [SDK] Training Client Conditions related unit tests #2253
• get_job_pods [SDK] Add more unit tests for TrainingClient APIs - get_job_pods #2175
• get_job_pod_names [SDK] Unit tests for TrainingClient APIs - get_job_pod_names and update_job #2192
• get_job_logs [SDK] test: add unit test for get_job_logs method of the training_client #2275
• update_job [SDK] Unit tests for TrainingClient APIs - get_job_pod_names and update_job #2192
• delete_job [SDK] test: add unit tests for delete_job() method #2232

cc @kubeflow/wg-training-leads @deepanker13 @Electronic-Waste @tariq-hasan

/good-first-issue
/area sdk
/area testing

[google-oss-prow]

@andreyvelich:
This request has been marked as suitable for new contributors.

Please ensure the request meets the requirements listed here.

If this request no longer meets these requirements, the label can be removed
by commenting with the /remove-good-first-issue command.

In response to this:

We need to add more unit tests for TrainingClient APIs by updating the training_client_test.py.
That should help us to detect bugs like this one: #2160.
Let's use this issue to track unit tests for various public APIs.

Feel free to submit PR to add unit tests for one of the following APIs, just comment on this issue on which unit tests you are working on:

• create_job
• wait_for_job_conditions
• get_job
• list_jobs
• get_job_conditions
• is_job_created
• is_job_running
• is_job_restarting
• is_job_succeeded
• is_job_failed
• get_job_pods
• get_job_pod_names
• get_job_logs
• update_job
• delete_job

cc @kubeflow/wg-training-leads @deepanker13 @Electronic-Waste @tariq-hasan

/good-first-issue
/area sdk
/area testing

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[ayushrakesh]

Hello @andreyvelich I am working on writing unit test for get_job, so should I seperate the test_data both for create_job and get_job individually or combine them into single test_data by adding one more identifier to each sample which tells for which function ( create_job or get_job) the test is for?

Example test data -

( "create_job_valid_flow_to_create_job_using_image", "create_job", { "name": "test-job", "namespace": "test", "base_image": "docker.io/test-training", "num_workers": 2, }, "success", )

( "get_job_valid", "get_job", {"namespace": "test", "job_name": "valid_job"}, {"mock": "job_response"} ),

[andreyvelich]

Thank you for your contribution @ayushrakesh!

I think it is easier to separate tests data for create_job and get_job API, so it will be easier for us to maintain those lists.

WDYT @droctothorpe @deepanker13 @kubeflow/wg-training-leads ?

[tenzen-y]

Thank you for your contribution @ayushrakesh!

I think it is easier to separate tests data for create_job and get_job API, so it will be easier for us to maintain those lists.

WDYT @droctothorpe @deepanker13 @kubeflow/wg-training-leads ?

As my understanding, this is unit tests, not integration tests. So, dedicated test data every for functions would be better.

[droctothorpe]

In general, if you can recycle fixtures to reduce redundancy, that's usually advisable; whether or not that's possible depends on the expected IO of the two functions; create_jobs doesn't return anything (though maybe it should), so I'm not sure if it makes sense to consolidate the fixtures in this context.

[Electronic-Waste]

@andreyvelich Thank you for guiding me in contributing to Training-Operator and Katib.

I'll write some test cases for training_client_test.py as soon as I understand the whole logics and the source code of Training-Operator.

[andreyvelich]

Thank you @Electronic-Waste.
So, maybe @ayushrakesh can create unit test for get_job API and @Electronic-Waste can work on
wait_for_job_conditions API.

[YosiElias]

I'm interested in working on the unit test for get_job_pods, any objection?

[andreyvelich]

Sure, thank you for your time @YosiElias!
/assign @ayushrakesh @Electronic-Waste @YosiElias

[deepanker13]

Having a different list will make code more readable, Also should we create a folder and have a test file for each function to make it more organised? @andreyvelich

[andreyvelich]

Having a different list will make code more readable, Also should we create a folder and have a test file for each function to make it more organised? @andreyvelich

@deepanker13 Usually, test file have the same name as the actual file with _test suffix,
E.g. you can check the KFP client example: https://github.com/kubeflow/pipelines/tree/master/sdk/python/kfp/client.

[tariq-hasan]

Will the number of test cases be too many to keep in a single training_client_test module?

I was thinking of moving test data, fixtures and utility functions from the katib_client_test module, for example, and putting these in newly created test_data, conftest and utils modules.

That way, I was thinking that the code may be easier to maintain in the katib_client_test module.

And katib_client_test and its associated test_data, conftest and utils modules would be moved to a newly created sdk/python/v1beta1/kubeflow/katib/tests folder.

Please let me know your thoughts.

Reference: kubeflow/katib@master...tariq-hasan:katib:add-unit-test-for-katib-client-tune

[andreyvelich]

@tariq-hasan I think, we discussed before with @droctothorpe that we want to keep _test file alongside with actual files: kubeflow/katib#2184 (comment).
I understand that this cause file to grow, but we are already doing the same for go unit tests. For example: https://github.com/kubeflow/katib/blob/master/pkg/webhook/v1beta1/experiment/validator/validator_test.go.

I feel that the test data should be associated with function that we test: get_job, create_job, train, etc.
KFP follows the same way: https://github.com/kubeflow/pipelines/tree/master/sdk/python/kfp/client.

Any concerns you see with that @tariq-hasan ?

[tariq-hasan]

That sounds good. Thanks for the clarification.

[YosiElias]

I'm interested in working on the unit test for get_job_pod_names and update_job, any objection?

[andreyvelich]

Absolutely, thank you @YosiElias!

[Electronic-Waste]

@andreyvelich UTs for wait_for_job_conditions have been added now! PTAL👀 when you are available.

Also, can I work on get_job_conditions and related functions whose names begin with is_job_?

[Bobbins228]

Hey @andreyvelich,

I can take a look at get_job and delete_job

[seanlaii]

Hi @andreyvelich, I am interested in contributing to training-operator. Can I take list_jobs? Thank you.

[tenzen-y]

Hi @andreyvelich, I am interested in contributing to training-operator. Can I take list_jobs? Thank you.

Feel free to take the task.

[seanlaii]

Hi, is anyone working on get_job_logs? If not, I can also work on it.

[andreyvelich]

@seanlaii Sure, please feel free to take unit test for get_job_logs API!

[andreyvelich]

Right now, we have unit test for every single public API of TrainingClient() 🎉
Thanks everyone for your time and work!
@deepanker13 @Electronic-Waste @Bobbins228 @seanlaii @YosiElias