KEP-2170: Add unit and Integration tests for model and dataset initializers #2323

seanlaii · 2024-11-09T03:18:51Z

What this PR does / why we need it:
I added unit tests and integration tests for model and dataset initializers.

Which issue(s) this PR fixes (optional, in Fixes #<issue number>, #<issue number>, ... format, will close the issue(s) when PR gets merged):
Fixes #2305

Checklist:

Docs included if any changes are user facing

google-oss-prow · 2024-11-09T03:18:56Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign johnugeorge for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

seanlaii · 2024-11-09T03:19:57Z

pkg/initializer_v2/test/e2e/test_dataset.py

+            # Private HuggingFace dataset test
+            # (
+            #     "HuggingFace - Private dataset",
+            #     "huggingface",
+            #     {
+            #         "storage_uri": "hf://username/private-dataset",
+            #         "use_real_token": True,
+            #         "expected_files": ["config.json", "dataset.safetensors"],
+            #         "expected_error": None
+            #     }
+            # ),
+            # Invalid HuggingFace dataset test


Do we have an access token for testing login and downloading resources from private repo?

Not yet, maybe we can track this in a separate issue that we should create Kubeflow-owned account in HF for the Token.

pkg/initializer_v2/test/e2e/test_dataset.py

.github/workflows/test-python.yaml

pkg/initializer_v2/test/conftest.py

coveralls · 2024-11-09T04:04:53Z

Pull Request Test Coverage Report for Build 12840565631

Details

0 of 0 changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall first build on initializer-test at 100.0%

Totals
Change from base Build 12834026562:	100.0%
Covered Lines:	85
Relevant Lines:	85

💛 - Coveralls

.github/workflows/integration-tests.yaml

pkg/initializer_v2/test/e2e/test_dataset.py

seanlaii · 2024-11-26T01:39:54Z

Hi @andreyvelich ,

Could you help review this PR? I have some questions. Once the SDK's PR gets approved, I will modify it accordingly.

Thank you!

andreyvelich · 2024-11-26T14:50:27Z

@seanlaii Sorry for the delay, sure, I will review it today

pkg/initializer_v2/test/conftest.py

andreyvelich

Thank you for this effort @seanlaii!
I left my initial thoughts.
Please take a look @Electronic-Waste @deepanker13 @kubeflow/wg-training-leads @varshaprasad96 @akshaychitneni @saileshd1402

.github/workflows/integration-tests.yaml

.github/workflows/test-python.yaml

pkg/initializer_v2/test/unit/dataset/test_dataset.py

pkg/initializer_v2/test/unit/model/test_model_config.py

pkg/initializer_v2/test/unit/model/test_model.py

pkg/initializer_v2/test/unit/test_utils.py

pkg/initializer_v2/test/e2e/test_model.py

pkg/initializer_v2/test/unit/dataset/test_dataset.py

seanlaii · 2024-12-21T19:39:03Z

Hi @andreyvelich , could you help review the PR? I addressed the comments. Thank you!

andreyvelich · 2025-01-07T18:22:11Z

Sorry for the delay @seanlaii!
I will review it this week.

Electronic-Waste

@seanlaii Thanks for your contributions! I left some comments for you.

As for the e2e test's pattern, we can discuss later with @andreyvelich :)

pkg/initializer_v2/dataset/huggingface_test.py

pkg/initializer_v2/model/huggingface_test.py

pkg/initializer_v2/dataset/huggingface_test.py

pkg/initializer_v2/model/huggingface_test.py

pkg/initializer_v2/model/main_test.py

sdk_v2/kubeflow/training/types/config_test.py

seanlaii · 2025-01-14T05:02:28Z

Hi @andreyvelich @Electronic-Waste , I addressed the comments. Please help review the PR when you are available. Thank you!

pkg/initializer_v2/model/main_test.py

pkg/initializer_v2/dataset/main_test.py

test/integration/initializer_v2/__init__.py

pkg/initializer_v2/model/main_test.py

test/integration/initializer_v2/dataset_test.py

pkg/initializer_v2/utils/utils_test.py

test/__init__.py

seanlaii · 2025-01-16T07:34:36Z

test/integration/initializer_v2/dataset_test.py

@@ -0,0 +1,75 @@
+import os
+import runpy
+from test.integration.initializer_v2.utils import setup_temp_path  # noqa: F401


Add # noqa: F401 for bypassing lint error
F401 'pkg.initializer_v2.utils.utils_test.mock_env_vars' imported but unused in flake8.

Is it really unused ?
I thought, we use it here:

training-operator/test/integration/initializer_v2/dataset_test.py

Line 16 in 78fd239

self.temp_dir = setup_temp_path("DATASET_PATH")

It's a known thing for flake8: https://stackoverflow.com/questions/75647682/how-can-i-resolve-flake8-unused-import-error-for-pytest-fixture-imported-from.
If we would like to remove this, I can implement the suggestion of creating a conftest.py file to include all fixtures.

Oh, I see.
If you could move them to conftest.py as suggested by pytest that would be better, I think.
I guess, this file should live under test/integration/initializer_v2, right ?

Yes, I also created one under pkg/initialier_v2 to include the shared fixture used in the tests under this directory.

test/integration/initializer_v2/dataset_test.py

test/integration/initializer_v2/utils.py

.github/workflows/test-python.yaml

pkg/initializer_v2/dataset/dataset_huggingface_test.py

andreyvelich

Thank you for the updates @seanlaii!
Just a few small comments from me.
/assign @kubeflow/wg-training-leads @Electronic-Waste @astefanutti

pkg/initializer_v2/dataset/main_test.py

pkg/initializer_v2/model/main_test.py

andreyvelich · 2025-01-17T22:25:19Z

test/integration/initializer_v2/dataset_test.py

@@ -0,0 +1,75 @@
+import os
+import runpy
+from test.integration.initializer_v2.utils import setup_temp_path  # noqa: F401


Is it really unused ?
I thought, we use it here:

training-operator/test/integration/initializer_v2/dataset_test.py

Line 16 in 78fd239

self.temp_dir = setup_temp_path("DATASET_PATH")

.github/workflows/integration-tests.yaml

…lizers Signed-off-by: wei-chenglai <qazwsx0939059006@gmail.com>

cmd/initializer_v2/model/requirements.txt

Signed-off-by: wei-chenglai <qazwsx0939059006@gmail.com>

google-oss-prow bot requested review from jinchihe and kuizhiqing November 9, 2024 03:18

google-oss-prow bot added the size/XL label Nov 9, 2024

seanlaii commented Nov 9, 2024

View reviewed changes

pkg/initializer_v2/test/e2e/test_dataset.py Outdated Show resolved Hide resolved

seanlaii commented Nov 9, 2024

View reviewed changes

.github/workflows/test-python.yaml Outdated Show resolved Hide resolved

seanlaii commented Nov 9, 2024

View reviewed changes

pkg/initializer_v2/test/conftest.py Outdated Show resolved Hide resolved

seanlaii force-pushed the initializer-test branch from f4167e5 to f6345df Compare November 9, 2024 04:00

seanlaii force-pushed the initializer-test branch from f6345df to 1887c5b Compare November 9, 2024 04:11

seanlaii commented Nov 9, 2024

View reviewed changes

.github/workflows/integration-tests.yaml Outdated Show resolved Hide resolved

seanlaii commented Nov 9, 2024

View reviewed changes

pkg/initializer_v2/test/e2e/test_dataset.py Outdated Show resolved Hide resolved

seanlaii force-pushed the initializer-test branch 4 times, most recently from 8930b80 to c6e0a83 Compare November 9, 2024 18:17

andreyvelich reviewed Nov 26, 2024

View reviewed changes

pkg/initializer_v2/test/conftest.py Outdated Show resolved Hide resolved

andreyvelich reviewed Nov 26, 2024

View reviewed changes

andreyvelich reviewed Nov 27, 2024

View reviewed changes

pkg/initializer_v2/test/e2e/test_model.py Outdated Show resolved Hide resolved

pkg/initializer_v2/test/unit/dataset/test_dataset.py Outdated Show resolved Hide resolved

seanlaii force-pushed the initializer-test branch 6 times, most recently from 08fbd57 to d867237 Compare December 21, 2024 19:34

Electronic-Waste reviewed Jan 8, 2025

View reviewed changes

pkg/initializer_v2/dataset/huggingface_test.py Outdated Show resolved Hide resolved

pkg/initializer_v2/model/huggingface_test.py Outdated Show resolved Hide resolved

andreyvelich reviewed Jan 8, 2025

View reviewed changes

pkg/initializer_v2/dataset/huggingface_test.py Outdated Show resolved Hide resolved

pkg/initializer_v2/model/huggingface_test.py Outdated Show resolved Hide resolved

pkg/initializer_v2/model/main_test.py Outdated Show resolved Hide resolved

andreyvelich reviewed Jan 8, 2025

View reviewed changes

sdk_v2/kubeflow/training/types/config_test.py Outdated Show resolved Hide resolved

seanlaii force-pushed the initializer-test branch from d867237 to 5c24c97 Compare January 14, 2025 04:14

seanlaii changed the title ~~KEP-2170: Add unit and E2E tests for model and dataset initializers~~ KEP-2170: Add unit and Integration tests for model and dataset initializers Jan 14, 2025

seanlaii force-pushed the initializer-test branch from 5c24c97 to 3f12b9a Compare January 15, 2025 03:19

andreyvelich reviewed Jan 15, 2025

View reviewed changes

pkg/initializer_v2/model/main_test.py Outdated Show resolved Hide resolved

pkg/initializer_v2/dataset/main_test.py Outdated Show resolved Hide resolved

test/integration/initializer_v2/__init__.py Outdated Show resolved Hide resolved

andreyvelich reviewed Jan 15, 2025

View reviewed changes

pkg/initializer_v2/model/main_test.py Outdated Show resolved Hide resolved

test/integration/initializer_v2/dataset_test.py Outdated Show resolved Hide resolved

seanlaii force-pushed the initializer-test branch 2 times, most recently from 7f06cf1 to f9d9ef2 Compare January 16, 2025 07:26