Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature: Client support for competitions #231

Merged
merged 26 commits into from
Jan 10, 2025
Merged
Show file tree
Hide file tree
Changes from 17 commits
Commits
Show all changes
26 commits
Select commit Hold shift + click to select a range
1dc18cc
competition integration wip
Andrewq11 Nov 26, 2024
adf60d2
working competitions integration
Andrewq11 Dec 10, 2024
11fe166
removing .DS_Store file & tracking
Andrewq11 Dec 10, 2024
59f9ce2
remove testing file
Andrewq11 Dec 10, 2024
2dc0910
updating docstring wording
Andrewq11 Dec 10, 2024
e63ed74
small updates
Andrewq11 Dec 10, 2024
3c2feef
addressing PR feedback
Andrewq11 Dec 12, 2024
0f7e71b
integrating metric updates
Andrewq11 Dec 12, 2024
d21cb8e
basic testing for competitions
Andrewq11 Dec 12, 2024
99e8cf3
removing imports of old competiton classes from docs
Andrewq11 Dec 12, 2024
d22214f
removing doc ref to DatasetV2 class
Andrewq11 Dec 12, 2024
9d88721
Fix failing tests and circular import. Have the CompetitionSpecificat…
jstlaurent Dec 20, 2024
33d72c6
Move DatasetV2 out of experimental
jstlaurent Dec 20, 2024
0234e9b
Fix failing integration test
jstlaurent Dec 20, 2024
4ad0149
Refactor to extract a split and task mixin
cwognum Dec 20, 2024
8acc10d
Documentation pass
cwognum Dec 20, 2024
9a356fc
Minor changes to tutorial
cwognum Dec 20, 2024
eb8f844
removing comment from split validator since no longer checking for it
Andrewq11 Jan 6, 2025
228e602
updating split validator docstring to reflect order of checks in code
Andrewq11 Jan 6, 2025
a77864e
typo/docs/import fixes
Andrewq11 Jan 6, 2025
2ae5642
Merge branch 'main' into feat/competitions-v2
Andrewq11 Jan 6, 2025
68478e0
Minor changes from testing
cwognum Jan 7, 2025
d5c76b2
Pin zarr version for now
cwognum Jan 10, 2025
2ca98fb
Addressed PR feedbacl
cwognum Jan 10, 2025
7fa7009
Reverted PR feedback, fixed syntax error
cwognum Jan 10, 2025
ad72c0b
Ruff formatting
cwognum Jan 10, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -139,3 +139,6 @@ rever/
# Generated requirements.txt and uv lock file
requirements.txt
uv.lock

# OS-specific files
.DS_store
5 changes: 0 additions & 5 deletions docs/api/competition.dataset.md

This file was deleted.

6 changes: 0 additions & 6 deletions docs/api/competition.evaluation.md
Original file line number Diff line number Diff line change
@@ -1,7 +1 @@
::: polaris.evaluate.CompetitionPredictions

---

::: polaris.evaluate.CompetitionResults

---
8 changes: 7 additions & 1 deletion docs/api/dataset.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,12 @@

---

::: polaris.dataset.DatasetV2
options:
filters: ["!^_"]

---

::: polaris.dataset._base.BaseDataset
options:
filters: ["!^_"]
Expand All @@ -20,4 +26,4 @@
options:
filters: ["!^_"]

---
---
10 changes: 5 additions & 5 deletions docs/tutorials/competition.participate.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -98,7 +98,7 @@
"The Polaris library is designed to make it easy to participate in a competition. In just a few lines of code, we can get the train and test partition, access the associated data in various ways and evaluate our predictions. There's two main API endpoints. \n",
"\n",
"- `get_train_test_split()`: For creating objects through which we can access the different dataset partitions.\n",
"- `evaluate()`: For evaluating a set of predictions in accordance with the competition protocol."
"- `submit_predictions()`: For submitting the predictions to an active competition."
]
},
{
Expand All @@ -116,7 +116,7 @@
"id": "e78bf878",
"metadata": {},
"source": [
"The created test and train objects support various flavours to access the data."
"Similar to benchmarks, the created test and train objects support various flavours to access the data."
]
},
{
Expand Down Expand Up @@ -144,7 +144,7 @@
"id": "5ec12825",
"metadata": {},
"source": [
"Now, let's create some predictions against the official Polaris `hello-world-competition`. We will train a simple random forest model on the ECFP representation through scikit-learn and datamol, and then we will submit our results for secure evaluation by the Polaris Hub."
"Now, let's create some predictions against the imaginary `hello-world-competition`. Let's assume we rain a simple random forest model on the ECFP representation through scikit-learn and datamol, and then we will submit our results for secure evaluation by the Polaris Hub."
]
},
{
Expand Down Expand Up @@ -204,7 +204,7 @@
"id": "5ff06a9c",
"metadata": {},
"source": [
"Once your `CompetitionPredictions` object is created, you're ready to submit them for evaluation! This will automatically save your result to the Polaris Hub, but it will be private. You can choose to make it public through the Polaris web application. "
"Once your `CompetitionPredictions` object is created, you're ready to submit them for evaluation! This will automatically save your result to the Polaris Hub, but it will be private until the competition closes."
]
},
{
Expand All @@ -224,7 +224,7 @@
"id": "44973556",
"metadata": {},
"source": [
"That's it! Just like that you have partaken in your first Polaris competition. Keep an eye on that leaderboard and best of luck in your future competitions!\n",
"That's it! Just like that you have partaken in your first Polaris competition. Keep an eye on that leaderboard when it goes public and best of luck in your future competitions!\n",
"\n",
"The End.\n",
"\n",
Expand Down
11 changes: 3 additions & 8 deletions mkdocs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -24,21 +24,16 @@ nav:
- PDB Datasets: tutorials/dataset_pdb.ipynb
- SDF Datasets: tutorials/dataset_sdf.ipynb
- Optimization: tutorials/optimization.ipynb
# NOTE (cwognum): Competitions are currently gated.
# - Competitions:
# - tutorials/competition.participate.ipynb
- Competitions:
- tutorials/competition.participate.ipynb
- API Reference:
- Load: api/load.md
- Core:
- Dataset: api/dataset.md
- Benchmark: api/benchmark.md
- Competition: api/competition.md
- Subset: api/subset.md
- Evaluation: api/evaluation.md
# NOTE (cwognum): Competitions are currently gated.
# - Competitions:
# - Competition Dataset: api/competition.dataset.md
# - Competition: api/competition.md
# - Competiton Evaluation: api/competition.evaluation.md
- Hub:
- Client: api/hub.client.md
- External Auth Client: api/hub.external_client.md
Expand Down
4 changes: 2 additions & 2 deletions polaris/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -4,9 +4,9 @@
from loguru import logger

from ._version import __version__
from .loader import load_benchmark, load_dataset
from .loader import load_benchmark, load_dataset, load_competition

__all__ = ["load_dataset", "load_benchmark", "__version__"]
__all__ = ["load_dataset", "load_benchmark", "load_competition", "__version__"]

# Configure the default logging level
os.environ["LOGURU_LEVEL"] = os.environ.get("LOGURU_LEVEL", "INFO")
Expand Down
Loading
Loading