📑 Feature Request: Playground documentation and usage #1491
Comments
DhanshreeA
changed the title
|
Hi @GemmaTuron thanks for the comments. Your comments highly valuable for better functionality of this testing systems. for Q1) I created a PR that supports macOS but to only check the docker status, unfortunately we can not manipulate docker from python and subprocess in the macOS machine. @register_rule("folder_exists")
class FolderExistsRule(CommandRule):
def __init__(self):
pass
def check(self, folder_path, expected_status):
actual_status = Path(folder_path).exists() and any(Path(folder_path).iterdir())
if actual_status != expected_status:
raise AssertionError(
f"Expectation failed for FolderExistsRule: "
f"Expected folder to {'exist' if expected_status else 'not exist'}, "
f"but it {'exists' if actual_status else 'does not exist'}."
)
return {
"name": f"Folder exists at {folder_path}",
"status": actual_status,
}This rule will be executed after we fetch a command. Using this rule for instance I can check if the necessary model folder is existed in the required folder after fetch. My expected status is true, since in this case I want the folder exists in Other rule example: @register_rule("file_exists")
class FileExistsRule(CommandRule):
def __init__(self):
pass
def check(self, file_path, expected_status):
actual_status = Path(file_path).exists()
if actual_status != expected_status:
raise AssertionError(
f"Expectation failed for FileExistsRule: "
f"Expected file to {'exist' if expected_status else 'not exist'}, "
f"but it {'exists' if actual_status else 'does not exist'}."
)
return {
"name": f"File exists at {file_path}",
"status": actual_status,
}The above rule can executed after we execute the So on a quick glance above, you can see that we play on those ersilia commands using any rule we defined to check the healthiness of the model or the CLI. Q5) Now it supports h5. nox -f noxfile.py test_from_github model_id=eos2db3 python_version=3.8 delete_mode=false
nox -f noxfile.py test_serve_multiple_models model_ids=[eos2db3, eos3b5e] runner=multiple Q9) Now you can also pass python version from the command as above |
|
@GemmaTuron those comments of yours are very useful. I will update the pipelines for better use. |
|
Thanks @Abellegese we will discuss more in detail about all of this tomorrow. I think there is too much redundancy between the playground and the test module and it does not really make sense to duplicate all these efforts. Please do not modify the documentation at this point, I will take care of it. |
|
Hi @Abellegese I am noting this down in preparation for tomorrow's meeting. Please do not modify any more code or open more PRs before we can discuss everything. Q1: Where was the documentation that specified the playground only worked for Linux? We were considering adding the Playground as the Model Test Workflow, if it cannot be used in a MacOS platform it is not useful to that end |
|
And as I play with it some more questions: Q2: about the different commands. can you explain why we have separately a fetch and serve for multiple models whereas for the single models they are fetched, served and run inside the same command (or I understand they are) Q3: why / when is the autofetcher used? Q4: In the conventional run, why are all those values hardcoded? the output file name for example? Q5: minor point -- if Docker is not active, the error you get is the following. I don't know if we want a more informative message or because this is aimed at developers it is enough: |
|
Hi @GemmaTuron we will discuss them in detail in the meeting. |
|
@Abellegese I don't understand, where is the example command not producing more than one input? |
|
Hi @DhanshreeA I attached it below. Same problem from the python API. |
|
@Abellegese Right, I see. The |
|
Thanks @DhanshreeA |
|
More info on the usage of the playground. @Abellegese I see these weird conda environments created by Nox, how do I delete them? |
|
Update and please @Abellegese confirm if this is correct: There were legacy files inside a ./nox folder and an entire /ersilia copy inside the playground after running it. I need to delete them I understand? |
|
Hi @GemmaTuron for the first question, nox create an isolated venv thats why it creates them. Second question you specifiy overwrite_ersilia_repo that deletes the ersilia folder if it exists. Deleting the files inside .nox folder I dont think will change things. Inside it, there sessions with isolated venv there not ersilia. Ersilia cloned ersilia from github is saved inside the |
|
Hi @Abellegese Sorry I do not understand.
Please let me know how can I delete the |
|
@Abellegese I found that there is a hidden folder inside ersilia/test/playground/.nox that if it is deleted eliminates these environments. Is that right? |
|
Yes @GemmaTuron, those are the session isolated environment. Yes indeed there has to be some ways to clean them up after some sessions. Also note that when you run nox it removes them and creates a new one. In the PR I have created, I added some feature to reuse if the The github repo yes we need to move it maybe to |
|
My observations from the current playground implementation:
|
|
Thanks @DhanshreeA . |
|
Hey @DhanshreeA below are the rules I came up with for the commands 1. FetchFlags:
Checks:
Notes:
2. ServeFlags:
Checks:
3. RunFlags:
Checks:
4. CatalogFlags:
Checks:
5. ExampleFlags:
Additional Checks:
6. DeleteFlags:
Checks:
7. CloseFlags:
Checks:
8. TestFlags:
Checks
|
|
I will write here how to use the commands in detail: |
Playground CLI Usage GuidelinesInstallationTo use Playground CLI, install ersilia first using instruction given here. Then install package for testing purposes as given below in ersilia activated venv: pip install -e ".[test]"
The playground test folder found in nox -s clean -- --cli <command> [options]Or, from ersilia root directory, simply: nox -s clean -f test/playground/noxfile.py [path/to/noxfile.py] -- --cli <command> [options]Command mutual dependencyThe commands in Ersilia are interdependent, meaning that running a single command often requires executing a series of prerequisite commands. For example, to run a model, you must first execute
nox -s execute -- --cli runother required commands will be executed first, which in the above case Handling python virtual and files
<style> table { width: 100%; border-collapse: collapse; } .flag-column { width: 150px; } th, td { border: 1px solid #000; padding: 8px; text-align: left; } </style> Options & Flags🔹 Nox built-in flagsNox provides built-in commands that can be used to run tasks. These commands are:
🔹 General Settings
🔹 Command Flags
Command Selection (
|
| Flag | Description | Default | Example |
|---|---|---|---|
--cli |
Specifies ersilia commands to run in order (fetch, serve, run, catalog, example, test, close, delete). Default is all, which executes commands in this order: "fetch", "serve", "run", "close", "catalog", "example", "delete", "test". |
all |
nox -s execute -- --cli fetch serve run nox -s execute -- --cli run |
Ersilia Command flags
Note: every ersilia flags for the commands such as fetch:
--from_gituhb, delete:--all, etc, should bas passed without--. For examplenox -s execute -- --fetch from_dockerhub version [img-tag].
| Flag | Description | Default | Example |
|---|---|---|---|
--fetch |
Fetches models from sources (from_github, from_dockerhub, from_s3, version) |
--from_github |
nox -s execute -- --fetch from_dockerhub version dev nox -s execute -- --fetch from_s3 |
--run |
We don't specifically use this flag, instead we use --input_types and --outputs which are used to specifiy the input types such as str, list, csv and output file types result.csv, result.json, result.h5. Then the flag will be generate automatically in such format ["-i", "input", "-o", "output"] |
None |
None |
--example |
Generates example input for a model (-n, --random, -f). If we specify file name eg. example.csv, it will be saved in the path we specify to it, meaning we should pass the path to the file. |
["-n", 10, "--random"] |
nox -s execute -- --example -n 10 random/predefined -c -f example.csv |
--catalog |
Retrieves model catalog from local or hub |
["--more", "--local", "--as-json"] |
nox -s execute -- --catalog hub |
--test |
Tests models at different levels (shallow, deep, from_github, from_dockerhub, from_s3) |
["--shallow", "--from_github"] |
nox -s execute -- --test deep from_dockerhub/from_s3/from_github |
--delete |
Used to delete modes and has one flag all |
None |
nox -s execute -- --delete all |
Note that when you pass values for any of the flags given below, will overwrite the default values.
🔹 Other flags
| Flag | Description | Default | Example |
|---|---|---|---|
--outputs |
This is used with run command and used to specify output files (result.csv, result.h5). Note that we only specified the file name, the path will be automatically set to ~/eos/playground/files/{file_name.{csv, json, h5}} |
[results.{csv, json, h5}] |
nox -s execute -- --outputs result.csv result.h5 |
--input_types |
This is also used with run command to define input formats (str, list, csv). The same |
List of (str, list, csv) |
nox -s execute -- --input_types str list csv |
--runner |
Specifies execution mode (single, multiple). The single mode is used to execute commands using one model (for instance, the default model ID for this mode is eos3b5e). Whereas the multiple mode will use multiple models to execute the given commands. For this mode, by default, we run eos5axz, eos4e40, eos2r5a, eos4zfy, eos8fma. |
single |
nox -s execute -- --runner multiple |
--single |
Used to specify or override the default model ID used for single running mode. | eos3b5e |
nox -s execute -- --single eosxxxx |
--multiple |
Used to specify or override the default model IDs used for multiple running mode. | [eos5axz, eos4e40, eos2r5a, eos4zfy, eos8fma] |
nox -s execute -- --multiple eosxxxx eosxxxx eosxxxx |
Example Usage
All example given below assumes you are at dir
test/playground. This does not require to specify the noxfile. Nox by default usenoxfile.pywhich found intest/playground.
Run all commands with their default values
nox -s execute -p 3.11Fetch Models from DockerHub, Serve it and run it. With python 3.10
In this example the run input types and output files are default.
nox -s execute -p 3.10 -- --cli fetch serve run --fetch from_dockerhub1. Fetch Models from DockerHub, Serve it and run it just with by specifying one command "run"
In this example we specifies input types and output files.
nox -s execute -p 3.10 -- --cli run --fetch from_dockerhub --input_types str list --outputs result.csv result.h52. Other examples
nox -s execute -p 3.10 -- --cli serve run catalog exampleThis will clean all nox related resources (venv, files, logs...)
nox -s cleanTo test the closing command, if it successfull cleared out sessions created during serve.
nox-s execute -p 3.10 -- --cli closeTest a model in shallow mode. We use delete first because there is a fetching process during test and we want to cleared out previously existing models in the syste.
nox -s execute -p 3.10 -- --cli delete testRunning with multiple mode
nox -s execute -p 3.10 -- --cli fetch serve run --fetch from_dockerhub --runner multiple --outputs result.csvEnvironment Variables
| Variable | Description | Example Value |
|---|---|---|
TEST_ENV |
Used to pass yes_or_no prompt at test command |
TEST_ENV=true |
CONFIG_DATA |
used to pass config data to the pytest file | CONFIG_DATA=config_json_data |
|
Okay, a lot of work here thanks @Abellegese ! Help me clarify a bit so we can write solid documentation for end-users. I don't think there is need for additions to the playground at the moment just to better understand what goes on under the hood and maybe some small bugfixes. We can then decide if any final edits are required. Basic steps I am running:
Comments:
are the more technical ones explained somewhere? and is the
|
|
Q1) Yes as you know we have docker manipulation system so it requires priviliges |
|
Q5) On elif system_platform == "Darwin": # macOS
print("Stopping Docker programmatically is not supported on macOS.")
raise RuntimeError("Cannot stop Docker programmatically on macOS.")This is also true for starting docker but at least I implemented opening desktop version of docker if it is installed in user system. |
Thanks @Abellegese this is what I understood but I am working on Ubuntu, so shouldn't it be automatic? Maybe it is a user setting somewhere on Docker Desktop. At the moment it does not activate Docker Desktop if I pass the flag as true |
|
Q6) Exactly if docker is not activated, since fetch flag by default is
|
Yes if this is ubuntu it should be automatic. I am using ubuntu but I could not reproduce this in my computer |
|
Q7) |
that helps. I will only reference |
|
Ok @Abellegese Before I ask more questions summary of what we have discussed agreed so far: Q1) Yes as you know we have docker manipulation system so it requires priviliges - Ok this is added in the documentation now |
|
Yep thats it @GemmaTuron nicely summerized. |
|
And then a few questions to better understand:
I have updated the documentation based on the information you provided! Hope you like it, its here |
|
Q1) Nope its not equivalent
1) Check 'Catalog json content is valid': False and Details: 'Validation failed for key 'Input Shape' in object: {'Index': 2, 'Identifier': 'eos3mk2', 'Slug': 'bbbp-marine-kinase-inhibitors', 'Title': 'BBBP model tested on marine-derived kinase inhibitors', 'Task': ['Classification'], 'Input Shape': None, 'Output': ['Probability'], 'Output Shape': None, 'Model Source': 'Local Repository'}' |
|
Hi @Abellegese Thanks, but then for Q1 it is the same to run --cli fetch than to run directly --fetch, correct? osrry if that was not clear. |
|
Hi @GemmaTuron this one is important question that weather |
|
The first question is, its not |
ahh makes sense thanks! |
|
Okay @Abellegese
For the rest I think the documentation is basic but sufficient also because this feature is aimed at more advanced developers and they can play with the different command combinations as they see fit |
|
Thanks @GemmaTuron I will a change for the 2 point. |
|
The rest of the problem will be addressed in the test command fix. |
Describe your feature request.
Hi @Abellegese
I have tried to use the Playground for model testing but I find it extremely difficult to use with the documentation currently provided in GitBook. User-friendly steps should be detailed as well as instructions on how to interact with the Nox command, similar to the issues I found in the Model Tester documentation. Below a few questions to clarify before I can rewrite the docs as well as some bugs I am encountering:
How many tests are currently available? Is only these four?
- test_from_github
- test_from_dockerhub
- test_auto_fetcher_decider
- test_conventional_run
Inside each test, aside from how the model is fetched, the tests that are run are the same? I struggle to understand the difference between the playground and the test command except in the way the model is fetched. All the tests that we want to do in the model outputs (ie not nulls, not wildly different between runs) only happen in the test command? Then isn't the playground incomplete?
The playground is not testing H5 files, but I guess this is part of the refactoring being done?
The models are not deleted, only closed? Meaning they will remain in the users computers after being tested? I see the delete command first in the line of Command Execution Summary, I do not know if this is a bug and is actually run after close or not:
nox -f test/playground/noxfile.py -s test_from_githubif the config.yml file is not manually edited on the fetch_flags line, it will still pull the model from dockerhub according to what I see in the command execution summary:ersilia -v fetch eos5guo --from_dockerhub. It is not clear to me how this should happen, I believe it is a bug?Same for model eos3b5e --from_github, it does fail at fetch time, I do not know if due to the example command as mentioned in the model test issue or a different reason
The text was updated successfully, but these errors were encountered: