Skip to content

Commit

Permalink
Update introduction of manual templates (#3897)
Browse files Browse the repository at this point in the history
  • Loading branch information
YohannParis authored Jun 17, 2024
1 parent f9b7fac commit 9bda06d
Show file tree
Hide file tree
Showing 30 changed files with 71 additions and 268 deletions.
4 changes: 3 additions & 1 deletion .github/ISSUE_TEMPLATE/qa-issue.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,9 @@ assignees: ''

---

**Name of Scenario and step**
- Scenario:
- Step:
- Link to the project:

**Describe the issue**
A clear and concise description of what the bug is.
7 changes: 1 addition & 6 deletions testing/_template.md
Original file line number Diff line number Diff line change
@@ -1,8 +1,6 @@
## [Name of the Test Scenario]
Please go through __every__ step of the test scenario.\
When blocked, an error, or a UI/UX anomaly occurs, please report which scenario and step to [\#askem-testing](https://unchartedsoftware.slack.com/archives/C06FGLXB2CE).

Estimated time to completion: [X] minutes
Report any issues into GitHub: [open an issue](https://github.com/DARPA-ASKEM/terarium/issues/new?assignees=&labels=bug%2C+Q%26A&template=qa-issue.md&title=%5BBUG%5D%3A+).

### 1. Begin test
1. Login to https://app.staging.terarium.ai using the test account
Expand Down Expand Up @@ -32,6 +30,3 @@ Estimated time to completion: [X] minutes
- Instead, provide the `SIR.json` file of the model pre-configured
- Link the file to its path in the `testing/data` folder.
- Please re-use existing files if possible.
### 4. End test
1. logout of the application
7 changes: 1 addition & 6 deletions testing/manual/add-dataset-to-project.md
Original file line number Diff line number Diff line change
@@ -1,8 +1,6 @@
## Add dataset to a project
Please go through __every__ step of the test scenario.\
When blocked, an error, or a UI/UX anomaly occurs, please report which scenario and step to [\#askem-testing](https://unchartedsoftware.slack.com/archives/C06FGLXB2CE).

Estimated time to completion: [X] minutes
Report any issues into GitHub: [open an issue](https://github.com/DARPA-ASKEM/terarium/issues/new?assignees=&labels=bug%2C+Q%26A&template=qa-issue.md&title=%5BBUG%5D%3A+).

### 1. Begin test
1. Login to https://app.staging.terarium.ai using the test account
Expand All @@ -16,6 +14,3 @@ Estimated time to completion: [X] minutes
1. Upload _the Dataset_ [EpiMatrix.csv](../data/EpiMatrix.csv) using `Upload Resources` button in the bottom left.
2. Check that _the Dataset_ appears in the left menu.
3. Open _the Dataset_ and check that it has been added correctly.
### 4. End test
1. logout of the application
7 changes: 1 addition & 6 deletions testing/manual/add-model-to-project.md
Original file line number Diff line number Diff line change
@@ -1,8 +1,6 @@
## Add model to a project
Please go through __every__ step of the test scenario.\
When blocked, an error, or a UI/UX anomaly occurs, please report which scenario and step to [\#askem-testing](https://unchartedsoftware.slack.com/archives/C06FGLXB2CE).

Estimated time to completion: [X] minutes
Report any issues into GitHub: [open an issue](https://github.com/DARPA-ASKEM/terarium/issues/new?assignees=&labels=bug%2C+Q%26A&template=qa-issue.md&title=%5BBUG%5D%3A+).

### 1. Begin test
1. Login to https://app.staging.terarium.ai using the test account
Expand All @@ -16,6 +14,3 @@ Estimated time to completion: [X] minutes
1. Upload _the Model_ [Configured SIR.json](../data/Configured SIR.json) using `Upload Resources` button in the bottom left.
2. Check that _the Model_ appears in the left menu.
3. Open _the Model_ and check that it has been added correctly.
### 4. End test
1. logout of the application
14 changes: 5 additions & 9 deletions testing/manual/compare-two-models.md
Original file line number Diff line number Diff line change
@@ -1,8 +1,6 @@
## Compare Two Models
Please go through __every__ step of the test scenario.\
When blocked, an error, or a UI/UX anomaly occurs, please report which scenario and step to [\#askem-testing](https://unchartedsoftware.slack.com/archives/C06FGLXB2CE).

Estimated time to completion: [X] minutes
Report any issues into GitHub: [open an issue](https://github.com/DARPA-ASKEM/terarium/issues/new?assignees=&labels=bug%2C+Q%26A&template=qa-issue.md&title=%5BBUG%5D%3A+).

### 1. Begin test
1. Login to https://app.staging.terarium.ai using the test account
Expand All @@ -13,7 +11,7 @@ Estimated time to completion: [X] minutes
2. Create, or open, project named `QA [Your Name]`
### 2. Load some models to compare
1. Switch to the Explorer
1. Switch to the Explorer
2. Set the search mode to **Models**
3. Search for **covid masking**
4. __There is currently a bug where each model is duplicated and the first of each duplicate is broken. The second Bertozzi2020 model also returns an error message, but oddly it loads anyway__
Expand All @@ -31,15 +29,13 @@ Estimated time to completion: [X] minutes
### 4. Working with the Compare models drilldown: WIZARD
1. The first thing you'll see is the **Wizard** tab.
2. The first section `Overview` will probably say: Analyzing models metadata to generate a detailed comparison analysis....
2. The first section `Overview` will probably say: Analyzing models metadata to generate a detailed comparison analysis....
3. It should change to a full paragraph within ten seconds. If it doesn't there's a problem.
4. Scroll down the page and confirm there are two columns, one for each model. Do the diagrams display correctly? Is there content in each row?
4. Scroll down the page and confirm there are two columns, one for each model. Do the diagrams display correctly? Is there content in each row?
### 5. Working with the Compare models drilldown: NOTEBOOK
1. Switch to the **Notebook** tab.
2. Type "Compare models" into the AI assistant (or try some of the other suggestions in the dropdown) and press the **Send** icon (or enter)
3. It should generate some Python code.
3. It should generate some Python code.
4. Click **Run**
5. It should genearte a crazy looking hairball of a graph. Click on it to view it in a full screen image viewer, zoom in and out as required.
10 changes: 2 additions & 8 deletions testing/manual/dataset-enrichment.md
Original file line number Diff line number Diff line change
@@ -1,10 +1,8 @@
## Dataset enrichment with/without a document
Please go through __every__ step of the test scenario.\
When blocked, an error, or a UI/UX anomaly occurs, please report which scenario and step to [\#askem-testing](https://unchartedsoftware.slack.com/archives/C06FGLXB2CE).
Report any issues into GitHub: [open an issue](https://github.com/DARPA-ASKEM/terarium/issues/new?assignees=&labels=bug%2C+Q%26A&template=qa-issue.md&title=%5BBUG%5D%3A+).

Estimated time to completion: [X] minutes

### 1. Begin test
### 1. Begin test
1. Login to https://app.staging.terarium.ai using the test account
```
email: qa@test.io
Expand Down Expand Up @@ -42,7 +40,3 @@ Estimated time to completion: [X] minutes
| s | S | The proportion of the population that is susceptible to the infection but not yet infected at time t | susceptible to infection of | Proportion | numeric | 0.143079518218444 - 0.68 - 0.99995 |
| i | I | The proportion of the population that is currently infected and can spread the infection at time t | infection incidence proportion | Proportion | numeric | 0.00005 - 0.08 - 0.218551123932438 |
| r | R | The proportion of the population that has been removed from the spread of infection, including recovered, deceased, and isolated individuals at time t | recovered population | Proportion | numeric | 0 - 0.24 - 0.810127005400101 |
### 5. End test
1. logout of the application
7 changes: 1 addition & 6 deletions testing/manual/download-asset-from-project.md
Original file line number Diff line number Diff line change
@@ -1,8 +1,6 @@
## Download asset in a project
Please go through __every__ step of the test scenario.\
When blocked, an error, or a UI/UX anomaly occurs, please report which scenario and step to [\#askem-testing](https://unchartedsoftware.slack.com/archives/C06FGLXB2CE).

Estimated time to completion: [X] minutes
Report any issues into GitHub: [open an issue](https://github.com/DARPA-ASKEM/terarium/issues/new?assignees=&labels=bug%2C+Q%26A&template=qa-issue.md&title=%5BBUG%5D%3A+).

### 1. Begin test
1. Login to https://app.staging.terarium.ai using the test account
Expand All @@ -21,6 +19,3 @@ Estimated time to completion: [X] minutes
1. Click _the Document_ and select PDF.
2. In the far right `...` menu select `Download this File`
3. The file should download successfully.
### 4. End test
1. logout of the application
15 changes: 5 additions & 10 deletions testing/manual/ensemble-calibrate.md
Original file line number Diff line number Diff line change
@@ -1,9 +1,8 @@
## Ensemble Calibrate
model and dataset found here: https://drive.google.com/drive/u/0/folders/1sU3-CG9g9GBuTda_30lahiHAXu7U2-Xt
Please go through __every__ step of the test scenario.\
When blocked, an error, or a UI/UX anomaly occurs, please report which scenario and step to [\#askem-testing](https://unchartedsoftware.slack.com/archives/C06FGLXB2CE).

Estimated time to completion: [X] minutes
Please go through __every__ step of the test scenario.\
Report any issues into GitHub: [open an issue](https://github.com/DARPA-ASKEM/terarium/issues/new?assignees=&labels=bug%2C+Q%26A&template=qa-issue.md&title=%5BBUG%5D%3A+).

### 1. Begin test
1. Login to https://app.staging.terarium.ai using the test account
Expand All @@ -17,7 +16,7 @@ Estimated time to completion: [X] minutes
1. Create a new workflow named `Ensemble Calibrate`
2. Upload `Ensemble Calibrate Test Dataset` and `Giordano2020 - SIDARTHE model of COVID-19 spread in Italy` into your project
3. Drag and drop the model into your workflow
4. Link both models to individual _Configure Model_ operators
4. Link both models to individual _Configure Model_ operators
6. Save each configuration by giving it a name and clicking `Run`
7. drag and drop the dataset into the workflow.
Expand All @@ -29,17 +28,13 @@ Estimated time to completion: [X] minutes
Find the `Susceptible_state` in the ensemble variables dropdown, select it and hit add mapping.
Link the `Susceptible` for both of your identical configurations
### 5. End test
1. logout of the application
### Misc:
You should be able to add mapping as well as delete mapping. close and open the drilldown and see saved changes.
You should be able to set weights to whatever you like as well as hit the `Set weights to be equal` to reset them.
You should be able to utilize the `Stop` button to stop the run. In order to do this is may help to increase the
You should be able to utilize the `Stop` button to stop the run. In order to do this is may help to increase the
defaults to make a longer lasting run so you dont need to rely on reaction time
Once complete:
You should be able to add and delete charts
You should see changes to the select on the charts duplicated in the node as well as the drilldown views
You should be able to save the dataset
You should be able to save the dataset
17 changes: 6 additions & 11 deletions testing/manual/ensemble-simulate.md
Original file line number Diff line number Diff line change
@@ -1,9 +1,8 @@
## Ensemble Simulate
Models found here: https://drive.google.com/drive/u/0/folders/1sU3-CG9g9GBuTda_30lahiHAXu7U2-Xt
Please go through __every__ step of the test scenario.\
When blocked, an error, or a UI/UX anomaly occurs, please report which scenario and step to [\#askem-testing](https://unchartedsoftware.slack.com/archives/C06FGLXB2CE).

Estimated time to completion: [X] minutes
Please go through __every__ step of the test scenario.\
Report any issues into GitHub: [open an issue](https://github.com/DARPA-ASKEM/terarium/issues/new?assignees=&labels=bug%2C+Q%26A&template=qa-issue.md&title=%5BBUG%5D%3A+).

### 1. Begin test
1. Login to https://app.staging.terarium.ai using the test account
Expand All @@ -17,13 +16,13 @@ Estimated time to completion: [X] minutes
1. Create a new workflow named `Ensemble Simulate`
2. Upload `Giordano2020 - SIDARTHE model of COVID-19 spread in Italy` and `Test SIR` into your project
3. Drag and drop both models into your workflow
4. Link both models to individual _Configure Model_ operators
4. Link both models to individual _Configure Model_ operators
6. Save each configuration by giving it a name and clicking `Run`
### 3. Ensemble Simulate Operator setup
1. Link both configurations to a new `Simulate ensemble` node
2. Open the `Simulate ensemble` node by clicking on edit
3. Add the following mappings titling the ensemble variable
3. Add the following mappings titling the ensemble variable
EnsembleVariableOne `SIR's` S with `Giordano2020`'s Susceptible
EnsembleVariableTwo `SIR's` I with `Giordano2020`'s Cases
EnsembleVariableThree `SIR's` R with `Giordano2020`'s Healed
Expand All @@ -37,17 +36,13 @@ you should see the following options in your chart's dropdown:
EnsembleVariableThree
(or whatever arbitrary name you gave your mappings)
### 5. End test
1. logout of the application
### Misc:
You should be able to add mapping as well as delete mapping. close and open the drilldown and see saved changes.
You should be able to set weights to whatever you like as well as hit the `Set weights to be equal` to reset them.
You should be able to utilize the `Stop` button to stop the run. In order to do this is may help to increase the
You should be able to utilize the `Stop` button to stop the run. In order to do this is may help to increase the
defaults to make a longer lasting run so you dont need to rely on reaction time
Once complete:
You should be able to add and delete charts
You should see changes to the select on the charts duplicated in the node as well as the drilldown views
You should be able to save the dataset
You should be able to save the dataset
7 changes: 1 addition & 6 deletions testing/manual/model-configuration-extracting-variables.md
Original file line number Diff line number Diff line change
@@ -1,8 +1,6 @@
## Extract configuations from document and dataset
Please go through __every__ step of the test scenario.\
When blocked, an error, or a UI/UX anomaly occurs, please report which scenario and step to [\#askem-testing](https://unchartedsoftware.slack.com/archives/C06FGLXB2CE).

Estimated time to completion: [X] minutes
Report any issues into GitHub: [open an issue](https://github.com/DARPA-ASKEM/terarium/issues/new?assignees=&labels=bug%2C+Q%26A&template=qa-issue.md&title=%5BBUG%5D%3A+).

### 1. Begin test
1. Login to https://app.staging.terarium.ai using the test account
Expand All @@ -25,6 +23,3 @@ Estimated time to completion: [X] minutes
### 3. Apply configuration values
1. Under the suggested configurations table click the "apply configurations values" button
2. Ensure that the configuration has been populated with the selected configuration's values
### 4. End test
1. logout of the application
9 changes: 2 additions & 7 deletions testing/manual/model-configuration-input.md
Original file line number Diff line number Diff line change
@@ -1,8 +1,6 @@
## Configure a model configuration
Please go through __every__ step of the test scenario.\
When blocked, an error, or a UI/UX anomaly occurs, please report which scenario and step to [\#askem-testing](https://unchartedsoftware.slack.com/archives/C06FGLXB2CE).

Estimated time to completion: [X] minutes
Report any issues into GitHub: [open an issue](https://github.com/DARPA-ASKEM/terarium/issues/new?assignees=&labels=bug%2C+Q%26A&template=qa-issue.md&title=%5BBUG%5D%3A+).

### 1. Begin test
1. Login to https://app.staging.terarium.ai using the test account
Expand All @@ -23,7 +21,7 @@ Estimated time to completion: [X] minutes
## 2a. Add an uncertainty
1. In the parameters table click the "Add Uncertainty" button table to add uncertainties to any constant parameter
2. Click the check to apply the uncertainty and make sure that all selected constant values have a uniform distribution applied with the correct uncertainty (i.e an uncertainty of 10% on a constant of 5 will yield a uniform distribution of min: 4.5 max 5.5).
2. Click the check to apply the uncertainty and make sure that all selected constant values have a uniform distribution applied with the correct uncertainty (i.e an uncertainty of 10% on a constant of 5 will yield a uniform distribution of min: 4.5 max 5.5).
### 3. Configure the model again
Expand All @@ -34,6 +32,3 @@ Estimated time to completion: [X] minutes
### 4. Configure a stratified model
1. Repeat #2, #2a, and #3 for a stratified model
### 5. End test
1. logout of the application
9 changes: 2 additions & 7 deletions testing/manual/model-configuration-notebook.md
Original file line number Diff line number Diff line change
@@ -1,8 +1,6 @@
## Model Configuration Notebook
Please go through __every__ step of the test scenario.\
When blocked, an error, or a UI/UX anomaly occurs, please report which scenario and step to [\#askem-testing](https://unchartedsoftware.slack.com/archives/C06FGLXB2CE).

Estimated time to completion: [X] minutes
Report any issues into GitHub: [open an issue](https://github.com/DARPA-ASKEM/terarium/issues/new?assignees=&labels=bug%2C+Q%26A&template=qa-issue.md&title=%5BBUG%5D%3A+).

### 1. Begin test
1. Login to https://app.staging.terarium.ai using the test account
Expand All @@ -24,12 +22,9 @@ Estimated time to completion: [X] minutes
3. At the top, use the AI assistant to ask a question: `What are the model parameters?`
4. Ensure that code is generated (a `get_params`) function.
5. Execute the code and validate the output prints the various parameters.
6. Ask the AI: `Update R0 to 0.2`
6. Ask the AI: `Update R0 to 0.2`
7. Run the code
### 4. Validation
1. Go back to the wizard view
2. Verify that the value of `R0` is 0.2
### 5. End test
1. logout of the application
11 changes: 3 additions & 8 deletions testing/manual/model-edit.md
Original file line number Diff line number Diff line change
@@ -1,8 +1,6 @@
## Model edit
Please go through __every__ step of the test scenario.\
When blocked, an error, or a UI/UX anomaly occurs, please report which scenario and step to [\#askem-testing](https://unchartedsoftware.slack.com/archives/C06FGLXB2CE).

Estimated time to completion: [X] minutes
Report any issues into GitHub: [open an issue](https://github.com/DARPA-ASKEM/terarium/issues/new?assignees=&labels=bug%2C+Q%26A&template=qa-issue.md&title=%5BBUG%5D%3A+).

### 1. Begin test
1. Login to https://app.staging.terarium.ai using the test account
Expand Down Expand Up @@ -47,7 +45,7 @@ Estimated time to completion: [X] minutes
2. In the Notebook tab copy the code.
3. Select the original model output (first output).
4. Paste the code into the Notebook view and click the 'Run' button
5. Verify if your model edits from step 3 are reapplied.
5. Verify if your model edits from step 3 are reapplied.
### 7. Reset
1. Make any model edits (you can follow step 3a again)
Expand All @@ -61,10 +59,7 @@ Estimated time to completion: [X] minutes
4. Verify that the edit is applied correctly to the model and is reflected in the Wizard tab's decomposed view.
5. If it didn't work, notify the Beaker developers about the code that was generated. There is also a chance that the prompt that the AI came up with lacked accuracy and couldn't possibly be applied to the model.
### 9. Incompatible port connections
### 9. Incompatible port connections
The following port combos should not connect if...
- they are from the same template card
- they are already connected
### 10. End test
1. Close the operator and logout of the application.
9 changes: 2 additions & 7 deletions testing/manual/model-enrichment.md
Original file line number Diff line number Diff line change
@@ -1,10 +1,8 @@
## Model enrichment with/without a document
Please go through __every__ step of the test scenario.\
When blocked, an error, or a UI/UX anomaly occurs, please report which scenario and step to [\#askem-testing](https://unchartedsoftware.slack.com/archives/C06FGLXB2CE).
Report any issues into GitHub: [open an issue](https://github.com/DARPA-ASKEM/terarium/issues/new?assignees=&labels=bug%2C+Q%26A&template=qa-issue.md&title=%5BBUG%5D%3A+).

Estimated time to completion: [X] minutes

### 1. Begin test
### 1. Begin test
1. Login to https://app.staging.terarium.ai using the test account
```
email: qa@test.io
Expand All @@ -28,6 +26,3 @@ Estimated time to completion: [X] minutes
1. Re do steps 1-2 from the above test, **3**
2. Click `Extract variables` -> Select the uploaded document (`SIR.pdf`) -> Click `Use Document to extract variables`
4. __Expected Result__: The process go through without any error (TODO: add more details on expected results)
### 5. End test
1. logout of the application
9 changes: 2 additions & 7 deletions testing/manual/model-extraction.md
Original file line number Diff line number Diff line change
@@ -1,10 +1,8 @@
## Model extraction
Please go through __every__ step of the test scenario.\
When blocked, an error, or a UI/UX anomaly occurs, please report which scenario and step to [\#askem-testing](https://unchartedsoftware.slack.com/archives/C06FGLXB2CE).
Report any issues into GitHub: [open an issue](https://github.com/DARPA-ASKEM/terarium/issues/new?assignees=&labels=bug%2C+Q%26A&template=qa-issue.md&title=%5BBUG%5D%3A+).

Estimated time to completion: [X] minutes

### 1. Begin test
### 1. Begin test
1. Login to https://app.staging.terarium.ai using the test account
```
email: qa@test.io
Expand All @@ -30,6 +28,3 @@ Estimated time to completion: [X] minutes
2. Is the TA1 _model card_, extracted with `mit-tr` available?
3. Is the TA4 _model card_, extracted from `gollm-taskrunner` available?
4. Are the _variables extractions_ available in the `Other` accordion?
### 5. End test
1. logout of the application
Loading

0 comments on commit 9bda06d

Please sign in to comment.