diff --git a/.github/ISSUE_TEMPLATE/qa-issue.md b/.github/ISSUE_TEMPLATE/qa-issue.md index b6afc9406f..f068a2e7ec 100644 --- a/.github/ISSUE_TEMPLATE/qa-issue.md +++ b/.github/ISSUE_TEMPLATE/qa-issue.md @@ -7,7 +7,9 @@ assignees: '' --- -**Name of Scenario and step** +- Scenario: +- Step: +- Link to the project: **Describe the issue** A clear and concise description of what the bug is. diff --git a/testing/_template.md b/testing/_template.md index 60475a1c07..9056d0e40a 100644 --- a/testing/_template.md +++ b/testing/_template.md @@ -1,8 +1,6 @@ ## [Name of the Test Scenario] Please go through __every__ step of the test scenario.\ -When blocked, an error, or a UI/UX anomaly occurs, please report which scenario and step to [\#askem-testing](https://unchartedsoftware.slack.com/archives/C06FGLXB2CE). - -Estimated time to completion: [X] minutes +Report any issues into GitHub: [open an issue](https://github.com/DARPA-ASKEM/terarium/issues/new?assignees=&labels=bug%2C+Q%26A&template=qa-issue.md&title=%5BBUG%5D%3A+). ### 1. Begin test 1. Login to https://app.staging.terarium.ai using the test account @@ -32,6 +30,3 @@ Estimated time to completion: [X] minutes - Instead, provide the `SIR.json` file of the model pre-configured - Link the file to its path in the `testing/data` folder. - Please re-use existing files if possible. - -### 4. End test -1. logout of the application diff --git a/testing/manual/add-dataset-to-project.md b/testing/manual/add-dataset-to-project.md index ba68d11b62..df8b001552 100644 --- a/testing/manual/add-dataset-to-project.md +++ b/testing/manual/add-dataset-to-project.md @@ -1,8 +1,6 @@ ## Add dataset to a project Please go through __every__ step of the test scenario.\ -When blocked, an error, or a UI/UX anomaly occurs, please report which scenario and step to [\#askem-testing](https://unchartedsoftware.slack.com/archives/C06FGLXB2CE). - -Estimated time to completion: [X] minutes +Report any issues into GitHub: [open an issue](https://github.com/DARPA-ASKEM/terarium/issues/new?assignees=&labels=bug%2C+Q%26A&template=qa-issue.md&title=%5BBUG%5D%3A+). ### 1. Begin test 1. Login to https://app.staging.terarium.ai using the test account @@ -16,6 +14,3 @@ Estimated time to completion: [X] minutes 1. Upload _the Dataset_ [EpiMatrix.csv](../data/EpiMatrix.csv) using `Upload Resources` button in the bottom left. 2. Check that _the Dataset_ appears in the left menu. 3. Open _the Dataset_ and check that it has been added correctly. - -### 4. End test -1. logout of the application diff --git a/testing/manual/add-model-to-project.md b/testing/manual/add-model-to-project.md index db8a97d085..536ec4025a 100644 --- a/testing/manual/add-model-to-project.md +++ b/testing/manual/add-model-to-project.md @@ -1,8 +1,6 @@ ## Add model to a project Please go through __every__ step of the test scenario.\ -When blocked, an error, or a UI/UX anomaly occurs, please report which scenario and step to [\#askem-testing](https://unchartedsoftware.slack.com/archives/C06FGLXB2CE). - -Estimated time to completion: [X] minutes +Report any issues into GitHub: [open an issue](https://github.com/DARPA-ASKEM/terarium/issues/new?assignees=&labels=bug%2C+Q%26A&template=qa-issue.md&title=%5BBUG%5D%3A+). ### 1. Begin test 1. Login to https://app.staging.terarium.ai using the test account @@ -16,6 +14,3 @@ Estimated time to completion: [X] minutes 1. Upload _the Model_ [Configured SIR.json](../data/Configured SIR.json) using `Upload Resources` button in the bottom left. 2. Check that _the Model_ appears in the left menu. 3. Open _the Model_ and check that it has been added correctly. - -### 4. End test -1. logout of the application diff --git a/testing/manual/compare-two-models.md b/testing/manual/compare-two-models.md index 042dbabf4d..e1432654da 100644 --- a/testing/manual/compare-two-models.md +++ b/testing/manual/compare-two-models.md @@ -1,8 +1,6 @@ ## Compare Two Models Please go through __every__ step of the test scenario.\ -When blocked, an error, or a UI/UX anomaly occurs, please report which scenario and step to [\#askem-testing](https://unchartedsoftware.slack.com/archives/C06FGLXB2CE). - -Estimated time to completion: [X] minutes +Report any issues into GitHub: [open an issue](https://github.com/DARPA-ASKEM/terarium/issues/new?assignees=&labels=bug%2C+Q%26A&template=qa-issue.md&title=%5BBUG%5D%3A+). ### 1. Begin test 1. Login to https://app.staging.terarium.ai using the test account @@ -13,7 +11,7 @@ Estimated time to completion: [X] minutes 2. Create, or open, project named `QA [Your Name]` ### 2. Load some models to compare -1. Switch to the Explorer +1. Switch to the Explorer 2. Set the search mode to **Models** 3. Search for **covid masking** 4. __There is currently a bug where each model is duplicated and the first of each duplicate is broken. The second Bertozzi2020 model also returns an error message, but oddly it loads anyway__ @@ -31,15 +29,13 @@ Estimated time to completion: [X] minutes ### 4. Working with the Compare models drilldown: WIZARD 1. The first thing you'll see is the **Wizard** tab. -2. The first section `Overview` will probably say: Analyzing models metadata to generate a detailed comparison analysis.... +2. The first section `Overview` will probably say: Analyzing models metadata to generate a detailed comparison analysis.... 3. It should change to a full paragraph within ten seconds. If it doesn't there's a problem. -4. Scroll down the page and confirm there are two columns, one for each model. Do the diagrams display correctly? Is there content in each row? +4. Scroll down the page and confirm there are two columns, one for each model. Do the diagrams display correctly? Is there content in each row? ### 5. Working with the Compare models drilldown: NOTEBOOK 1. Switch to the **Notebook** tab. 2. Type "Compare models" into the AI assistant (or try some of the other suggestions in the dropdown) and press the **Send** icon (or enter) -3. It should generate some Python code. +3. It should generate some Python code. 4. Click **Run** 5. It should genearte a crazy looking hairball of a graph. Click on it to view it in a full screen image viewer, zoom in and out as required. - - diff --git a/testing/manual/dataset-enrichment.md b/testing/manual/dataset-enrichment.md index 3c06901d0b..b065ea0a71 100644 --- a/testing/manual/dataset-enrichment.md +++ b/testing/manual/dataset-enrichment.md @@ -1,10 +1,8 @@ ## Dataset enrichment with/without a document Please go through __every__ step of the test scenario.\ -When blocked, an error, or a UI/UX anomaly occurs, please report which scenario and step to [\#askem-testing](https://unchartedsoftware.slack.com/archives/C06FGLXB2CE). +Report any issues into GitHub: [open an issue](https://github.com/DARPA-ASKEM/terarium/issues/new?assignees=&labels=bug%2C+Q%26A&template=qa-issue.md&title=%5BBUG%5D%3A+). -Estimated time to completion: [X] minutes - -### 1. Begin test +### 1. Begin test 1. Login to https://app.staging.terarium.ai using the test account ``` email: qa@test.io @@ -42,7 +40,3 @@ Estimated time to completion: [X] minutes | s | S | The proportion of the population that is susceptible to the infection but not yet infected at time t | susceptible to infection of | Proportion | numeric | 0.143079518218444 - 0.68 - 0.99995 | | i | I | The proportion of the population that is currently infected and can spread the infection at time t | infection incidence proportion | Proportion | numeric | 0.00005 - 0.08 - 0.218551123932438 | | r | R | The proportion of the population that has been removed from the spread of infection, including recovered, deceased, and isolated individuals at time t | recovered population | Proportion | numeric | 0 - 0.24 - 0.810127005400101 | - - -### 5. End test -1. logout of the application diff --git a/testing/manual/download-asset-from-project.md b/testing/manual/download-asset-from-project.md index bf1563605a..42dfd2ee7d 100644 --- a/testing/manual/download-asset-from-project.md +++ b/testing/manual/download-asset-from-project.md @@ -1,8 +1,6 @@ ## Download asset in a project Please go through __every__ step of the test scenario.\ -When blocked, an error, or a UI/UX anomaly occurs, please report which scenario and step to [\#askem-testing](https://unchartedsoftware.slack.com/archives/C06FGLXB2CE). - -Estimated time to completion: [X] minutes +Report any issues into GitHub: [open an issue](https://github.com/DARPA-ASKEM/terarium/issues/new?assignees=&labels=bug%2C+Q%26A&template=qa-issue.md&title=%5BBUG%5D%3A+). ### 1. Begin test 1. Login to https://app.staging.terarium.ai using the test account @@ -21,6 +19,3 @@ Estimated time to completion: [X] minutes 1. Click _the Document_ and select PDF. 2. In the far right `...` menu select `Download this File` 3. The file should download successfully. - -### 4. End test -1. logout of the application diff --git a/testing/manual/ensemble-calibrate.md b/testing/manual/ensemble-calibrate.md index 02f90b6d61..4a0c5868d6 100644 --- a/testing/manual/ensemble-calibrate.md +++ b/testing/manual/ensemble-calibrate.md @@ -1,9 +1,8 @@ ## Ensemble Calibrate model and dataset found here: https://drive.google.com/drive/u/0/folders/1sU3-CG9g9GBuTda_30lahiHAXu7U2-Xt -Please go through __every__ step of the test scenario.\ -When blocked, an error, or a UI/UX anomaly occurs, please report which scenario and step to [\#askem-testing](https://unchartedsoftware.slack.com/archives/C06FGLXB2CE). -Estimated time to completion: [X] minutes +Please go through __every__ step of the test scenario.\ +Report any issues into GitHub: [open an issue](https://github.com/DARPA-ASKEM/terarium/issues/new?assignees=&labels=bug%2C+Q%26A&template=qa-issue.md&title=%5BBUG%5D%3A+). ### 1. Begin test 1. Login to https://app.staging.terarium.ai using the test account @@ -17,7 +16,7 @@ Estimated time to completion: [X] minutes 1. Create a new workflow named `Ensemble Calibrate` 2. Upload `Ensemble Calibrate Test Dataset` and `Giordano2020 - SIDARTHE model of COVID-19 spread in Italy` into your project 3. Drag and drop the model into your workflow -4. Link both models to individual _Configure Model_ operators +4. Link both models to individual _Configure Model_ operators 6. Save each configuration by giving it a name and clicking `Run` 7. drag and drop the dataset into the workflow. @@ -29,17 +28,13 @@ Estimated time to completion: [X] minutes Find the `Susceptible_state` in the ensemble variables dropdown, select it and hit add mapping. Link the `Susceptible` for both of your identical configurations -### 5. End test -1. logout of the application - ### Misc: You should be able to add mapping as well as delete mapping. close and open the drilldown and see saved changes. You should be able to set weights to whatever you like as well as hit the `Set weights to be equal` to reset them. -You should be able to utilize the `Stop` button to stop the run. In order to do this is may help to increase the +You should be able to utilize the `Stop` button to stop the run. In order to do this is may help to increase the defaults to make a longer lasting run so you dont need to rely on reaction time Once complete: You should be able to add and delete charts You should see changes to the select on the charts duplicated in the node as well as the drilldown views -You should be able to save the dataset - +You should be able to save the dataset diff --git a/testing/manual/ensemble-simulate.md b/testing/manual/ensemble-simulate.md index f5d6e93854..17ec4f3a46 100644 --- a/testing/manual/ensemble-simulate.md +++ b/testing/manual/ensemble-simulate.md @@ -1,9 +1,8 @@ ## Ensemble Simulate Models found here: https://drive.google.com/drive/u/0/folders/1sU3-CG9g9GBuTda_30lahiHAXu7U2-Xt -Please go through __every__ step of the test scenario.\ -When blocked, an error, or a UI/UX anomaly occurs, please report which scenario and step to [\#askem-testing](https://unchartedsoftware.slack.com/archives/C06FGLXB2CE). -Estimated time to completion: [X] minutes +Please go through __every__ step of the test scenario.\ +Report any issues into GitHub: [open an issue](https://github.com/DARPA-ASKEM/terarium/issues/new?assignees=&labels=bug%2C+Q%26A&template=qa-issue.md&title=%5BBUG%5D%3A+). ### 1. Begin test 1. Login to https://app.staging.terarium.ai using the test account @@ -17,13 +16,13 @@ Estimated time to completion: [X] minutes 1. Create a new workflow named `Ensemble Simulate` 2. Upload `Giordano2020 - SIDARTHE model of COVID-19 spread in Italy` and `Test SIR` into your project 3. Drag and drop both models into your workflow -4. Link both models to individual _Configure Model_ operators +4. Link both models to individual _Configure Model_ operators 6. Save each configuration by giving it a name and clicking `Run` ### 3. Ensemble Simulate Operator setup 1. Link both configurations to a new `Simulate ensemble` node 2. Open the `Simulate ensemble` node by clicking on edit -3. Add the following mappings titling the ensemble variable +3. Add the following mappings titling the ensemble variable EnsembleVariableOne `SIR's` S with `Giordano2020`'s Susceptible EnsembleVariableTwo `SIR's` I with `Giordano2020`'s Cases EnsembleVariableThree `SIR's` R with `Giordano2020`'s Healed @@ -37,17 +36,13 @@ you should see the following options in your chart's dropdown: EnsembleVariableThree (or whatever arbitrary name you gave your mappings) - -### 5. End test -1. logout of the application - ### Misc: You should be able to add mapping as well as delete mapping. close and open the drilldown and see saved changes. You should be able to set weights to whatever you like as well as hit the `Set weights to be equal` to reset them. -You should be able to utilize the `Stop` button to stop the run. In order to do this is may help to increase the +You should be able to utilize the `Stop` button to stop the run. In order to do this is may help to increase the defaults to make a longer lasting run so you dont need to rely on reaction time Once complete: You should be able to add and delete charts You should see changes to the select on the charts duplicated in the node as well as the drilldown views -You should be able to save the dataset +You should be able to save the dataset diff --git a/testing/manual/model-configuration-extracting-variables.md b/testing/manual/model-configuration-extracting-variables.md index 8fd9984e23..abc4a66fdf 100644 --- a/testing/manual/model-configuration-extracting-variables.md +++ b/testing/manual/model-configuration-extracting-variables.md @@ -1,8 +1,6 @@ ## Extract configuations from document and dataset Please go through __every__ step of the test scenario.\ -When blocked, an error, or a UI/UX anomaly occurs, please report which scenario and step to [\#askem-testing](https://unchartedsoftware.slack.com/archives/C06FGLXB2CE). - -Estimated time to completion: [X] minutes +Report any issues into GitHub: [open an issue](https://github.com/DARPA-ASKEM/terarium/issues/new?assignees=&labels=bug%2C+Q%26A&template=qa-issue.md&title=%5BBUG%5D%3A+). ### 1. Begin test 1. Login to https://app.staging.terarium.ai using the test account @@ -25,6 +23,3 @@ Estimated time to completion: [X] minutes ### 3. Apply configuration values 1. Under the suggested configurations table click the "apply configurations values" button 2. Ensure that the configuration has been populated with the selected configuration's values - -### 4. End test -1. logout of the application diff --git a/testing/manual/model-configuration-input.md b/testing/manual/model-configuration-input.md index aa441496dd..fd3fcff998 100644 --- a/testing/manual/model-configuration-input.md +++ b/testing/manual/model-configuration-input.md @@ -1,8 +1,6 @@ ## Configure a model configuration Please go through __every__ step of the test scenario.\ -When blocked, an error, or a UI/UX anomaly occurs, please report which scenario and step to [\#askem-testing](https://unchartedsoftware.slack.com/archives/C06FGLXB2CE). - -Estimated time to completion: [X] minutes +Report any issues into GitHub: [open an issue](https://github.com/DARPA-ASKEM/terarium/issues/new?assignees=&labels=bug%2C+Q%26A&template=qa-issue.md&title=%5BBUG%5D%3A+). ### 1. Begin test 1. Login to https://app.staging.terarium.ai using the test account @@ -23,7 +21,7 @@ Estimated time to completion: [X] minutes ## 2a. Add an uncertainty 1. In the parameters table click the "Add Uncertainty" button table to add uncertainties to any constant parameter -2. Click the check to apply the uncertainty and make sure that all selected constant values have a uniform distribution applied with the correct uncertainty (i.e an uncertainty of 10% on a constant of 5 will yield a uniform distribution of min: 4.5 max 5.5). +2. Click the check to apply the uncertainty and make sure that all selected constant values have a uniform distribution applied with the correct uncertainty (i.e an uncertainty of 10% on a constant of 5 will yield a uniform distribution of min: 4.5 max 5.5). ### 3. Configure the model again @@ -34,6 +32,3 @@ Estimated time to completion: [X] minutes ### 4. Configure a stratified model 1. Repeat #2, #2a, and #3 for a stratified model - -### 5. End test -1. logout of the application diff --git a/testing/manual/model-configuration-notebook.md b/testing/manual/model-configuration-notebook.md index f5ada1e125..08d58ea345 100644 --- a/testing/manual/model-configuration-notebook.md +++ b/testing/manual/model-configuration-notebook.md @@ -1,8 +1,6 @@ ## Model Configuration Notebook Please go through __every__ step of the test scenario.\ -When blocked, an error, or a UI/UX anomaly occurs, please report which scenario and step to [\#askem-testing](https://unchartedsoftware.slack.com/archives/C06FGLXB2CE). - -Estimated time to completion: [X] minutes +Report any issues into GitHub: [open an issue](https://github.com/DARPA-ASKEM/terarium/issues/new?assignees=&labels=bug%2C+Q%26A&template=qa-issue.md&title=%5BBUG%5D%3A+). ### 1. Begin test 1. Login to https://app.staging.terarium.ai using the test account @@ -24,12 +22,9 @@ Estimated time to completion: [X] minutes 3. At the top, use the AI assistant to ask a question: `What are the model parameters?` 4. Ensure that code is generated (a `get_params`) function. 5. Execute the code and validate the output prints the various parameters. -6. Ask the AI: `Update R0 to 0.2` +6. Ask the AI: `Update R0 to 0.2` 7. Run the code ### 4. Validation 1. Go back to the wizard view 2. Verify that the value of `R0` is 0.2 - -### 5. End test -1. logout of the application diff --git a/testing/manual/model-edit.md b/testing/manual/model-edit.md index 711b4e50ad..6ef17ed3b4 100644 --- a/testing/manual/model-edit.md +++ b/testing/manual/model-edit.md @@ -1,8 +1,6 @@ ## Model edit Please go through __every__ step of the test scenario.\ -When blocked, an error, or a UI/UX anomaly occurs, please report which scenario and step to [\#askem-testing](https://unchartedsoftware.slack.com/archives/C06FGLXB2CE). - -Estimated time to completion: [X] minutes +Report any issues into GitHub: [open an issue](https://github.com/DARPA-ASKEM/terarium/issues/new?assignees=&labels=bug%2C+Q%26A&template=qa-issue.md&title=%5BBUG%5D%3A+). ### 1. Begin test 1. Login to https://app.staging.terarium.ai using the test account @@ -47,7 +45,7 @@ Estimated time to completion: [X] minutes 2. In the Notebook tab copy the code. 3. Select the original model output (first output). 4. Paste the code into the Notebook view and click the 'Run' button -5. Verify if your model edits from step 3 are reapplied. +5. Verify if your model edits from step 3 are reapplied. ### 7. Reset 1. Make any model edits (you can follow step 3a again) @@ -61,10 +59,7 @@ Estimated time to completion: [X] minutes 4. Verify that the edit is applied correctly to the model and is reflected in the Wizard tab's decomposed view. 5. If it didn't work, notify the Beaker developers about the code that was generated. There is also a chance that the prompt that the AI came up with lacked accuracy and couldn't possibly be applied to the model. -### 9. Incompatible port connections +### 9. Incompatible port connections The following port combos should not connect if... - they are from the same template card - they are already connected - -### 10. End test -1. Close the operator and logout of the application. diff --git a/testing/manual/model-enrichment.md b/testing/manual/model-enrichment.md index 1a9385c46e..3aa6857566 100644 --- a/testing/manual/model-enrichment.md +++ b/testing/manual/model-enrichment.md @@ -1,10 +1,8 @@ ## Model enrichment with/without a document Please go through __every__ step of the test scenario.\ -When blocked, an error, or a UI/UX anomaly occurs, please report which scenario and step to [\#askem-testing](https://unchartedsoftware.slack.com/archives/C06FGLXB2CE). +Report any issues into GitHub: [open an issue](https://github.com/DARPA-ASKEM/terarium/issues/new?assignees=&labels=bug%2C+Q%26A&template=qa-issue.md&title=%5BBUG%5D%3A+). -Estimated time to completion: [X] minutes - -### 1. Begin test +### 1. Begin test 1. Login to https://app.staging.terarium.ai using the test account ``` email: qa@test.io @@ -28,6 +26,3 @@ Estimated time to completion: [X] minutes 1. Re do steps 1-2 from the above test, **3** 2. Click `Extract variables` -> Select the uploaded document (`SIR.pdf`) -> Click `Use Document to extract variables` 4. __Expected Result__: The process go through without any error (TODO: add more details on expected results) - -### 5. End test -1. logout of the application diff --git a/testing/manual/model-extraction.md b/testing/manual/model-extraction.md index 743e3d6b6a..b5cb85f62c 100644 --- a/testing/manual/model-extraction.md +++ b/testing/manual/model-extraction.md @@ -1,10 +1,8 @@ ## Model extraction Please go through __every__ step of the test scenario.\ -When blocked, an error, or a UI/UX anomaly occurs, please report which scenario and step to [\#askem-testing](https://unchartedsoftware.slack.com/archives/C06FGLXB2CE). +Report any issues into GitHub: [open an issue](https://github.com/DARPA-ASKEM/terarium/issues/new?assignees=&labels=bug%2C+Q%26A&template=qa-issue.md&title=%5BBUG%5D%3A+). -Estimated time to completion: [X] minutes - -### 1. Begin test +### 1. Begin test 1. Login to https://app.staging.terarium.ai using the test account ``` email: qa@test.io @@ -30,6 +28,3 @@ Estimated time to completion: [X] minutes 2. Is the TA1 _model card_, extracted with `mit-tr` available? 3. Is the TA4 _model card_, extracted from `gollm-taskrunner` available? 4. Are the _variables extractions_ available in the `Other` accordion? - -### 5. End test -1. logout of the application diff --git a/testing/manual/model-from-code.md b/testing/manual/model-from-code.md index 4e87af31e7..8f7a97de1f 100644 --- a/testing/manual/model-from-code.md +++ b/testing/manual/model-from-code.md @@ -1,8 +1,6 @@ ## Create model from code Please go through __every__ step of the test scenario.\ -When blocked, an error, or a UI/UX anomaly occurs, please report which scenario and step to [\#askem-testing](https://unchartedsoftware.slack.com/archives/C06FGLXB2CE). - -Estimated time to completion: [X] minutes +Report any issues into GitHub: [open an issue](https://github.com/DARPA-ASKEM/terarium/issues/new?assignees=&labels=bug%2C+Q%26A&template=qa-issue.md&title=%5BBUG%5D%3A+). ### 1. Begin test 1. Login to https://app.staging.terarium.ai using the test account @@ -23,6 +21,3 @@ Estimated time to completion: [X] minutes 4. Add a _Create model from code_ node to the workflow. 5. Edit the node, select the code block and create the model. 6. Save the model. - -### 3. End test -1. logout of the application diff --git a/testing/manual/model-from-equations.md b/testing/manual/model-from-equations.md index 4d91a922f2..e78eff9628 100644 --- a/testing/manual/model-from-equations.md +++ b/testing/manual/model-from-equations.md @@ -1,8 +1,6 @@ ## Create model from equations Please go through __every__ step of the test scenario.\ -When blocked, an error, or a UI/UX anomaly occurs, please report which scenario and step to [\#askem-testing](https://unchartedsoftware.slack.com/archives/C06FGLXB2CE). - -Estimated time to completion: [X] minutes +Report any issues into GitHub: [open an issue](https://github.com/DARPA-ASKEM/terarium/issues/new?assignees=&labels=bug%2C+Q%26A&template=qa-issue.md&title=%5BBUG%5D%3A+). ### 1. Begin test 1. Login to https://app.staging.terarium.ai using the test account @@ -24,6 +22,3 @@ Estimated time to completion: [X] minutes \frac{dR}{dt} = \gamma I_v + \gamma I ``` 4. Save the model. - -### 3. End test -1. logout of the application diff --git a/testing/manual/operator-annotations.md b/testing/manual/operator-annotations.md index d874e28b01..94cb5178ef 100644 --- a/testing/manual/operator-annotations.md +++ b/testing/manual/operator-annotations.md @@ -1,10 +1,8 @@ -## `[Operator Annotations]` +## Operator Annotations Please go through __every__ step of the test scenario.\ -When blocked, an error, or a UI/UX anomaly occurs, please report which scenario and step to [\#askem-testing](https://unchartedsoftware.slack.com/archives/C06FGLXB2CE). +Report any issues into GitHub: [open an issue](https://github.com/DARPA-ASKEM/terarium/issues/new?assignees=&labels=bug%2C+Q%26A&template=qa-issue.md&title=%5BBUG%5D%3A+). -Estimated time to completion: [X] minutes - -### 1. Begin test +### 1. Begin test 1. Login to https://app.staging.terarium.ai using the test account ``` email: qa@test.io @@ -23,7 +21,7 @@ Estimated time to completion: [X] minutes 6. Tap ENTER when finished typing to accept changes. 7. Click the pencil icon button in the model node to re-enter edit mode. 8. Click the checkmark icon button to accept changes. -9. __Expected Result__: The annotation entered via the text field should be added to the operator successfully by clicking Enter and/or by clicking the checkmark icon button. +9. __Expected Result__: The annotation entered via the text field should be added to the operator successfully by clicking Enter and/or by clicking the checkmark icon button. `[Task 2: Editing an annotation in an operator's workflow node]` 1. Click the pencil icon button located at the end of the annotation text in the model node. @@ -55,6 +53,3 @@ Estimated time to completion: [X] minutes 1. Click the pencil icon button located at the end of the annotation text in the drill down header. 2. Click the trash can icon button. 3. __Expected Result__: The annotation should no longer be visible in the model operator drill down header. The "Add a note" button should now be visible and functional. - -### 3. End test -1. logout of the application diff --git a/testing/manual/project-search-sort.md b/testing/manual/project-search-sort.md index 68b29376b0..dc62429c3b 100644 --- a/testing/manual/project-search-sort.md +++ b/testing/manual/project-search-sort.md @@ -1,8 +1,6 @@ ## Search and Sort Please go through __every__ step of the test scenario.\ -When blocked, an error, or a UI/UX anomaly occurs, please report which scenario and step to [\#askem-testing](https://unchartedsoftware.slack.com/archives/C06FGLXB2CE). - -Estimated time to completion: [X] minutes +Report any issues into GitHub: [open an issue](https://github.com/DARPA-ASKEM/terarium/issues/new?assignees=&labels=bug%2C+Q%26A&template=qa-issue.md&title=%5BBUG%5D%3A+). ### 1. Begin test 1. Login to https://app.staging.terarium.ai using the test account @@ -59,8 +57,3 @@ Estimated time to completion: [X] minutes - the order should switch between `(2,3,1) & (1,3,2)` 7. Click the `Last updated` a few times keeping an on the 3 projects order - the order should switch between `(3,1,2) & (2,1,3)` - -### 6. End test -1. In the browser with the other account remove the share access to the `Quality Assurance` account -2. Open the project card menu and remove your 3 projects -3. logout of the application diff --git a/testing/manual/project-sharing.md b/testing/manual/project-sharing.md index e55256d757..50a7612a0b 100644 --- a/testing/manual/project-sharing.md +++ b/testing/manual/project-sharing.md @@ -1,8 +1,6 @@ ## Sharing a Project Please go through __every__ step of the test scenario.\ -When blocked, an error, or a UI/UX anomaly occurs, please report which scenario and step to [\#askem-testing](https://unchartedsoftware.slack.com/archives/C06FGLXB2CE). - -Estimated time to completion: [X] minutes +Report any issues into GitHub: [open an issue](https://github.com/DARPA-ASKEM/terarium/issues/new?assignees=&labels=bug%2C+Q%26A&template=qa-issue.md&title=%5BBUG%5D%3A+). ### 1. Begin test 1. Login to https://app.staging.terarium.ai using the test account @@ -35,6 +33,3 @@ Estimated time to completion: [X] minutes - Change the description to 'Editing project' 5. Open the Project. - Change the description to 'done editing' - -### 5. End test -1. Logout of the application for both accounts. diff --git a/testing/manual/pyciemss-calibrate.md b/testing/manual/pyciemss-calibrate.md index 2b4fbfe791..cc265d8ff9 100644 --- a/testing/manual/pyciemss-calibrate.md +++ b/testing/manual/pyciemss-calibrate.md @@ -1,10 +1,9 @@ ## [Name of the Test Scenario] Please go through __every__ step of the test scenario.\ -When blocked, an error, or a UI/UX anomaly occurs, please report which scenario and step to [\#askem-testing](https://unchartedsoftware.slack.com/archives/C06FGLXB2CE). +Report any issues into GitHub: [open an issue](https://github.com/DARPA-ASKEM/terarium/issues/new?assignees=&labels=bug%2C+Q%26A&template=qa-issue.md&title=%5BBUG%5D%3A+). -Estimated time to completion: [X] minutes - -Note: sampling combinations in PyCIEMSS can result in numerical instability, when this happens, you can: +### Note +sampling combinations in PyCIEMSS can result in numerical instability, when this happens, you can: - Retry the simulation, or - Fiddle with the parameter distribution ranges, make the intervals larger or smaller @@ -40,7 +39,3 @@ Note: sampling combinations in PyCIEMSS can result in numerical instability, whe 1. Verify that there is a loss chart available, showing the loss value as a function of solver iterations 2. Verify that there is a Variables chart available, showing time-series plots for each model variable based on a Simulate run with the calibrate parameter values. 3. Should be able to add/remove Variables charts - - -### 5. End test -1. Logout of the application diff --git a/testing/manual/pyciemss-simulate.md b/testing/manual/pyciemss-simulate.md index 2bbed0e6b9..e547ad86b1 100644 --- a/testing/manual/pyciemss-simulate.md +++ b/testing/manual/pyciemss-simulate.md @@ -1,14 +1,12 @@ ## PyCIEMSS Simulate Please go through __every__ step of the test scenario.\ -When blocked, an error, or a UI/UX anomaly occurs, please report which scenario and step to [\#askem-testing](https://unchartedsoftware.slack.com/archives/C06FGLXB2CE). - -Estimated time to completion: [X] minutes +Report any issues into GitHub: [open an issue](https://github.com/DARPA-ASKEM/terarium/issues/new?assignees=&labels=bug%2C+Q%26A&template=qa-issue.md&title=%5BBUG%5D%3A+). +### Note Note: sampling combinations in PyCIEMSS can result in numerical instability, when this happens, you can: - Retry the simulation, or - Fiddle with the parameter distribution ranges, make the intervals larger or smaller - ### 1. Begin test 1. Login to https://app.staging.terarium.ai using the test account ``` @@ -40,6 +38,3 @@ Note: sampling combinations in PyCIEMSS can result in numerical instability, whe 1. Verify that you can go back and forth between the two outputs created 2. Switch from the default solver to something else, rerun the simulation 3. Verify that there are now three outputs associated with the operator and you can switch between these outputs - -### 6. End test -1. logout of the application diff --git a/testing/manual/send-asset-to-another-project.md b/testing/manual/send-asset-to-another-project.md index 3d39cbbf8b..7927a79f7d 100644 --- a/testing/manual/send-asset-to-another-project.md +++ b/testing/manual/send-asset-to-another-project.md @@ -1,6 +1,6 @@ ## Send asset to another project Please go through __every__ step of the test scenario.\ -When blocked, an error, or a UI/UX anomaly occurs, please report which scenario and step to [\#askem-testing](https://unchartedsoftware.slack.com/archives/C06FGLXB2CE). +Report any issues into GitHub: [open an issue](https://github.com/DARPA-ASKEM/terarium/issues/new?assignees=&labels=bug%2C+Q%26A&template=qa-issue.md&title=%5BBUG%5D%3A+). ### 1. Begin test 1. Login to https://app.staging.terarium.ai using the test account @@ -21,6 +21,3 @@ When blocked, an error, or a UI/UX anomaly occurs, please report which scenario 3. Choose a project to add the file to. 4. Select the other project. 5. Check that the file has been added successfully. - -### 4. End test -1. logout of the application diff --git a/testing/manual/simulate-calibrate-model-configuration-w-sciml.md b/testing/manual/simulate-calibrate-model-configuration-w-sciml.md index 4f97ed2de7..f8802e80c5 100644 --- a/testing/manual/simulate-calibrate-model-configuration-w-sciml.md +++ b/testing/manual/simulate-calibrate-model-configuration-w-sciml.md @@ -1,8 +1,6 @@ ## [Name of the Test Scenario] Please go through __every__ step of the test scenario.\ -When blocked, an error, or a UI/UX anomaly occurs, please report which scenario and step to [\#askem-testing](https://unchartedsoftware.slack.com/archives/C06FGLXB2CE). - -Estimated time to completion: [X] minutes +Report any issues into GitHub: [open an issue](https://github.com/DARPA-ASKEM/terarium/issues/new?assignees=&labels=bug%2C+Q%26A&template=qa-issue.md&title=%5BBUG%5D%3A+). ### 1. Begin test @@ -65,8 +63,3 @@ Estimated time to completion: [X] minutes 8. Click `Run`. 9. As the calibration runs, the "Loss function" section on the right panel should have a chart showing a history of the loss. Expect it to be a timeseries that decreases quickly then slowly. 10. When the calibration is completed successfully, there should be a chart showing the predicted values for each model variable and the dataset values to which it has been mapped. - -### 4. End test - -1. Click on "QA" on the top right of the screen. -2. Click "Logout" to log out of the application. diff --git a/testing/manual/stratify-model.md b/testing/manual/stratify-model.md index cabec88b31..238be0e6a5 100644 --- a/testing/manual/stratify-model.md +++ b/testing/manual/stratify-model.md @@ -1,8 +1,6 @@ ## [Name of the Test Scenario] Please go through __every__ step of the test scenario.\ -When blocked, an error, or a UI/UX anomaly occurs, please report which scenario and step to [\#askem-testing](https://unchartedsoftware.slack.com/archives/C06FGLXB2CE). - -Estimated time to completion: [X] minutes +Report any issues into GitHub: [open an issue](https://github.com/DARPA-ASKEM/terarium/issues/new?assignees=&labels=bug%2C+Q%26A&template=qa-issue.md&title=%5BBUG%5D%3A+). ### 1. Begin test 1. Login to https://app.staging.terarium.ai using the test account @@ -69,6 +67,3 @@ Estimated time to completion: [X] minutes 2. `Initial variables` should have new stratifications 3. `Parameters` should not have new stratifications 4. `Transitions` should have new stratifications - -### 4. End test -1. logout of the application diff --git a/testing/manual/stratify-notebook.md b/testing/manual/stratify-notebook.md index 74f0222ff9..af504dfd6a 100644 --- a/testing/manual/stratify-notebook.md +++ b/testing/manual/stratify-notebook.md @@ -1,6 +1,6 @@ ## Stratify model via the notebook interface Please go through __every__ step of the test scenario.\ -When blocked, an error, or a UI/UX anomaly occurs, please report which scenario and step to [\#askem-testing](https://unchartedsoftware.slack.com/archives/C06FGLXB2CE). +Report any issues into GitHub: [open an issue](https://github.com/DARPA-ASKEM/terarium/issues/new?assignees=&labels=bug%2C+Q%26A&template=qa-issue.md&title=%5BBUG%5D%3A+). ### 1. Begin test 1. Login to https://app.staging.terarium.ai using the test account @@ -13,8 +13,8 @@ When blocked, an error, or a UI/UX anomaly occurs, please report which scenario ### 2. Setup workflow 1. Make sure you have a model in your project. 2. Create a workflow -3. Drop in your model in the workflow -4. Create a Stratify node +3. Drop in your model in the workflow +4. Create a Stratify node 5. Connect the model to the stratify node 6. Drill down into the Stratify node and go to the notebook section. @@ -35,7 +35,7 @@ model = stratify( ``` 2. Stratify my model by the ages young and old where young can transition to old -``` +``` model = stratify( template_model=model, key= "Age", @@ -48,7 +48,7 @@ model = stratify( ``` 3. Stratify my model by the ages young and old where young and old can become old, but old cannot become young -``` +``` model = stratify( template_model=model, key= "Age", @@ -59,7 +59,7 @@ model = stratify( modify_names=True ) ``` -4. Stratify my model by the locations Toronto and Montreal where Toronto and Montreal cannot interact +4. Stratify my model by the locations Toronto and Montreal where Toronto and Montreal cannot interact ``` model = stratify( template_model=model, @@ -98,9 +98,5 @@ model = stratify( ) ``` -6. What is cartesian_control in stratify? +6. What is cartesian_control in stratify? - This should have no code response, just a message in the thought section. - - -### 4. End test -1. logout of the application diff --git a/testing/manual/Stratify-LLM-Tests.md b/testing/manual/stratify-sciml.md similarity index 93% rename from testing/manual/Stratify-LLM-Tests.md rename to testing/manual/stratify-sciml.md index 1caf0c05f5..d565cb96bd 100644 --- a/testing/manual/Stratify-LLM-Tests.md +++ b/testing/manual/stratify-sciml.md @@ -1,6 +1,6 @@ ## [Name of the Test Scenario] Please go through __every__ step of the test scenario.\ -When blocked, an error, or a UI/UX anomaly occurs, please report which scenario and step to [\#askem-testing](https://unchartedsoftware.slack.com/archives/C06FGLXB2CE). +Report any issues into GitHub: [open an issue](https://github.com/DARPA-ASKEM/terarium/issues/new?assignees=&labels=bug%2C+Q%26A&template=qa-issue.md&title=%5BBUG%5D%3A+). ### 1. Begin test 1. Login to https://app.staging.terarium.ai using the test account @@ -22,7 +22,7 @@ Ask the follow questions one at a time, wait for the response, check the respons reset the code block move on to the next. Q) "Stratify my model by the ages young and old", -A) +A) model = stratify( template_model=model, key= "Age", @@ -36,7 +36,7 @@ model = stratify( ) Q) Stratify my model by the ages young and old where young can transition to old -A) +A) model = stratify( template_model=model, key= "Age", @@ -50,7 +50,7 @@ model = stratify( ) Q) Stratify my model by the ages young and old where young and old can become old, but old cannot become young -A) +A) model = stratify( template_model=model, key= "Age", @@ -62,7 +62,7 @@ model = stratify( ) Q) Stratify my model by the locations Toronto and Montreal where Toronto and Montreal cannot interact -A) +A) model = stratify( template_model=model, key= "Location", @@ -76,7 +76,7 @@ model = stratify( ) Q) Stratify my model by the locations Toronto and Montreal where any population can travel between Toronto and Montreal -A) +A) model = stratify( template_model=model, key= "Location", @@ -105,7 +105,3 @@ model = stratify( Q) What is cartesian_control in stratify? A) No code response, instead just a message in the thought section. - - -### 4. End test -1. logout of the application diff --git a/testing/manual/test-workflow-interactions.md b/testing/manual/test-workflow-interactions.md deleted file mode 100644 index 1b20feb6d5..0000000000 --- a/testing/manual/test-workflow-interactions.md +++ /dev/null @@ -1,61 +0,0 @@ -## `[Workflow Interactions (add, remove, connect, disconnect, zoom, pan, drag and drop)]` -Please go through __every__ step of the test scenario.\ -When blocked, an error, or a UI/UX anomaly occurs, please report which scenario and step to [\#askem-testing](https://unchartedsoftware.slack.com/archives/C06FGLXB2CE). - -Estimated time to completion: [X] minutes - -### 1. Begin test -1. Login to https://app.staging.terarium.ai using the test account - ``` - email: qa@test.io - password: askem-quality-assurance - ``` -2. Create, or open, project named `QA [Your Name]` - -### 2. Test Scenarios - -`[Task 1: Adding an operator to the workflow via the "+ Add a component" menu']` -1. Create a new workflow - open it. -2. Click the "+ Add a component" primary button in the top right corner -3. Select Work with mode > Stratify model -4. __Expected Result__: The Stratify model operator node should appear on the workflow canvas - -`[Task 2: Adding a resource to the workflow via drag and drop from resource panel']` -1. Add a model (e.g. [Configured SIR.json](https://github.com/DARPA-ASKEM/terarium/blob/main/testing/data/Configured%20SIR.json)) to the project with Explorer. -2. Open the workflow created in Task 1. -3. In the Resources panel - left side of the screen - click "Model (1)" to expand it so that the model you added from Explorer is now visible -3. Click and drag that model and drag it out onto the workflow canvas. -4. __Expected Result__: The model resource node should appear on the canvas in the mouse position when the mouse button is release during click and drag. - -`[Task 3: Duplicate and removing an operator from the workflow']` -1. Click the kebab menu in the top right of the model operator node, select "Duplicate" from the menu -2. Click the kebab menu in the top right of the newly added duplicate model operator node, select "Remove" from the menu -3. __Expected Result__: The model operator node should have been duplicated, then deleted from the canvas. - -`[Task 4: Connecting an operator nodes to another operator node']` -1. Click the output port of the model operator node -2. Drag the pipe line to the input port of the Stratify model operator node, click mouse to release -3. __Expected Result__: The pipe should connect the output of the model operator node to the input port of the Stratify model operator node with the ends of the connection pipe attached to the ports. The input label on the Stratify node should be the same as the output port label of the mode, and an Edit button should appear in the Stratify model node. - -`[Task 5: Disconnecting an operator node']` -1. Hover over the Stratify Model input port label, -2. Click the Unlink button that appears on hover. -3. __Expected Result__: The pipe connecting the model and stratify model operator nodes should be deleted. The input label on Stratify Model should change back to "Model or Model Configuration" hint text and the edit button should disappear. - -`[Task 6: Re-positioning a disconnected node on the canvas']` -1. Hover the mouse anywhere over an operator node. A hover state should be indicated by the node's header changing to a darker shade of green and the node's shadow darkening. -2. Click anywhere on the operator node, drag in somewhere else on the canvas and release the mouse button. -3. __Expected Result__: The node's hover state should trigger on mouse over and the node should be repositioned on the canvas in the position of the mouse when the mouse button is released. - -`[Task 7: Re-positioning a connected node on the canvas']` -1. Repeat the steps in Task 5 to connect two nodes. -2. Repeate the steps in Task 7 to re-position one of the nodes. -3. __Expected Result__: The node should reposition as in Task 7, and the pipe connecting the nodes should flex contort to accomodate the re-position of the node. - -`[Task 8: Canvas zoom in / out and pan']` -1. With the mouse positioned over the canvas, scroll the mouse wheel up and down to zoom the canvas in and out. -2. With the mouse positioned over the canvas, click and drag the mouse around. -3. __Expected Result__: The canvas view should zoom in/out on mouse wheel scroll and pan on click and drag. - -### 3. End test -1. logout of the application diff --git a/testing/manual/transform-dataset.md b/testing/manual/transform-dataset.md index 03ed0f0c7f..db2134062f 100644 --- a/testing/manual/transform-dataset.md +++ b/testing/manual/transform-dataset.md @@ -1,10 +1,8 @@ ## Transform Dataset Operator Please go through __every__ step of the test scenario.\ -When blocked, an error, or a UI/UX anomaly occurs, please report which scenario and step to [\#askem-testing](https://unchartedsoftware.slack.com/archives/C06FGLXB2CE). +Report any issues into GitHub: [open an issue](https://github.com/DARPA-ASKEM/terarium/issues/new?assignees=&labels=bug%2C+Q%26A&template=qa-issue.md&title=%5BBUG%5D%3A+). -Estimated time to completion: [X] minutes - -### 1. Begin test +### 1. Begin test 1. Login to https://app.staging.terarium.ai using the test account ``` email: qa@test.io @@ -24,6 +22,3 @@ Estimated time to completion: [X] minutes 9. After the question has been answered, select the new dataframe (merged_df) in the dropdown located beside the "Save as" button. 10. Click the "Save as" button to save the dataframe. Name it `result`. 11. __Expected Result__: New dataset `result` should exist under your project's Datasets and contain 4 columns (id, count, label, level). - -## End test -1. logout of the application diff --git a/testing/manual/validate-model-configuration.md b/testing/manual/validate-model-configuration.md index 4d90338a93..0175ba8af1 100644 --- a/testing/manual/validate-model-configuration.md +++ b/testing/manual/validate-model-configuration.md @@ -1,6 +1,6 @@ ## Validate Model Configuration Please go through __every__ step of the test scenario.\ -When blocked, an error, or a UI/UX anomaly occurs, please report which scenario and step to [\#askem-testing](https://unchartedsoftware.slack.com/archives/C06FGLXB2CE). +Report any issues into GitHub: [open an issue](https://github.com/DARPA-ASKEM/terarium/issues/new?assignees=&labels=bug%2C+Q%26A&template=qa-issue.md&title=%5BBUG%5D%3A+). ### 1. Begin test 1. Login to https://app.staging.terarium.ai using the test account @@ -52,7 +52,7 @@ When blocked, an error, or a UI/UX anomaly occurs, please report which scenario 7. Verify that a _β:γ pairwise drilldown_ chart appears and selecting each of the rectangular regions (labelled `box`) highlights a trajectory in the _Trajectory State_ chart (both the selected rectangle and highlighted trajectory should shared the same colour) 8. Verify that the x-axis is labelled `β` with the same range as configured (`[0.4, 0.8]`) and ditto for the y-axis (`γ` and `[0.1, 0.3]`) 9. Verify that every green rectangular region maps to a green trajectory which is always between `I = 0.0` and `I = 750.0` -10. Click _X_ on the _drilldown_ chart should close it +10. Click _X_ on the _drilldown_ chart should close it ### 6. Tighten sanity check 1. Lower the upper bound of the `Infected cap` constraint from `750.0` to `100.0` (this should restrict the regions of parameter space that can satisfy this sanity check) @@ -69,9 +69,6 @@ When blocked, an error, or a UI/UX anomaly occurs, please report which scenario 5. Verify that all parameter space regions are coloured green and map to trajectories that always decrease in value over time, within the simulated time range (i.e. `dS/dt <= 0`) ### 8. Create an impossible monotonicity constraint -1. Change the direction of the `Sus` constraint from `Decreasing` to `Increasing` +1. Change the direction of the `Sus` constraint from `Decreasing` to `Increasing` 2. Click _Run_ again 3. Verify that all parameter space regions are coloured orange and all trajectories do not satisfy the new constraint, i.e. no combination of parameter values as input into the model produces outputs with `S` increasing over time - -### 5. End test -1. Logout of the application