diff --git a/docs/docs/about.md b/docs/docs/about.md index e69de29b..f26ec124 100644 --- a/docs/docs/about.md +++ b/docs/docs/about.md @@ -0,0 +1,19 @@ +# About + +GLADOS, the Automated Science Pipeline, started when Dr. Jason Yoder, a CSSE Professor at Rose-Hulman Institute of Technology, found himself building the same interface for many different computational research projects over his years of research. Finding a natural need to make this repeated process easier to set up, the project began in 2021 as a senior capstone project, with continuous iterations done by different senior capstone project teams each academic year. + +As of 2024, the project has reached basic usability, allowing for multiple experiments to be ran on the system in parallel, without requiring large changes to the original code. + +## FAQ + +* Why 'GLADOS'? + +GLADOS stands for 'General Learning and Automatic Discovery for Operationalizing Science', which is a mouthful, but has a memorable acronym, taking inspiration from the Portal series. + +* What technology does this project run on? + +GLADOS uses React and Next.js on the frontend, Python on the backend, and MongoDB for data storage, with Firebase for account management. The production server currently runs on a single node Kubernetes cluster within the Rose-Hulman firewall, though older versions primarily used Docker containers. + +* Future goals? + +In the future, GLADOS hopes to expand the Kubernetes cluster, wider experiment support, and more options for data aggregation, among other bells and whistles. This is currently being handed off to next year's senior capstone project team. diff --git a/docs/docs/documentation/guide.md b/docs/docs/documentation/guide.md index 7b686e00..5ce3a1b5 100644 --- a/docs/docs/documentation/guide.md +++ b/docs/docs/documentation/guide.md @@ -5,7 +5,7 @@ Documentation is powered by [MkDocs](https://www.mkdocs.org/) with [MkDocs Mater ## Setup For developers, a MkDocs setup is included as part of the dev install script. When working with the documentation locally, remember to activate the venv: -``` +```sh source docs/.venv/Scripts/activate ``` @@ -13,15 +13,19 @@ Otherwise, mkdocs commands will likely not work. To exit the venv, while in the venv, run -``` +```sh deactivate ``` +!!! note + + For any mkdocs commands, you must be in the outermost `docs` folder for it to find your files correctly. + ## Updating Updating documentation is as simple as updating Markdown files, all contained within the `docs` folder in the Monorepo. To preview your changes, this will build the site locally, with automatic updates as you make changes: -``` +```sh mkdocs serve ``` @@ -30,14 +34,14 @@ mkdocs serve With Github Actions, updates to the `main` branch will automatically build and deploy the static site to the `gh-pages` branch. To deploy manually, run -``` +```sh mkdocs gh-deploy --force ``` This updates the `gh-pages` branch with the static site. Keep in mind, if there are unpublished changes, it will display those changes as well, so beware! To preview what files will be generated and published, this generates the static website files under `docs/site/`: -``` +```sh mkdocs build ``` diff --git a/docs/docs/index.md b/docs/docs/index.md index 3594263d..9ded4641 100644 --- a/docs/docs/index.md +++ b/docs/docs/index.md @@ -1,19 +1,21 @@ # Welcome to GLADOS -## For Users +GLADOS is an Automated Science Pipeline, allowing for computational research experiments to get aggregated results from multiple experiment runs easily. Currently developed by seniors at Rose-Hulman Institute of Technology, the goal is to develop a system that makes it simple to run experiments and gather relevant data, through a low cost, open-source, community-built project. When you're on the Rose-Hulman internal network, you can access the deployed project at . -**To get a local copy of the system running on your own computer**: Start by following the [User guide](tutorial/installation.md). +## Usage -**To run an experiment on a running copy of the system**: Follow the [Running an Experiment guide](tutorial/usage.md). +**To run an experiment on a running copy of the system**, [get started here](tutorial/usage.md). -## For Devs +**To get a local copy of the system running on your own computer**, start by following [installation steps](tutorial/installation.md). -To set up a new machine for development, follow the [Dev guide](tutorial/development_installation.md). +## Development + +To set up a new machine for development, follow the [developer installation guide](tutorial/developer_installation.md). To get familiar with the components of the system, check out the [Repository Guide](repository_guide.md). -This wiki also contains docs from the [old repo](https://github.com/GLADOSPipeline/Monorepo/wiki) prefixed with `Old`. + diff --git a/docs/docs/repository_guide.md b/docs/docs/repository_guide.md index 5c77b705..47d4bbd9 100644 --- a/docs/docs/repository_guide.md +++ b/docs/docs/repository_guide.md @@ -1,7 +1,9 @@ -# This page is currently being migrated, and consists of old information - # Repository Guide +!!! warning + + This document is out of date, originating from the 2022-23 team, and is currently being migrated. There are some broken links, and some unfinished content. + In order to keep the entire codebase in sync as it is developed, we have opted to organize every service into a singular monorepo. diff --git a/docs/docs/tutorial/developer_installation.md b/docs/docs/tutorial/developer_installation.md new file mode 100644 index 00000000..e73c0a80 --- /dev/null +++ b/docs/docs/tutorial/developer_installation.md @@ -0,0 +1,157 @@ +# Developer Installation + +This guide contains a short tutorial on getting a local development environment up and running. + +!!! warning + + As of March 2024, we have shifted towards running the system on Kubernetes, rather than Docker. These steps still work with the current codebase, but as we fully migrate the codebase to our Kubernetes version, these steps will need to be updated accordingly. + +!!! note + + These steps mainly concern developing on a Windows computer, with Git Bash. Steps on other OS's, or with WSL terminal, will have the same steps, although the specific steps on installation, as well as configuration, may differ. + +## Dependencies + +If you have already followed the [Installation guide](installation.md), [skip ahead](developer_installation.md#tool-installation). If not, continue reading. + +## Required Tools + +### Git + +Git is a code version management tool that integrates with Github, where our codebase is stored. + +If you don't already have it installed, you can download it on the [git website](https://git-scm.com/downloads). + +### Docker + +Docker is a utility that manages small virtual machine "containers." + +To install it, follow the instructions on [Docker's documentation](https://docs.docker.com/get-docker/). + +You might encounter a permissions error on Windows +mentioning not being part of the docker-users group. +Directions to fix that can be found [here](https://icij.gitbook.io/datashare/faq-errors/you-are-not-allowed-to-use-docker-you-must-be-in-the-docker-users-group-.-what-should-i-do). + +#### WSL Error + +To get Docker to work properly, you will need to install WSL. +This may fail for Windows. In this case, check that virtualization is enabled in the BIOS. + +## Retrieve the Code + +In a location of your choice in a drive with at least 1 GB of free space, +use Git to clone the repo. + +Your command will probably look like this: + +```bash +git clone --recurse-submodules https://github.com/AutomatingSciencePipeline/Monorepo.git +``` + +Or this if you have [ssh keys](https://docs.github.com/en/authentication/connecting-to-github-with-ssh) set up (Recommended): + +```bash +git clone --recurse-submodules git@github.com:AutomatingSciencePipeline/Monorepo.git +``` + +The `--recurse-submodules` can be omitted if you don't want to download a copy of the docs as well. + +If cloning isn't working for some reason, +you can download a zip from [here](https://github.com/AutomatingSciencePipeline/Monorepo/archive/refs/heads/main.zip) instead. +Note that, if you download a zip, Windows or antivirus programs may block you from running the code, +in which case you will have to navigate various security dialogs to unblock it. +It may also not have the correct line endings (CRLF vs LF) for your system. + +## Get the .env file(s) + +The system keeps track of private keys (strings that act as passwords to various services) in a `.env` file. +For security reasons, these cannot be publicly distributed with the rest of the code. + +You can get a copy of our `.env` file from our private Discord in the [#env-files channel](https://discord.com/channels/1017208818989539368/1042935101601873970). Note that you must already be a member of the Discord to open this link; contact us if you need access. + +The file should be named exactly `.env` and placed in the root folder of the repository. + +#### Alternative: Set up your own .env file + +You can create your own `.env` file, if you want to set up your own GLADOS system. There is an `.example.env` file in the repository that has sample fields to fill in. During this process, you will need to set up a Firebase project for account management. Note that the frontend code has the Firebase public key in it, which does not need to be secure, but will need to be replaced when hosting your own GLADOS system. + +## Tool Installation + +We include a `dev-install.sh` script that assists in the process of installing everything needed to develop on the system. This setup process includes: + +* Installing Docker +* Installing Python 3.8 via pyenv +* Creating pipenvs for both the backend and documentation packages, containing project dependencies +* Installing Node.JS via npm, with project dependencies + +Of course, it is recommended to look through the scripts themselves, before running it, for a sanity check. + +To run the script, run the following command with a Git Bash terminal in the root folder of the repository: + +```sh +bash dev-install.sh +``` + +This process may require you to close and reopen the terminal, uninstall programs you may have, or restart your computer. You may need to run the script multiple times, as a result. + +## IDE Setup + +We strongly suggest using Visual Studio Code (VSCode) as your IDE. +Numerous configuration files and utilities, such as suggested extensions, are already included with the repo. + +It can be downloaded [here](https://code.visualstudio.com/download). + +After installing VSCode, you should install the recommended extensions after opening the repository in VSCode. A prompt may appear after opening the project, but if not, you can also go to the Extensions view, filter by '@recommended', and install the extensions under 'Workspace Recommendations'. + +Configure the python interpreter via Ctrl+Shift+P > `Python: Select Interpreter` > `Use Python from 'python.defaultInterpreterPath' setting` + +### Check Linter Setup + +In order to test the frontend linter, we'll introduce a syntax error and make sure that it reports it as a problem. You can undo this change after you're done. + +* Open `/apps/frontend/pages/_app.tsx` +* Add the line `var unused = 'hi';` as shown in the screenshot below and confirm that the same 3 errors are reported when you hover over the symbol `unused`. + +![Intentional Error](https://i.imgur.com/6at100I.png) + +If the linter doesn't detect 3 errors, make sure the recommended extensions are installed, and check the VSCode "Output" View, select ESLint from the dropdown, and check for any errors. + +You may run into CI issues, with build failures, without a working linter. + +You are now fully set up to develop on the system! + +## Quick Launching with Docker + +You can quickly build and launch all service containers via VSCode Tasks. + +Ctrl+Shift+P > `Tasks: Run Task` > `DEVELOPMENT Docker Compose Up (Attached)` + +If you don't want to use VSCode tasks to start the commands, look at the `.vscode/tasks.json` file to see what commands are run behind the scenes. + +This supports enable hot reloading for the backend and frontend. +Once the docker containers are built (this will take a while for the first run) you should start seeing log messages from the containers. + +The Docker Desktop application will tell you what port the frontend is accessible from, but it should match the port specified in your `.env` file. + +![Docker Desktop screenshot](https://i.imgur.com/W8zJPks.png) + +Click on the link shown in the screenshot and it should open a page to the frontend in your browser. +Once you try to connect, the frontend will start to build for hot reloading. + +The frontend's hot reloading is really slow at first but gets better after its first builds are done. It may take up to **a few minutes** to turn on for the first time. +While it is building, your browser will be sitting on a blank page still trying to connect. This is normal. + +After it finishes loading, you should be taken to the homepage. + +## Extras + +!!! note + + These are optional resources that you may consider using, although we have found a simpler setup to work better. + +### Remote Development Extensions + +The Remote Development extension pack for VS Code allows you to connect your IDE to connect to remote servers, or containers, and develop from them as if they were your local computer. + +It can be installed with [this quick install link](vscode:extension/ms-vscode-remote.vscode-remote-extensionpack) +or downloaded via its [marketplace page](https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.vscode-remote-extensionpack). diff --git a/docs/docs/tutorial/development_installation.md b/docs/docs/tutorial/development_installation.md deleted file mode 100644 index e69de29b..00000000 diff --git a/docs/docs/tutorial/installation.md b/docs/docs/tutorial/installation.md index 649095a0..ae066a73 100644 --- a/docs/docs/tutorial/installation.md +++ b/docs/docs/tutorial/installation.md @@ -37,8 +37,6 @@ Directions to fix that can be found [here](https://icij.gitbook.io/datashare/faq To get Docker to work properly, you will need to install WSL. This may fail for Windows. In this case, check that virtualization is enabled in the BIOS. - - ## Retrieve the Code In a location of your choice in a drive with at least 1 GB of free space, diff --git a/docs/docs/tutorial/usage.md b/docs/docs/tutorial/usage.md index e69de29b..f8e100c8 100644 --- a/docs/docs/tutorial/usage.md +++ b/docs/docs/tutorial/usage.md @@ -0,0 +1,215 @@ +# Usage + +!!! note + + This section is currently being rewritten. + +This section covers how to run an experiment on the system. + +Definitions: + +* **Trial**: A single run of submitted code with a corresponding config file + +## Connect to the System + +When on the Rose network, you can access the live copy at (note: this is **not** https). + +If you need a local copy the system, view the [installation guide](installation.md). + +## Create An Account + +If you don't have an account yet, create one by filling in the relevant fields on the homepage. + +If you have an account, it should be auto-detected and offer to take you to the dashboard. Otherwise, got to the sign in page via the "Log In" button and enter your credentials. + +## Prepare your Code + +Currently, the system only accepts single file Python projects, Java .jar applications that have a single return value or output to a csv file, or Unix-compiled executables (C projects) that output to a csv file. + +### Python + +Example experiments written in Python can be found [in the repository](https://github.com/AutomatingSciencePipeline/Monorepo/tree/main/example_experiments/python). + +Before submitting an experiment to the system, you may have to: + +* Ensure it can run on Python 3.8 +* Ensure it does not require downloading any external packages +* Ensure it does not use Python's multiprocessing/threading features +* Modify your code so that it accepts the location of an `.ini` config file as a command line argument +* Use the `configparser` library to read the passed ini file to obtain trial parameters +* Modify your code to print out the return value to the command line (if you wish to use that approach for collecting experiment output) + +Consider checking out the examples linked above to see how to do this in practice. + +A guide for using .ini config files can be found here: (TODO migrate to wiki page) + +Set up your code such that it will accept a .ini config file in the form of + +```ini +[DEFAULT] +g = 5 +p = 50 +gl = 1 +mr = 0.2 +s = 1 +``` + +Once you're ready, proceed to [Uploading to the System](#uploading-to-the-system). + +### Java + +Example experiments written in Java can be found [in the repository](https://github.com/AutomatingSciencePipeline/Monorepo/tree/main/example_experiments/java). + +Similarly to Python configure your Java project to accept the location of a properties config file as a command line argument to get access to experiment parameters, also have your file print out the return value to the command line. + +Set up your code such that it will accept a .properties config file in the form of + +```properties +g = 5 +p = 50 +gl = 1 +mr = 0.2 +s = 1 +``` + +After editing your Java project, export it into a .jar file. + +A good guide on how to do so in IntelliJ can be found [here](https://www.jetbrains.com/help/idea/compiling-applications.html) + +### C + +Example experiments written in Java can be found [in the repository](https://github.com/AutomatingSciencePipeline/Monorepo/tree/main/example_experiments/c). + +Configure your C project to accept the location of a properties config file as a command line argument to get access to experiment parameters, and have your file print out the return value to the command line. + +Set up your code such that it will accept a .ini config file in the form of: + +```ini +[DEFAULT] +g = 5 +p = 50 +gl = 1 +mr = 0.2 +s = 1 +``` + +To do this, utilizing [this ini parser](https://github.com/benhoyt/inih) works great. + +Compile your C project for Unix systems via gcc: + +```bash +gcc -o +``` + +More info can be found [here](https://steffanynaranjo.medium.com/how-to-compile-c-files-with-gcc-step-by-step-5939ab8a6c47) + +### All types + +Set up your code to output to a two-line csv that consists of headers and results like so + +```csv +HeaderFor1, HeaderFor2 +Result1, Result2 +``` + +## Uploading to the System + +This example will walk through running the Python Genetic Algorithm Experiment, which can be found [in the repository's examples](https://github.com/AutomatingSciencePipeline/Monorepo/tree/main/example_experiments/python). + +When first logged in, this should be what you see (except you may not have any experiments listed yet) + +![landing page](https://user-images.githubusercontent.com/23245825/201237091-42cd26fa-8649-4ecb-9386-bbabc3a85a01.PNG) + +To start an experiment click the blue 'new experiment' button on the top left corner on the page. +This will bring up the Experiment Information pane. + +![informationStep](https://user-images.githubusercontent.com/23245825/223880912-8f234bb7-0958-4a04-ad18-81bd33d3b9cc.png) + +TODO we should add this kind of info as hover-tooltip-helpers near each field so you don't need to reference other documents for help + +Field Specifications: + +* **Trial Result (This field is required):** If your project creates a two-line csv file to store the output of the trial, specify the name and extension of the csv file in the Trial Result textbox. (This can be the same as Extra Trial File) +* **Trial Extra File:** If your project creates an extra file as its result, specify the name and extension of the file in the Extra Trial File textbox so that the system knows how to find it. (This can be the same as Trial Result) +* **Keep Logs**: If you want to store any system prints from a trial, leave this checked, otherwise uncheck it. + +Here's an example of a filled out information page: + +![filledInformationPage](https://user-images.githubusercontent.com/23245825/223881669-9d3790fa-115b-498b-9751-346b6047820f.png) + +Clicking next will bring you to the parameter input page: + +![parameterPage](https://user-images.githubusercontent.com/23245825/223881744-33f91bb3-0eb1-44dd-bff2-4bb2964be5a1.png) + +There are 4 supported parameter types; Integers, Floats, Strings, and Booleans. Strings and booleans are treated as constants and are not iterated upon. + +* Integers and floats have 4 fields, default, min, max, and step. default is the default value the variable will have when not being iterated. min, max, and step determine how the variable will be iterated. +* Strings and Booleans are constants + +To add a parameter to the experiment click on the box that contains the name of desired type of parameter. This will add a form item that you can enter more information into like so. + +![emptyParams](https://user-images.githubusercontent.com/23245825/223882117-d28d2a96-c3af-425b-97cb-16005c3b7a42.png) + +You can then enter in the required information. + +Here's an example of a filled parameter page: + +![filledParams](https://user-images.githubusercontent.com/23245825/223882365-97057fe0-9e48-49ec-bcd4-8ddc88661b02.png) + +Once you have entered your parameter information, clicking next will bring you to the user defined constants page. + +![constsPage](https://user-images.githubusercontent.com/23245825/223882904-ae4646b3-bcd8-417a-9f74-143c67092bea.png) + +On this page you can optionally define the values for the variables that do not need to be iterated upon. +You must input information into the text area as if it were text inside an .ini or .config file, because it will be appended directly to the end of each trial's input ini file. + +Here's an example of a possible valid entry in this field: + +```ini +[const] +a = -1 +;test comment +b = 10.5 +invert = False +[Strings] +c = Test String +``` + +Any input to the textarea will not be validated, so make sure that the data is formatted correctly! + +Clicking next will bring you to the Post Process page: + +![postProcess](https://user-images.githubusercontent.com/23245825/223883336-350b4a7a-f85a-4763-9d52-4cfc33425e20.png) + +There is currently only one post processing option: it generates a Scatter Plot with a line of best fit using the data from the results of the experiment. The dependent and independent variables can be any of the user defined variables or constants. You can also use any of the headers that are defined in the TrialResults csv + +Clicking next will bring up the confirmation page: + +![confirmationPage](https://user-images.githubusercontent.com/23245825/223884249-eab24c91-2b8f-43f9-9423-4c9e156e99df.png) + +This contains a json file of what will be passed to the backend you can check it again here. +Clicking next will bring up the Dispatch Page. Drop the .py or .jar file that you want the experiment to run on and click Dispatch which will start the experiment. + +## Experiment is Running + +While the system is generating the config files this is what will appear on the dashboard: + +![awaitingExpStart](https://user-images.githubusercontent.com/23245825/223901376-b047a2bc-9dc2-40d2-9d0e-6a0185f7f6cb.png) + +Once the experiment starts running trials the item on the dashboard will provide; how many trials are to be run, how many trials have been run so far, the number of successes and failures, and an estimation of how long it will take the experiment to complete. + +![experimentInProgress](https://user-images.githubusercontent.com/23245825/223901790-f6476518-b0bf-4745-814a-c621dc564ac3.png) + +Once the experiment has completed you will be able to download the result.csv from the "Download Results" button which contains the output and the configuration used to get said result for each trial run. + +If you had an extra file produced by the experiment, chose to keep the logs from the experiment, or chose to do any post processing, you will also be able to download a zip containing those files. + +An example of a completed experiment can be seen here: + +![completedExperiment](https://user-images.githubusercontent.com/23245825/223902650-ed63760d-ce95-4f57-a571-a38dda141848.png) + +If you wish to run the same experiment again while changing parameters you can click on "Copy Experiment" and it will open up a new experiment window with the values from the previous experiment copied over. + +## Known bugs + +* If something catastrophically goes wrong and the experiment cannot continue, the user is not notified. So if an experiment is stuck on "Experiment Awaiting Start" or stuck on a trial run for a long period of time it is likely that the experiment has crashed. This is currently being addressed and should be fixed in the future. diff --git a/docs/mkdocs.yml b/docs/mkdocs.yml index fc1a5e19..548691f0 100644 --- a/docs/mkdocs.yml +++ b/docs/mkdocs.yml @@ -4,7 +4,20 @@ site_description: Documentation for GLADOS - Automated Science Pipeline site_url: https://automatingsciencepipeline.github.io/Monorepo/ repo_url: https://github.com/AutomatingSciencePipeline/Monorepo -edit_uri: '' +edit_uri: '' # edit is hidden + +# Site navigation - remember to update when creating new pages! +nav: + - Home: index.md + - About GLADOS: about.md + - Tutorial: + - Usage: tutorial/usage.md + - Installation: tutorial/installation.md + - Developer Installation: tutorial/developer_installation.md + - Troubleshooting: tutorial/troubleshooting.md + - Documentation: + - Updating Documentation: documentation/guide.md + - (Old) Repository Guide: repository_guide.md # Read about themes here: https://www.mkdocs.org/user-guide/choosing-your-theme/ theme: @@ -30,14 +43,3 @@ markdown_extensions: - pymdownx.details - pymdownx.superfences -# Site navigation - remember to update when creating new pages! -nav: - - Home: index.md - - About GLADOS: about.md - - Tutorial: - - Usage: tutorial/usage.md - - Dev Installation: tutorial/installation.md - - Troubleshooting: tutorial/troubleshooting.md - - Documentation: - - Updating Guide: documentation/guide.md -