From 17ba9ee768dd5f54f7fa7a52c48a98f26e972544 Mon Sep 17 00:00:00 2001 From: rHorsey Date: Tue, 6 Feb 2018 10:05:08 -0700 Subject: [PATCH 1/2] Update README.md --- README.md | 161 ++++++++++++++++++++++-------------------------------- 1 file changed, 64 insertions(+), 97 deletions(-) diff --git a/README.md b/README.md index cadb495ad..46df4b49e 100644 --- a/README.md +++ b/README.md @@ -5,134 +5,101 @@ OSX Build: [![Build Status][travis-img]][travis-url] Windows Build (Under Development): [![Build status][appveyor-img]][appveyor-url] -## Description -The preferred development approach for this application is to use Vagrant to provision and test the server. To see the server instructions go to [OpenStudio Rails Application](./openstudio-server/README.md) +## Standard Use Cases: -## Instructions +There are two primary ways for non-application-developers to use this codebase. The first is through the Parametric Analysis Tool (PAT) which both runs this codebase locally on a system and interfaces with local and AWS docker based instances. This can be accessed through downloading the [official OpenStudio release](https://www.openstudio.net/downloads). The second is through the OpenStudio Analysis Spreadsheet, (the Spreadsheet) which is can be downloaded or cloned from it's [github repository](https://github.com/NREL/OpenStudio-analysis-spreadsheet). -### Vagrant +## Application Development and Deployment: -- Install [Vagrant] *1.8.5*, [VirtualBox] *5.1.2*, and [ChefDK] - -[Vagrant]: http://www.vagrantup.com/ "Vagrant" -[VirtualBox]: https://www.virtualbox.org/ "VirtualBox" -[ChefDK]: https://downloads.chef.io/chef-dk/ "ChefDK" +There are primarily three ways to utilize and deploy this codebase. The first pathway is through the [openstudio_meta](./bin/openstudio_meta) CLI. This deployment strategy allows for the server to be deployed on a local desktop without docker through a pre-compilation process of all required gem dependencies. Additionally, it allows for cloud instances to be created and analyses run on them. The second pathway utilizes [Docker Compose](https://docs.docker.com/compose/). This is the preferred environment for application development, as it is allows for rapid iteration and does not encumber developers with deployment configuration details. The recommended deployment pathway uses [Docker Swarm](https://docs.docker.com/engine/swarm/), an orchestration engine which allows for multi-node clusters and provides significant benefits in the forms of customization and hardening of network and storage fundamentals. -- Check out this git repo: see the instruction on the Wiki. +### openstudio_meta: -- Install the Vagrant plugins for bootstrapping Chef and Berkshelf +The [openstudio_meta](./bin/openstudio_meta) file is a ruby script which provides access to packaging and execution commands which allow for this codebase to be embedded in applications deployed to computers without docker. Deployment requires that [MongoDB v3.2](https://www.mongodb.com/download-center#previous) and [Ruby v2.2](https://www.ruby-lang.org/en/news/2014/12/25/ruby-2-2-0-released/) are additionally packaged. For an example of cross-platform deployment please see the OpenStudio build guide for the [2.X releases](https://github.com/NREL/OpenStudio/wiki/Configuring-OpenStudio-Build-Environments) and the [CMake lists](https://github.com/NREL/OpenStudio/blob/develop/openstudiocore/CMakeLists.txt). -```sh -vagrant plugin install vagrant-omnibus vagrant-berkshelf -``` - -- Also install the Vagrant AWS plugin as the vagrant file will fail if not installed - -```sh -vagrant plugin install vagrant-aws -``` - -- Virtualbox 4.3+ users may want to install the vagrant-vbguest plugin to update the guest version +The openstudio_meta deployment relies on the `install_gems` command, which uses local system libraries to build all required gem dependencies of the server. Additionally, the export flag allows for the resulting package to be automatically assembled and zipped for deployment. It is important to note that when used on OSX and Linux systems, it is critical to not specify the export path with home (`~`) substitution. Instead, pass a fully specified path to the desired output directory. -```sh -vagrant plugin install vagrant-vbguest -``` +Once compiled or unpacked, the openstudio_meta file can be used for starting and stopping local and remote server, and submitting analyses to both. Assembling the required files for the analysis is left to either the OpenStudio Analysis Spreadsheet (the Spreadsheet) or the Parametric Assessment Tool (PAT). The Spreadsheet has a similar interface for submitting analyses to servers, and PAT makes complete use of the openstudio_meta features. For more details, please refer to the [wiki](https://github.com/NREL/OpenStudio-server/wiki/CLI). -- Start VirtualBox (Windows only) +### Local Docker Development: -- **NREL ONLY** Set environment variables or bypass SSL proxy -Either login to the SSL Developer VPN or set the environment variables below. -- **NREL Windows** -```bat -cd \path\to\Vagrantfile -rem for each cmd window set the environment variable (or set globally (for NREL only) -set OMNIBUS_INSTALL_URL=http://www.opscode.com/chef/install.sh +To develop locally the following dependency stack is recommended. First, please install Docker. The recommended path for OSX users is to install Docker CE for Mac](https://docs.docker.com/docker-for-mac/install/). If using Windows 10, please use [Docker CE for Windows](https://docs.docker.com/docker-for-windows/install/); if using a Windows version pre 10, please use Docker Toolbox. You will need to install and configure dependencies, including [VirtualBox](https://docs.docker.com/toolbox/toolbox_install_windows/#next-steps). Please refer to [this guide]() from Docker on configuring your system pre Windows 10, [this guide](https://docs.docker.com/docker-for-windows/) if using Docker CE on Windows 10, and [this guide](https://docs.docker.com/docker-for-mac/) if using Docker CE on OSX. If using a linux flavour, please follow the instructions in the [appropriate guide](https://www.docker.com/community-edition) to install Docker CE on your system and configure it. Note that Docker Compose version 1.13.0 or higher is required, and that Docker Server version 17.9.01 is strongly recommended. Although generally newer versions of docker will behave as expected, certain CLI interactions change between releases, leading to scripts breaking and default behaviours, particularly regarding persistence, changing. The docker version installed and running can be found by typing `docker info` on the command line. +#### Run Docker Compose ``` -- **NREL Mac / Linux** -```sh -cd /path/to/Vagrantfile -# for each cmd window set the environment variable (or set globally (for NREL only) -OMNIBUS_INSTALL_URL=http://www.opscode.com/chef/install.sh +docker-compose build ``` - -- Start the VM and let it provision: -```sh -vagrant up +... [be patient](https://www.youtube.com/watch?v=f4hkPn0Un_Q) ... If the containers build successfully start them by running `docker volume create --name=osdata && docker volume create --name=dbdata && OS_SERVER_NUMBER_OF_WORKERS=5 docker-compose up` where N is equal to the number of worker nodes you wish to run. For single node servers this should not be greater than the total number of available cores minus 4. + +Resetting the containers can be accomplished by running: +``` +docker-compose rm -f +docker volume rm osdata dbdata +docker volume create --name=osdata +docker volume create --name=dbdata +docker-compose up +docker-compose service scale worker=N + +# Or one line +docker-compose rm -f && docker-compose build && docker volume rm osdata dbdata && docker volume create --name=osdata && docker volume create --name=dbdata && OS_SERVER_NUMBER_OF_WORKERS=5 docker-compose up && docker-compose service scale worker=N ``` - Note, if the Vagrant provision fails, run `vagrant provision` at command line again and see if it gets past the issue. -- **NREL ONLY** Disable HTTPS -If you are inside the NREL firewall then you will need to disable HTTPS on rubygems. +Congratulations! Visit `http://localhost:8080` to see the OpenStudio Server Management Console. - - Log into Vagrant VM +#### Running the Docker CI testing locally -```sh -vagrant ssh ``` - - (Or use [PuTTy](http://stackoverflow.com/questions/9885108/ssh-to-vagrant-box-in-windows) on Windows.) - -- Add http://rubygems.org to gem sources - -```sh -sudo -i -gem sources -r https://rubygems.org/ -gem sources -a http://rubygems.org/ +docker-compose rm -f +docker volume rm osdata +docker volume create --name=osdata +export RAILS_ENV=docker-test +export CI=true +export CIRCLECI=true +sed -i -E "s/#TEST#//g" Dockerfile +sed -i -E "s/.git//g" .dockerignore +docker-compose -f docker-compose.test.yml build +docker-compose -f docker-compose.test.yml run -d rserve +docker-compose -f docker-compose.test.yml run -d web-background +docker-compose -f docker-compose.test.yml run -d db +mkdir -p reports/rspec +docker-compose -f docker-compose.test.yml run web + +# Or condenced +sed -i -E "s/#TEST#//g" Dockerfilesu +sed -i -E "s/.git//g" .dockerignore +docker-compose rm -f && docker-compose -f docker-compose.test.yml build && docker volume rm osdata && docker volume create --name=osdata && docker-compose -f docker-compose.test.yml up +git checkout -- Dockerfile .dockerignore ``` -- Exit the VM and then reprovision the VM +### Docker Deployment: -```sh -vagrant provision -``` +To deploy the OpenStudio Server in a docker-based production environment one or more machines need to be running Docker Server version 17.9.01. If using docker on a linux machine it is recommended that significant storage be available to the `/var` folder. This is where Docker reads and writes all data to by default. In addition, advanced users may wish to consider using specialized [storage drivers](https://docs.docker.com/engine/userguide/storagedriver/). Please refer to the [wiki](https://github.com/NREL/OpenStudio-server/wiki) page for additional details and a [configuration and reset guide](). Deploying a production docker swarm system outside of AWS (where complications are managed and support by NREL) can be a non-trivial problem that may require significant systems administration experience. Those embarking on this process are encouraged to refer to the scripts used by Packer to configure [Ubuntu](https://github.com/NREL/OpenStudio-server/blob/develop/docker/deployment/scripts/aws_system_init.sh) and [docker](https://github.com/NREL/OpenStudio-server/blob/develop/docker/deployment/scripts/aws_osserver_init.sh) in the base AMI images, as well as the scripts used to provision the [server](https://github.com/NREL/OpenStudio-server/blob/develop/docker/deployment/scripts/server_provision.sh) and [worker](https://github.com/NREL/OpenStudio-server/blob/develop/docker/deployment/scripts/worker_provision.sh) nodes upon instantiation in a cluster. + +## Testing procedure: - Note, if provisioning fails continue to call the `vagrant provision` command +The OpenStudio Server project uses several CI systems to test both local and cloud deployments across multiple platforms. CircleCI is used to build and test docker containers for each commit. TravisCI is used to build and test local deployments of the server on OSX hardware for each commit. It is important to note that durring the middle of the day, these tests can take several hours to begin. Finally, AppVeyor is used to build and test local deployments against Windows. -- Test the Rails application by pointing your local browser to [http://localhost:8080](http://localhost:8080) +In the case of local deployments, (Travis and AppVeyor,) the build step uses the meta-cli's install_gems command to create a new set of cached ruby dependencies to test against. The test phase is made up of two spereate testing methodologies. The first uses rspec to run a number of unit tests against using a locally instantiated server. The second instantiates the server in the same maner as PAT, runs analyses against said server, and ensures that it stops as expected, using the meta-cli. +For cloud deployments, the two critical artifacts are the docker containers and AMIs. Currently AMI testing is not automated, and unlikely to be automated for several reason. The docker containers, however, are extensivly tested using the same rspec functionality as mentioned above. - **Windows** +For a pull request to be merged under regular order, 5 CI tests need to return green: TravisCI PR and push, AppVeyor PR and push, and CircleCI PR. All of these tests, except for the CircleCI tests, write verbose results and logs on failure, which should allow for local reproduction of the bug and subsequent fixes. In the case of a failure of the CI infrastructure, please open an issue in the repository regarding the failure. - Windows users may want to install cwRsync and use the rsync method of mounting shared drives. - This eliminates the performance hit in standard vagrant sharing, especially with drives on different indexes. - Make sure there is only one rsync process found on the system (ex, remove the Rtools/rsync.exe) +## Commands to update gems used in PAT manually: -## Git SSH Issues +To test the impact of upgraded gems on PAT's functionality the currently recommended path is to manually remove and recreate the cached set of gems, including compiled binary components. This process is platform specific. Currently instructions are only available for OSX, due to complications compiling the binary component of gems with the ruby instillation provided in the OpenStudio installer package. -If you experience issues accessing git:// protocols issues (typically because of a proxy denying access), then you can globally set the https:// protocol +### OSX: ``` -git config --global url."https://".insteadOf git:// +cd /Applications/OpenStudio-X.Y.Z/ParametricAnalysisTool.app/Contents/Resources/OpenStudio-server # Change directory to the install location of the Server +rm -rf /gems # Remove the pre-packaged gems +vi server/Gemfile # Edit the Gemfile +rm server/Gemfile.lock # Remove the cached gem specifications +../ruby/bin/ruby ./bin/openstudio_meta install_gems # Reinstall the gems required (including new gems) +chmod -R 777 gems # Modify privileges on the installed gems ``` -## Deploying to Amazon EC2 - -### Releasing a New Version - -- Create a pull request from your feature branch to develop -- Ensure the tests are passing (to be added) for the pull request -- Merge the pull request into develop -- Ensure the version id, found in the server [version.rb](./server/lib/openstudio_server/version.rb) file, is incremented to be one ahead of the master verion, using standard semantic versioning -- Create a pull request to master and assign it to @bball, @nllong, @rhorsey, or @evanweaver -- One of the above will review the pull request, and either re-assign it for further update, or merge it to master -- When it is merged to master, the merger will release a new build using the `rake release` script -- Check the [available OSServer AMI json](http://s3.amazonaws.com//openstudio-resources/server/api/v2/amis.json) regularly to see when the build becomes available - -### Releasing a One Off Version - -- Create a new branch off of a commit of develop or master which corresponds to the desired base state of your server -- Update the server code as needed -- After making all changes, it is recomended that you create a pull request back to develop, applying the DO NOT MERGE tag, to allow the continuous integration tests to run against your changes -- After ensuring all test are sucessfull, close the pull request. -- Add a suffix to the version id by setting a flag in the [version.rb](./server/lib/openstudio_server/version.rb) file -- NOTE: THIS STEP WILL BE CHANGED SHORTLY: Follow the instructions found [here](./vagrant) to run the [create_vms.rb](./vagrant/create_vms.rb) script. This will build the AMIs and register them with Amazon EC2. -- Check the [available OSServer AMI json](http://s3.amazonaws.com//openstudio-resources/server/api/v2/amis.json) regularly to see when the build becomes available - -### Official AMI Generation - -For OpenStudio PAT, Alex Swindler has to copy over the AMI ID from this file: http://developer.nrel.gov/downloads/buildings/openstudio/api/amis_v2.json to this file: http://developer.nrel.gov/downloads/buildings/openstudio/rsrc/amis.json - ## Questions? Please contact @rhorsey, @bball, or @nllong with any question regarding this project. Thanks for you interest! From 2d9755806046e21d099051ca30ffa9e3244da310 Mon Sep 17 00:00:00 2001 From: rHorsey Date: Wed, 28 Feb 2018 18:23:47 -0700 Subject: [PATCH 2/2] Adding issue template --- .github/ISSUE_TEMPLATE.md | 170 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 170 insertions(+) create mode 100644 .github/ISSUE_TEMPLATE.md diff --git a/.github/ISSUE_TEMPLATE.md b/.github/ISSUE_TEMPLATE.md new file mode 100644 index 000000000..90e812ab9 --- /dev/null +++ b/.github/ISSUE_TEMPLATE.md @@ -0,0 +1,170 @@ + +*General Issue* + +**Description** + + + +**Reproduction steps:** +1. +2. +3. + +**Actual outcome:** + + +**Expected outcome:** + + +**Other information (i.e. issue is intermittent):** + + +*Meta-CLI / PAT Server Fails to Launch Details* + +**Client operating system version:** + +(i.e. OSX 10.10.2 or Windows 7 SP1) + + +**OpenStudio Meta-CLI providence** + +(i.e. PAT 2.4.1 or clone of openstudio-server SHA abcd1234) + +**Command executed:** + +``` +(paste the executed command run here - if using PAT this can be retrieved +from the developer tools console window. You may need to enable debug +messages and reproduce the error again.) +``` + +**Result of command:** + +``` +(paste the command results here - again if using PAT this can be +retrieved from the developer console window. Please ensure no +sensitive information - e.g. sensitive environment variables +- is included in the log.) +``` + + + +*Error on Local Server Details* + +**Client operating system version:** + +(i.e. OSX 10.10.2 or Windows 7 SP1) + + +**OpenStudio Server providence** + +(i.e. PAT 2.4.1 or clone of openstudio-server SHA abcd1234) + +**Local log files** + + +(While the server is still running in the errored state - i.e. PAT is still +open - please go to the project directory, and zip the log directory. Then +attach the zip file to the issue submission, and replace this text with +'Attached'.) + + + +*Error on Remote Server Details* + +**AMI version being used** + +(i.e. 2.4.1 - this information can be found under the Server Information +header on the Admin page of the OpenStudio Server instance.) + +**Server management log file** + +(Navigate to the server management console, click on the analysis that +errored, and then click on the 'View: Log' button. Copy the entire page +using control/command a into a text editor, and save the file as a .txt +file. Then, please attach the file to the issue submission.) + + + +*Docker Provisioning Error Details* + +**Deployment machine operating system version** + +(i.e. Ubuntu 16.04 or CentOS 7.3) + +**Docker version** + +``` +(paste the results of the 'docker version' command on the deployment +machine) +``` + +**Openstudio Server version / dockerhub tag used** + +(i.e. SHA abcd1234 or nrel/openstudio-server:latest) + +**docker-compose.yml file used** + +(Please attach the docker-compose.yml file being used, if applicable) + +