Temporary import sites for semi-automated Open Doors imports.
The Open Doors project is dedicated to preserving at-risk fanworks. Part of that work involves importing works into Archive of Our Own from external archives which are being closed down.
Due to the variety of formats in external websites, some of which are completely incompatible with Archive of Our Own's importing process, some archives are imported using a semi-automated process involving two technical stages after the initial Open Doors processes:
- The metadata and works are extracted from a backup of the old archive and converted into standardised MySQL tables (see Open Doors eFiction and Open Doors Code (ODAP) which are used for this stage)
- The standardised data is hosted in a temporary website accessible to import operators and which allows them to perform the import in stages, verifying imported works as they go along.
This repository houses the Rails + React application used to create those sites.
This is a Ruby on Rails site with a React front-end mounted by the react-rails
gem.
Requirements:
- Ruby 2.7.3
- MySQL 5.7 or MariaDB
- Bundler
- Node 14
- Yarn https://yarnpkg.com/lang/en/docs/install/
Note: for ease of local development, the Ruby and MySQL versions should be kept in step with the otwarchive project
See Rails Getting Started Guide for instructions on installing and configuring Rails.
Once this is done, you can install dependencies and run the Rails application from the command line:
$ cd path/to/this/repo
$ bundle install
$ bin/rails server
In your browser, navigate to http://localhost:3010/opendoorstempsite to view the temp site.
There is also an option to set up the local environment using Docker.
- On Mac, install Docker Desktop, which includes Docker Composer. On Linux, install
docker
anddocker-compose
. - On the command line, navigate to the root of this repository.
- If necessary, set permissions on the init file:
chmod +x scripts/docker/init.sh
- Run the init file to create the containers and run them. This will prompt for the MySQL password to use.
./scripts/docker/init.sh
- Follow any instructions on screen. Once the script has finished running, navigate to http://localhost:3010/opendoorstempsite to view the temp site.
- Refer to the Windows Subsystem for Linux (WSL 2) documentation to set up a development environment, specifically the sections on installing and setting up WSL 2, Visual Studio Code, Git and Docker Desktop. All of the development should be done from within the WSL distro you install.
- In WSL, navigate to the root of the repo and run the script that initializes everything.
sudo bash scripts/docker/init.sh
- The script will first prompt you to set the MySQL password, and will update docker-compose.yml and config/secrets.yml as long as the files have "change_me" where the password should be. Then the script builds the image with all the Ruby gems, node modules, etc. needed for development and starts the containers and volumes, so it will take a while for this script to finish running. At the very end it will populate the MySQL database with sample data. To test with real data, put the SQL dump file somewhere inside the repo (so the container can access it) and use the last command in the script, replacing the variables accordingly. Once the script has finished running, navigate to http://localhost:3010/opendoorstempsite to view the temp site.
Before you proceed, you will need to install Ansible Playbook (https://docs.ansible.com/ansible/latest/network/getting_started/first_playbook.html).
-
In the
scripts/ansible
directory, create a file calledhosts
with the following contents:[otw] [[SERVER_NAME]] ansible_ssh_user=[[SERVER_USER]]
Where
SERVER_NAME
andSERVER_USER
are the server DNS host and ssh user for the web server. You can optionally include the local path to your private SSH key usingansible_ssh_private_key_file
:[otw] server.transformativeworks.org ansible_ssh_user=username ansible_ssh_private_key_file=/path/to/.ssh/something_rsa
-
Make a copy of
scripts/variables.yml.example
asscripts/variables.yml
and fill it in with the details of the site you're creating. Thesitekey
must be a string with no spaces: this will be used as the installation directory, database name and site path. Note: a filled-invariables.yml
for use with the live temp site server is available for OTW staff. -
Make a copy of
config/secrets.yml.example
asconfig/secrets.yml
(you'll probably need to have one for development anyway - make sure it has a validproduction
section). Note:secrets.yml
is included in the.gitignore
to prevent it being accidentally uploaded to Github. -
Run
$ cd open-doors-temp-site/ $ ./scripts/deploy-site-from-variables.sh
Note: the bash script is just provided as a convenience to load parameters from variables.yml
. If you are working on
multiple sites, you might want to have different variable files and an edited copy of the bash script for each one.
This will create a shell site populated with dummy data. You can then upload the real tables produced by the ODAP
directly to the MySQL database on the temp site, which will have the sitekey
you specified as its name.
When you are finished with the site and it has been fully imported, you can archive it using the following procedure:
- Edit your
scripts/variables.yml
to include the details for your site. - Run
$ cd open-doors-temp-site/ $ ./scripts/remove-site-from-variables.sh
This will bundle the MySQL tables and app directory into a zip file and download it to the root of the repository on your local machine.
To test the Ansible provisioning and deployment using Vagrant:
-
Make a copy of
variables.yml.example
asvariables.yml
and fill it in with the details of the site you're creating. -
Navigate to the root of the project and then type:
$ cd <root of this repo>
$ vagrant up
- Navigate to http://localhost:8080/ to view the deployed site.
The same provisioning script used to set up the remote server will be used to provision a local Vagrant image on Ubuntu 16.04. (Note that this deploys a production environment which isn't suitable for development)
To deploy a site manually:
$ ansible-playbook scripts/ansible/deploy-site.yml -u ubuntu -i .vagrant/provisioners/ansible/inventory/vagrant_ansible_inventory --extra-vars "@scripts/variables.yml"
To set up a new server from scratch on Ubuntu 16, including fresh installations of Nginx, MySQL and Rails, run the
provision-server.yml
playbook, using a populated variables.yml
:
$ cd <root of this repo>
$ ansible-playbook scripts/provision-server.yml -i scripts/hosts --extra-vars "@scripts/variables.yml"
If you have set use_ssl
to true
in your variables, you will need to install a PEM certificate file and key at /etc/nginx/od-import.crt
and /etc/nginx/od-import.key
respectively.
- Webpacker compilation fails with no explanation on Linode: this is probably due to lack of memory for the compilation - stop one of the other sites to resolve this.