This README file contains instructions for installing a private copy of OneZoom, either the tree explorer or the entire website.
Details of how to customize the OneZoom javascript viewer, along with information about the OneZoom APIs, are available by following the instructions below, then opening the compiled markdown file (at OZprivate/rawJS/OZTreeModule/docs/_compiled.markdown or if you are running your own private OneZoom web server, at /dev/DOCS - for example, https://127.0.0.1:8000/dev/DOCS).
If you simply want to run a local copy of OneZoom, but not modify the code yourself, we recommend using our Docker image. The rest of this README provides details for compiling and running a OneZoom instance (something that is done under the hood when creating the docker image).
If you are using Visual Studio Code or another editor that supports Dev Containers, the easiest way to set up a full development environment is to use the included Dev Container configuration. This will automatically create two containers: one for development (dev), and another (web) with the MySQL database and production web server (nginx + uwsgi), derived from the Docker image mentioned above. Your source code will be mounted simultaneously into both containers: dev will let you run build commands and web will serve it as a web server you can access via port forwarding. You can also run your own server from the dev container by running web2py.py directly -- this is particularly useful for debugging.
If you are using Visual Studio Code, perform the following steps (you will need to modify these for another editor):
- Follow the instructions at either https://hub.docker.com/r/onezoom/oztree or https://hub.docker.com/r/onezoom/oztree-complete to save a docker image with IUCN data.
- Follow the VSCode instructions for installing Dev Container support.
git clone https://github.com/OneZoom/OZtreeinto the directory of your choice. If using Windows, it is highly recommended to clone on the WSL2 filesystem both for performance reasons and to avoid permissions issues. If you wish to fork the repository and clone your fork, you will need to copy the tags from upstream, otherwise you will see build issues later. You can do this withgit fetch --tags upstreamfollowed bygit push --tags.- Open the cloned directory in VSCode.
- Create a
.envfile in the.devcontainerdirectory and addWEB_IMAGE_NAME=onezoom/oztree-with-iucn, changing the value to whatever image name you choose in step 1. - Open the command palette and choose "Dev Containers: Reopen in Container". This may take several minutes to run.
- Your repository is mounted at
/opt/web2py/applications/OZtreeand the original production docker container application is mounted at/opt/web2py/applications/OZtree_original. In order to sync your repository with the production database state, open an integrated terminal and run the following:
cp /opt/web2py/applications/OZtree_original/private/appconfig.ini /opt/web2py/applications/OZtree/private/
rm -f /opt/web2py/applications/OZtree/databases/*.table
cp /opt/web2py/applications/OZtree_original/databases/*.table /opt/web2py/applications/OZtree/databases/- Run
npm ci && grunt dev. You will need to rerungrunt devany time you make code changes. - Run
./web2py-runto start the server locally. - Load https://localhost:8000/. The first page load can take a long time (10+ minutes) as the database is updated. Do not kill the server while this is running. Future page loads will be much faster.
- You can now visit http://localhost:8080 at any time to load the page via nginx + uwsgi! You can also continue to rebuild and run the server with
grunt devand./web2py-runand use https://localhost:8000/. - (Optional) Once tables are created, and everything is working, you can set
migrate = 0inprivate/appconfig.ini. This will mean that web2py will not make any changes to table structures in the DB, and also that changes to appconfig.ini will require a web2py restart. - (Optional) Create a manager account in the auth table, e.g. so you can view docs.
- (Optional) MySQL is available on port 3307 if you wish to debug using local tools outside the container.
There are two ways in which you can install OneZoom on a personal computer: full installation and partial installation.
-
Partial installation does not create a standalone OneZoom site, but simply creates a local web file containing the javascript tree viewer. Instead of your tree viewer getting information from your own computer, it must do so by constantly requesting data from the OneZoom website (via the OneZoom APIs). This restricts your OneZoom viewer in various ways: you cannot make your own bespoke tree, you cannot change languages in the viewer, and you are dependent upon a permanent, fast internet connection. Also note that this installation method is also relatively untested, and there are unfixed problems with e.g. displaying lists of popular species. However, partial installation may be suitable for developers who simply want to re-program features of the tree viewer, such as colours, branch geometry, etc.
-
Full installation creates an entire duplicate of the OneZoom website, which is built using the web2py framework. This creates a fully self-contained local system (apart from the picture files, which can be downloaded separately). This is the most reliable installation method, but requires you to install and run extra software packages, in particular web2py and a MySQL server. Since this can be quite complicated, the majority of this readme contains instructions for full installation.
For any installation, OneZoom requires node (18+) & python (3.10).
apt install nodejs npm
apt install python3 python3-dev python3-venv
# NB: rust is required to build the python cryptography package
sudo pkg install lang/python310 node18 npm-node18 rust
In addition, for a full installation you also need MySQL:
apt install lsb-release
wget https://dev.mysql.com/get/mysql-apt-config_0.8.29-1_all.deb
dpkg -i mysql-apt-config_0.8.29-1_all.deb
apt update && apt install mysql-server
On Windows we recommended downloading the MSI installer as it will make it easier to configure the new server during the installation. Once mysql is installed, you will need to set a root password, and create a database for web2py to use. See http://dev.mysql.com/doc/refman/5.7/en/default-privileges.html. The mysqld program is responsible for running the new database just created. When this program is running, you can connect to the database.
If performing a full installation, you need to create a database for OneZoom to use:
mysql -p
CREATE DATABASE OneZoom;
CREATE USER 'oz'@'localhost' IDENTIFIED BY 'passwd-you-should-change-this';
GRANT ALL PRIVILEGES ON OneZoom . * TO 'oz'@'localhost';
Firstly, you need to check out the web2py / OneZoom repositories, and run npm:
# NB: The path should match the eventual hostname for this installation
export WEB2PY_PATH="/srv/.../onezoom.myhostname.org"
mkdir -p ${WEB2PY_PATH}
chown deploy:staff ${WEB2PY_PATH}
git clone https://github.com/web2py/web2py ${WEB2PY_PATH} --branch v2.27.1
git clone https://github.com/OneZoom/OZtree.git ${WEB2PY_PATH}/applications/OZtree --branch production
cd ${WEB2PY_PATH}/applications/OZtree
npm ci
Next, cp private/appconfig.ini.example private/appconfig.ini and edit to match your needs, taking care to:
- Use the same database credentials as configured earlier.
Once done, run ./node_modules/.bin/grunt dev for a development installation.
For production installation, see later.
If you'd like your installation to be a partial installation, run:
./node_modules/.bin/grunt partial-install
...and open static/minlife.html in a web browser.
Note that this file needs to stay within the static directory to work at all.
You may also need to allow your browser to allow local files to be loaded via AJAX (i.e. disabling some local cross-origin checks).
Different browsers do this in different ways: for example in Chrome you can start up your browser with the --allow-file-access-from-files option, and in Safari, you can choose "Disable Local File Restrictions" from the "Developer" menu.
Note that the normal minlife.html file will use local versions of data files and the javascript treeviewer,
but will get API information form the OneZoom website, and also use the OneZoom website as the source for the html page which embeds the viewer.
For developers only, who may wish to create a minlife version not only using modified javascript in the treeviewer but also with bespoke html, you can run grunt partial-local-install.
This is much more effort since it requires you to set up a full installation (as below) before creating the minlife scripts, but once created, the files in static will be enough for other users to view (and test) your modifications.
For full installations, you need to setup your database.
To create required tables, use web2py migrate:
- Edit private/appconfig.ini, setting
migrate=1 ./web2py-run tests/unit/test_modules_embed.py, on startup this will create tables as necessary- Edit private/appconfig.ini, migrate=0
A database dump should be used to add species information:
# NB: OneZoom.dump.sql can be extracted from https://github.com/OneZoom/OZtree-docker
mysql -p
USE OneZoom
SOURCE /OneZoom.dump.sql
Finally, ensure indexes are created:
mysql -p
USE OneZoom
SOURCE ${WEB2PY_PATH}/applications/OZtree/OZprivate/ServerScripts/SQL/create_db_indexes.sql
For a production installation of OneZoom, you also need nginx & supervisor:
apt install nginx supervisor
sudo pkg install nginx py39-supervisor lang/python310
Run grunt to configure onezoom for production use:
cd ${WEB2PY_PATH}/applications/OZtree
npm ci
./node_modules/.bin/grunt prod
Edit models/db.py, and set is_testing = False.
Then run the install scripts to set up nginx & supervisord:
sudo ./install-nginx.sh
sudo ./install-supervisord.sh
If everything works, restart both.
(optional) We find it useful to have a GUI interface to connect to the database and run SQL scripts, this can be used instead of using MySQL command line (similar to Windows command line) that is installed by default with MySQL. On Mac OS X we use the (excellent) https://sequel-ace.com. On windows you could try http://www.mysql.com/products/workbench/ or https://www.quest.com/products/toad-for-mysql/ Under windows, SQL Workbench can also be used, even if your MySQL server is installed under WSL2.
installing SQL Workbench on Windows works great to connect to the Ubuntu MySQL instance.
On the OneZoom main site, web2py is run using a combination of nginx and uwsgi. This is complete overkill if you just want to run a local copy of OneZoom for testing purposes. You can simply run a temporary and basic web2py server using Python 3. The simplest is to open a command-line prompt in the root web2py folder, and run the following (assuming the command python3 is linked to something like Python 3.7)
./web2py-run
When web2py is run, it will print instructions telling how to shut down the web2py server. For example, on Windows you might use taskkill /f /pid XXXX, where XXXX is the process id.
databases stores all the database structure.
controllers is where most of the bespoke web2py code that runs the site lives. The public pages are in controllers/default.py.
models stores the python back end server code.
static stores all static files including images, css, and compiled js. Files which are output by various server processes are stored in FinalOutputs. This includes very large numbers of thumbnail images (stored in FinalOutputs/img) and static data files such as the tree topology and the tree cut positions (stored in FinalOutputs/data). The OZTreeModule folder contains the compiled version of most of the core OneZoom code. static/OZLegacy contains most of the old trees.
views is where all the html is stored - it's OK to just use raw html in here if no server side functions are needed for that particular page.
OZprivate stores external files that are not formally part of web2py, such as the tree viewer code itself, and scripts which we use for updating the tree and our own database tables. The most important are:
data, which contains most of the data used to build the tree (e.g. EoL mappings, OpenTree components, Yan's specific tree-building coderawJS, which contains the uncompiled javascript that when compiled, creates the OneZoom viewerServerScripts, which contains scripts that the server can run to compile a tree, grab images from EoL, percolate images throughout the tree, etc.
Web2py uses an auth_ based system, which has tables for users, roles, and a mapping table assigning roles to users. This can be edited through the web interface: assuming you are running a temporary version of web2py on localhost, you can access the admin pages through http://127.0.0.1:8000/admin/default/design/OZtree, which will require you to enter the temporary administrator password ('pass', above) that you used in the web2py startup command. The database tables can be seen at the url http://127.0.0.1:8000/OZtree/appadmin. You need to click to edit the db.auth_user table, from where you can click to add a "New Record", and submit a first name, last name, email, username, and password. You then need to go back to the appadmin page and create a "manager" role by adding a New Record to db.auth_group table (you can type anything in the description box). Finally, you need to create a New Record in the db.auth_membership table, and assign the "manager" group ID to your user ID in the resulting page.
The main bulk of the data returned from the API is stored in the rest of the tables in the database, as detailed below. To get the API and the rest of the website working, you will have to obtain a database dump of the OneZoom tables. You can either do this by emailing mail@onezoom.org, or getting it from a Docker setup of OZTree. If you are loading new data on top of old, it is a good idea to truncate all the non-auth tables before loading data.
You can use the following steps to fill up the tables based on a Docker setup. Start by following the Docker instructions, including the 'Saving the image' section. Then, run these commands one by one:
# start by stopping/deleting any OZTree container instance that may still be running (your container name may be different):
docker stop oztree
# Create a folder on your local machine to hold the dump
mkdir ~/dbdump
# Run a new OZTree Docker container instance, mounting your local folder
docker run -d -p 8080:80 --name oztree --mount type=bind,source="$(pwd)"/dbdump,target=/dbdump onezoom/oztree-complete-with-iucn
# Start a shell in the container
docker exec -it oztree /bin/bash
# Dump the container's database into your mounted local folder. This is fairly fast. The resulting file is ~1GB.
mysqldump OneZoom > /dbdump/ozdb.sql
# Exit the running OZTree container. We're done with it now, so feel free to delete it.
exit
# Now import the dump into your local OneZoom database (you may need to pass -u & -p based on your MySql configuration)
# WARNING: this is an extremely long step that can take over an hour with no indication of progress!
sudo mysql OneZoom < ~/dbdump/ozdb.sql
# You're done, and your local database should be fully populated. Feel free to delete the dump file /dbdump/ozdb.sqlNote that mySQL stupidly has a restricted version of the unicode character set, so fields that could contain e.g. chinese characters need to be set to utf8mb4 (which is not the default). These are the vernacular field in the vernacular_by_ott and vernacular_by_name tables, the rights field in the images_by_ott and images_by_name tables, and the following fields in the reservations table: e_mail, twitter_name, user_sponsor_name, user_donor_name user_more_info, user_message_OZ, verified_sponsor_name, verified_donor_name verified_more_info. When we send you the tables, they should contain create syntax which makes sure the tables are correctly defined, but it may be worth checking too.
To get any decent performance out of your OneZoom instance, you will need to create indexes on the resulting tables. The commands for doing this are listed in OZtree/OZprivate/ServerScripts/SQL/create_db_indexes.sql, from where they can be copied and pasted into a mysql client.
The commands to create indices also include commands to convert some of the columns to 4-byte unicode if necessary (to incorporate e.g. full Japanese/Chinese common names), and to drop the indexes if they already exist. Some of these commands may cause SQL errors (e.g. "Can't DROP XXX") if you have not previously created any indices. These errors can be safely ignored. If you are using mysql workbench you may want to untick the option under Query to "Stop Script Execution on Errors", so that the index creation continues after each error.
OneZoom data list - this is data that's not stored anywhere else outside OneZoom and so should be treated with greater care
- Auth_* (see above: ours but easy to recreate)
- Banned is ours and is important but could be recreated
- eol_inspected is ours but is not important at all so could be lost and we wouldn't care
- eol_updated is ours but is not critical
- images_by_name and images_by_ott - entries put in by us where src=1 are ours and are semi critical because they include things like special images of sponsors etc. also includes hacked ratings and hacked pictures in general.
- IUCN - not ours at all
- leaves_in_unsponsored_tree - now not used any more can be deleted
- Ordered leaves and ordered nodes - can be recreated, but include derived products like popularity, matched IDs and Yan's curation of the tree. This can be regenerated any time provided our codebase and algorithms are fine
- PoWo - Kew list of things (for later in Kew tree)
- prices is ours semi critical we can recreate but includes our subjective choices.
- Reservations table THE MOST critical
- vernacular_by_name and vernacular_by_ott is the same as images - so src=1 means it's ours as before semi critical
- visit_count medium level of criticality it's our visit information, but we don't know if it's running. We may later split this up but tree. at the moment it's not really functioning as we want it.
Notes
- there is a question mark over tours information and associated things
- there are many critical source files in OZ_private, including tree sources.
NB: The server tests are not sandboxed, so have the potential to delete database data. You are advided to run this on a personal instance where data loss does not matter.
The server unit tests have no additional dependencies. To run, do:
grunt test-server
To run individual tests, do:
grunt exec:test_server:test_modules_username.py
The client unit tests have no additional dependencies. To run, do:
grunt test-client
Make sure required python modules are installed with:
pip3 install -r tests/requirements.txt
You also need chrome/chromium driver installed with, e.g.:
apt install chromium-driver
You can then run the tests with:
grunt test-server-functional
If you wish to use your editor's debugger to debug the JavaScript code, you will need to compile source maps. Run the following:
- Run
grunt devto perform the initial dev compilation. - Run
npm run compile_js_dev:watchto compile source maps and automatically recompile if files are changed. Note: this command will continue to run until it is killed.
Documentation is partially compiled from the source code using Grunt, and lives in OZprivate/rawJS/OZTreeModule/docs. Once compiled, it can be viewed online using your web2py server. For example, if you are running web2py on http://127.0.0.1:8000, you should be able to visit http://127.0.0.1:8000/OZtree/dev/DOCS, or (if you have manager access to the OneZoom site) at http://onezoom.org/dev/DOCS.
If you wish to make sure your OneZoom data is up-to-date, there are a few steps you can take:
-
Make sure the "representative species" for each node (the 4x2 pictures in the middle of each internal node) are up-to-date. To do this, run
OZprivate/ServerScripts/Utilities/picProcess.pyNote that these representative pictures can be customized too (see below)
-
Update IUCN statuses, from the IUCN API. This can be done by
OZprivate/ServerScripts/Utilities/IUCNquery.py -
Keep a running script that mines data from the Encyclopedia of Life (EoL). This will ensure that new images on EoL are eventually downloaded to OneZoom, but it does mean that your server will continuously be sending online requests to EoL. You wll need to obtain an EoL API key (http://eol.org/info/api_overview) and add it into your appconfig.ini file. Then you can run the script as follows:
OZprivate/ServerScripts/Utilities/EoLQueryPicsNames.py -
Recompile your own tree and database tables. Instructions for creating your own tree are in OZprivate/ServerScripts/TreeBuild/README.markdown.
A few suggestions about ways to customize OneZoom
- Change the colour schemes: the code in
rawJS/OZTreeModule/src/themesgives examples of how to create new colour schemes for branches, leaves, etc of the tree. They can be gathered together into themes (including different leaf styles, default fractal views etc) by adding to the code inrawJS/OZTreeModule/src/tree_settings.js.
-
Customizing pictures: by running alternative versions of
picProcess.py, you can choose different pictures to represent taxonomic groups. This is left as an exercise to the customiser. -
Alternative UI layer: the current OneZoom viewer uses the UIkit framework to layer a user interface on top of the viewing canvas. The UI can send commands to the viewing canvas, and the canvas is instatiated with UI callbacks, so it is quite possible to write alternative user interfaces (e.g. using different UI toolkits)
-
Alternative projections: the code in
rawJS/OZTreeModule/src/projectioncan be supplemented with alternative "projections" of the tree, such as treemaps, other fractal layouts, etc.