TeMMPo (Text Mining for Mechanism Prioritisation) is a web-based tool to enable researchers to identify the quantity of published evidence for specific mechanisms between an exposure and outcome. The tool identifies co-occurrence of MeSH headings in scientific publications to indicate papers that link an intermediate mechanism to either an exposure or an outcome. TeMMPo is particularly useful when a specific lifestyle or dietary exposure is known to associate with a disease outcome, but little is known about the underlying mechanisms. Understanding these mechanisms may help develop interventions, sub-classify disease or establish evidence for causality. TeMMPo quantifies the body of published literature to establish which mechanisms have been researched the most, enabling these mechanisms to be subjected to systematic review.
- Vagrant https://www.vagrantup.com/
- VirtualBox https://www.virtualbox.org/ or another provider, see https://www.vagrantup.com/docs/providers/ NB: The vagrant installation also requires an additional plugin to mount the development source code cloned on your local machine.
Tested with these versions:
- VirtualBox 6.1.32 r149290 (Qt5.6.3)
- Vagrant 2.2.19
- vagrant-sshfs 1.3.6
NB: Additional development IDE support for Visual Code can be added by installing additional packages within your development environment
cd /usr/local/projects/temmpo/lib/dev/bin
sudo pip3 install pylint==2.7.4
vagrant plugin install vagrant-sshfs
git clone git@github.com:MRCIEU/temmpo.git
Use one of the techniques below to set up your virtual environment and create your Django application. Various options exist. For example set up with Apache proxying and that by default run database migrations.
cd temmpo/deploy
vagrant up && vagrant ssh
fab make_virtualenv:env=dev,configure_apache=False,clone_repo=False,branch=None,migrate_db=True,use_local_mode=True,requirements=dev -f /usr/local/projects/temmpo/lib/dev/src/temmpo/deploy/fabfile.py
cd temmpo/deploy
vagrant up && fab make_virtualenv:env=dev,configure_apache=False,clone_repo=False,branch=None,migrate_db=True,use_local_mode=False,requirements=dev -u vagrant -i ~/.vagrant.d/insecure_private_key -H 127.0.0.1:2200 && vagrant ssh
c. Installing a Vagrant Apache fronted virtual environment not mounted to your local development drive
cd temmpo/deploy
vagrant up db && vagrant up apache && vagrant ssh apache
fab make_virtualenv:env=dev,configure_apache=True,clone_repo=True,branch=master,migrate_db=True,use_local_mode=True,requirements=dev -f /vagrant/deploy/fabfile.py
d. Installing a Vagrant Apache fronted virtual environment not mounted to your local development drive using a remotely run Fabric command
cd temmpo/deploy
vagrant up db && vagrant up apache && fab make_virtualenv:env=dev,configure_apache=True,clone_repo=True,branch=master,migrate_db=True,use_local_mode=False,requirements=dev -u vagrant -i ~/.vagrant.d/insecure_private_key -H 127.0.0.1:2200 && vagrant ssh apache
cd /usr/local/projects/temmpo/lib/dev/bin && source activate && cd /usr/local/projects/temmpo/lib/dev/src/temmpo
python manage.py createsuperuser --settings=temmpo.settings.dev
To be able to run the applications browsing and searching functionality Mesh Terms will need to be imported, either by using fixtures or the custom management command.
- Load fixture data
NB: this can take a few minutes.
python manage.py loaddata browser/fixtures/mesh_terms_2015_2018_2019_2020.json --settings=temmpo.settings.dev
-
Management command
Annually MeSH terms are released. This can be as early as November for the following year. There is a management command that can be run annually once the new terms have been sourced. Reference: ftp://nlmpubs.nlm.nih.gov/online/mesh/MESH_FILES/meshtrees/ or see newer location: ftp://nlmpubs.nlm.nih.gov/online/mesh/MESH_FILES/meshtrees/mtrees2021.bin
NB: This command each take over 50 minutes to run depending on your environment.
python manage.py import_mesh_terms ./temmpo/prepopulate/mtrees2021.bin 2021
After importing a new year of mesh terms, create a fixture file for testing and development purposes. For example:
python manage.py dumpdata browser.MeshTerm --indent 4 --output browser/fixtures/mesh_terms_2015_2018_2019_2020_2021.json
A database of existing gene terms can be imported into the Django application database, either by using fixtures or the slower custom management command.
-
Load fixture data
NB: This can take a few minutes.
python manage.py loaddata browser/fixtures/genes_snap_shot_2020_06_29.json --settings=temmpo.settings.dev
-
Management command
A sample set is stored and loaded from this GENE_FILE_LOCATION setting location.
python manage.py import_genes --settings=temmpo.settings.dev
In development you will need to restart the worker whenever any changes to the matching code are made. Run the following in a separate window and restart to see changes to the mathcing code.
fab restart_rqworker_service:use_local_mode=True -f /usr/local/projects/temmpo/lib/dev/src/temmpo/deploy/fabfile.py
In a separate terminal window run the development server
python manage.py runserver 0.0.0.0:59099 --settings=temmpo.settings.dev
http://localhost:59099
http://localhost:8800
NB: If you want to manually run migrations you need to use the --database flag
python manage.py migrate --database=admin --settings=temmpo.settings.dev
NB: This can take a while as we also generate hashes for additional security.
fab pip_sync_requirements_file:env=dev,use_local_mode=True -f /usr/local/projects/temmpo/lib/dev/src/temmpo/deploy/fabfile.py
NB: This can take a while as we also generate hashes for additional security.
Optionally pass in a package or update them all within any requirements.in file constraints
fab pip_tools_update_requirements:env=dev,use_local_mode=True,package="" -f /usr/local/projects/temmpo/lib/dev/src/temmpo/deploy/fabfile.py
Alternatively
docker build -f deploy/Dockerfile -t temmpo-web .
docker run --rm -it -v $PWD:/srv -w /srv temmpo-web bash /srv/entrypoints/update-requirements.sh
docker build -f deploy/Dockerfile -t temmpo-web-dev . --build-arg REQUIREMENTS_FILE=dev.txt
docker build -f deploy/Dockerfile -t temmpo-web-test . --build-arg REQUIREMENTS_FILE=test.txt
example dev command
docker run --rm -it -v $PWD:/srv -w /srv temmpo-web-dev bash /srv/entrypoints/django-upgrade.sh
fab deploy:env=dev,branch=master,using_apache=True,migrate_db=True,use_local_mode=False,use_pip_sync=True,requirements=base -u vagrant -i ~/.vagrant.d/insecure_private_key -H 127.0.0.1:2200
fab deploy:env=dev,branch=demo_stable,using_apache=True,migrate_db=True,use_local_mode=False,use_pip_sync=True,requirements=base -u vagrant -i ~/.vagrant.d/insecure_private_key -H 127.0.0.1:2200
fab deploy:env=dev,branch=prod_stable,using_apache=True,migrate_db=True,use_local_mode=False,use_pip_sync=True,requirements=base -u vagrant -i ~/.vagrant.d/insecure_private_key -H 127.0.0.1:2200
Entire test suite
cd /usr/local/projects/temmpo/lib/dev/bin && source activate && cd /usr/local/projects/temmpo/lib/dev/src/temmpo
python manage.py test --settings=temmpo.settings.test_mysql
Run the entire test suite using MySQL and generate a coverage report.
coverage run --source='.' manage.py test --settings=temmpo.settings.test_mysql && coverage report --skip-empty --skip-covered -m
e.g. Just the searching related tests and fail at the first error
python manage.py test tests.test_searching --settings=temmpo.settings.test_mysql --failfast
e.g. Skipping slow tests
python manage.py test --settings=temmpo.settings.test_mysql --exclude-tag=slow
e.g. Skipping selenium and clamav tests
python manage.py test --settings=temmpo.settings.test_mysql --exclude-tag=selenium-test --exclude-tag=clamav
e.g. Run selenium tests only
python manage.py test --settings=temmpo.settings.test_mysql --tag=selenium-test
Using a locally instally node environment
NB: Some tests require these environment variables CREDENTIALS_USR
and CREDENTIALS_PSW
to be defined, to be able log into the site being tested. These details can be found in Research IT's LastPass account. Based on the cypress.example.env.json
create a cypress.env.json
file and fill in the details required from LastPass.
npx cypress open
Using docker and electron browser
docker run --rm -it -v $PWD:/e2e -w /e2e cypress/included:13.13.3
IntegrityError at /search/ovidmedline/
(1048, "Column 'mesh_terms_year_of_release' cannot be null")
This suggests attempting to create a search when no mesh terms have been imported into the database as yet.
The project needs the following additional services to be running:
sudo systemctl status redis
sudo systemctl status rqworker1
sudo systemctl status rqworker2
sudo systemctl status rqworker3
sudo systemctl status rqworker4
sudo systemctl status httpd # Not relevant for the django Vagrant VM
sudo systemctl status
- 2015, 2018, & 2019 MeSH® - Medical Subject Headings terms provided by U.S. National Library of Medicine
- Apache - Web server/proxy
- Centos - Operating System
- D3 Data-Driven Documents - Visualization tools
- DataTables - Dynamic UI
- Django - Web framework
- Google Charts - Visualization tools
- Jenkins - Continuous Integration/Deployment
- jsTree - Dynamic UI
- MEDLINE® - Export format of the National Library of Medicine® (NLM®) journal citation database
- MySQL - Database server
- Puppet - Configuration management
- Python - Programming language
- Redis - Message queue
- SB Admin 2 - Web design
For the versions available, see the CHANGELOG and the tags on this repository.
- Tom Gaunt - Initial code - MRC Integrative Epidemiology Unit
- Ben Elsworth - Code contributions - MRC Integrative Epidemiology Unit
- Tessa Alexander - Developer - Research IT, University of Bristol
- Kieren Pitts - Developer cover - Research IT, University of Bristol
- Jon Hallett - Systems Administrator - Research IT, University of Bristol
- Mike Rodwell - Acceptance testing - Research IT, University of Bristol
- Rick Henry - Systems Administrator - Research IT, University of Bristol
This project is licensed under the GNU General Public License v3.0 - see the LICENSE file for details
- Funded by World Cancer Research Fund UK
- Funded by UK Medical Research Council (MRC)
- Conceived by the MRC Integrative Epidemiology Unit, University of Bristol
- Packaged and developed by Research IT, University of Bristol
- Hosting infrastructure provided by IT Services, University of Bristol