This document describes setting up a development environment for working on CommCare HQ. Such an environment is not suitable for real projects. Production environments should be deployed and managed using commcare-cloud.
These instructions are for Mac or Linux computers. For Windows, consider using an Ubuntu virtual machine.
Once your environment is up and running, bookmark the dev FAQ for common issues encountered in day-to-day HQ development.
If you're setting up HQ on a new computer, you may have an old, functional environment around. If you don't want to start from scratch, back up your Postgres and Couch data.
-
PostgreSQL
-
Create a pg dump. You'll need to verify the host IP address:
pg_dump -h localhost -U commcarehq commcarehq > /path/to/backup_hq_db.sql
-
-
CouchDB
- From a non-Docker install: Copy
/var/lib/couchdb2/
. - From a Docker install: Copy
~/.local/share/dockerhq/couchdb2
.
- From a non-Docker install: Copy
-
Shared Directory
- If you are following the default instructions, copy the
sharedfiles
directory from the HQ root folder, otherwise copy the directory referenced bySHARED_DRIVE_ROOT
inlocalsettings.py
- If you are following the default instructions, copy the
Save those backups to somewhere you'll be able to access from the new environment.
NOTE: Developers on Mac OS have additional prerequisites. See the Supplementary Guide for Developers on Mac OS.
-
sudo apt install git
-
-
Linux:
In Ubuntu you will also need to install the modules for
python-dev
,pip
, andvenv
explicitly.The deadsnakes PPA allows you to use multiple versions of Python on your own machine as we do in production environments. The deadsnakes PPA supports Ubuntu LTS releases, but you can use their Python versions on interim Ubuntu releases as follows:
-
Find the name of your Ubuntu release if you don't already know it:
$ cat /etc/lsb-release DISTRIB_ID=Ubuntu DISTRIB_RELEASE=23.04 DISTRIB_CODENAME=lunar # <-- This is the name you want DISTRIB_DESCRIPTION="Ubuntu 23.04"
-
Pin your release's package priority to 1001 so that deadsnakes packages can't replace official packages, even if they are newer:
$ cat << EOF | sudo tee /etc/apt/preferences.d/99lunar Package: * Pin: release lunar Pin-Priority: 1001 EOF
but change "lunar" to the name of the release you are using.
-
Add the deadsnakes PPA, and install the Python version that CommCare HQ requires:
$ sudo add-apt-repository ppa:deadsnakes/ppa
This will create a file in
/etc/apt/sources.list.d/
with the name of your release. Change the filename, and the name inside the file, to the latest LTS release instead (e.g. "jammy"). -
Install the version of Python that CommCare HQ requires.
$ sudo apt update $ sudo apt install python3.9 python3.9-dev python3-pip python3.9-venv
-
-
Mac:
Mac OS 12.x still comes shipped with Python 2.7 (?!), so you need to explicitly use
python3
instead ofpython
(unless you usepyenv
—which we highly recommend!). First install homebrewbrew install python@3.9
-
-
Requirements of Python libraries, if they aren't already installed. Some of this comes from pyenv's recommendations
-
Linux:
sudo apt-get update; sudo apt install libncurses-dev libxml2-dev libxmlsec1-dev \ libxmlsec1-openssl libxslt1-dev libpq-dev pkg-config gettext make build-essential \ libssl-dev zlib1g-dev libbz2-dev libreadline-dev libsqlite3-dev wget curl llvm \ libncursesw5-dev xz-utils tk-dev libffi-dev liblzma-dev
-
macOS:
brew install libmagic libxmlsec1 libxml2 libxslt openssl readline sqlite3 xz zlib tcl-tk
-
-
Java (JDK 17)
-
Linux:
install
openjdk-17-jre
via apt:sudo apt install openjdk-17-jre
-
macOS: See the Supplementary Guide for Developers on Mac OS as this is quite lengthy.
-
-
PostgreSQL
Executing postgres commands (e.g.
psql
,createdb
,pg_dump
, etc) requires installing postgres. These commands are not explicitly necessary, but having the ability to run them may be useful.-
Linux (optional) install the
postgresql-client
package:sudo apt install postgresql-client
-
macOS (required) the postgres binaries can be installed via Homebrew:
brew install postgresql
Possible alternative to installing postgres (from this SO answer). Prior to
pip install
commands (outlined later in this doc):xcode-select --install export LDFLAGS="-I/usr/local/opt/openssl/include -L/usr/local/opt/openssl/lib"
If you have an M1 chip and are using a Rosetta-based install of Postgres and run into problems with psycopg2, see this solution.
-
xmlsec
is a pip
dependency that will require some non-pip
-installable
packages. The above notes should have covered these requirements for linux and
macOS, but if you are on a different platform or still experiencing issues,
please see xmlsec
's install notes.
If you encounter issues installing xmlsec
on a M1 mac, you can try following a workaround
outlined in the Mac setup Supplementary Guide.
- Install
pyenv
Full installation instructions are here and here. Check here and here to troubleshoot.
-
Linux:
curl -L https://github.com/pyenv/pyenv-installer/raw/master/bin/pyenv-installer | bash exec $SHELL
-
macOS:
-
Install python 3.9 with
pyenv
:pyenv install 3.9:latest
To set Python 3.9 as the global
python
, run:pyenv global 3.9.xx # or whatever version was just installed - it should tab complete
Pro-tip: this is great for Mac OS users working around having to explicitly use
python3
:)
- Create the virtualenv
hq
with Python 3.9.xx:Then to enter the environment:pyenv virtualenv 3.9.xx hq
That's it! You may now proceed to Step 2.pyenv activate hq
-
Set the
WORKON_HOME
environment variable to the path where you keep your virtual environments. If you don't already have a home for your virtual environments, ~/venv is not a bad choice:export WORKON_HOME=$HOME/venv mkdir -p $WORKON_HOME
-
Create a virtual environment for CommCare HQ. "commcare-hq" is a good name, but naming it "hq" might save you some typing in the future:
python3 -m venv $WORKON_HOME/hq
-
Ubuntu no longer ships with Python 2 and its Python binary is named "python3" to avoid ambiguity. You may need to tell virtualenvwrapper where to find Python:
export VIRTUALENVWRAPPER_PYTHON=/usr/bin/python3
-
Enable virtualenvwrapper:
source /usr/local/bin/virtualenvwrapper.sh
-
You will want to add virtualenvwrapper settings to your startup script, say, ~/.bashrc, or ~/.zshrc. For example:
$ cat <<EOF >> ~/.bashrc export WORKON_HOME=~/venv export VIRTUALENVWRAPPER_PYTHON=/usr/bin/python3 source /usr/local/bin/virtualenvwrapper.sh EOF
-
Activate your virtual environment:
workon hq
-
Ensure your vitualenv
pip
is up-to-date:python3 -m pip install --upgrade pip
Virtual environments were introduced as a standard in Python 3.3 with PEP 405.
By convention, virtual environments use a ".venv" or "venv" directory in the root of the codebase. Once you have cloned the CommCare HQ repo in "Step 2" below, create a Python 3.9 virtual environment in the root of the codebase with:
$ python3.9 -m venv .venv
For convenience, you can create an alias to activate virtual environments in
".venv" and "venv" directories. To do that, add the following to your
.bashrc
or .zshrc
file:
alias venv='if [[ -d .venv ]] ; then source .venv/bin/activate ; elif [[ -d venv ]] ; then source venv/bin/activate ; fi'
Then you can activate virtual environments with
$ venv
-
Once all the dependencies are in order, please do the following:
git clone https://github.com/dimagi/commcare-hq.git cd commcare-hq git submodule update --init --recursive git-hooks/install.sh setvirtualenvproject # optional, virtualenvwrapper only - sets this directory as the project root
-
Next, install the appropriate requirements (only one is necessary).
NOTE: If this fails you may need to install the prerequisite system dependencies.
Mac OS: Issues? See the Supplementary Guide for Developers on Mac OS.
-
Recommended for those developing CommCare HQ
pip install -r requirements/dev-requirements.txt
- Recommended for developers or others with custom requirements. Use this
pip install ...
workflow for initial setup only. Then create a copy of local.in.sample,and follow the instructions incp requirements/local.in.sample requirements/local.in
local.in
to keep requirements in sync.
If you have problems installing pip dependencies related to a missing wheel package, try installing wheel and upgrade pip before attempting to install dependencies.
- If you have ARM64 architecture (Apple M1 chip) and you're having trouble installing ReportLab:
Source
CFLAGS="-Wno-error=implicit-function-declaration" pip install -r requirements/local.in
- Recommended for developers or others with custom requirements. Use this
Note that once you're up and running, you'll want to periodically re-run these
steps, and a few others, to keep your environment up to date. Some developers
have found it helpful to automate these tasks. For pulling code, instead of git pull
,
you can run this script
to update all code, including submodules. This script
will update all code and do a few more tasks like run migrations and update
libraries, so it's good to run once a month or so, or when you pull code and
then immediately hit an error.
First create your localsettings.py
file:
cp localsettings.example.py localsettings.py
If you have not modified SHARED_DRIVE_ROOT
, then run:
mkdir sharedfiles
Once you have completed the above steps, you can use Docker to build and run all of the service containers. There are detailed instructions for setting up Docker in the docker folder. But the following should cover the needs of most developers.
- Install docker packages.
-
Mac: see Install Docker Desktop on Mac for docker installation and setup.
-
Linux:
# install docker sudo apt install docker.io # ensure docker is running systemctl is-active docker || sudo systemctl start docker # add your user to the `docker` group sudo adduser $USER docker # log in as yourself again to activate membership of the "docker" group su - $USER # re-activate your virtualenv (with your venv tool of choice) # (virtualenvwrapper) workon hq # or (pyenv) pyenv activate hq # or (virtualenv) source $WORKON_HOME/hq/bin/activate
- Install
docker compose
- Mac: comes with Docker Desktop
- Linux:
sudo apt install docker-compose-plugin
-
Ensure the elasticsearch config files are world-readable (their containers will fail to start otherwise).
chmod 0644 ./docker/files/elasticsearch*.yml
-
Bring up the docker containers.
In either of the following commands, omit the
-d
option to keep the containers attached in the foreground../scripts/docker up -d # Optionally, start only specific containers. ./scripts/docker up -d postgres couch redis elasticsearch5 zookeeper kafka minio formplayer
Mac OS: Note that you will encounter many issues at this stage. We recommend visiting the Docker section in the Supplementary Guide.
-
If you are planning on running Formplayer from a binary or source, stop the formplayer container to avoid port collisions.
./scripts/docker stop formplayer
If you previously created backups of another HQ install's data, you can now copy that to the new install. If not, proceed to Step 5B.
-
Postgres
-
Make sure Postgres is running:
./scripts/docker ps
-
Make sure
psql
is installed: (Ubuntu)sudo apt install postgresql postgresql-contrib
-
Restore the backup:
psql -U commcarehq -h localhost commcarehq < /path/to/backup_hq_db.sql
-
-
CouchDB
-
Stop Couch:
./scripts/docker stop couch
-
Copy the
couchdb2/
dir to~/.local/share/dockerhq/couchdb2
. -
Start Couch
./scripts/docker start couch
-
Fire up Fauxton to check that the dbs are there: http://localhost:5984/_utils/
-
As of CouchDB 3.x, Fauxton is no longer shipped in the container and must be installed separately:
npm install -g fauxton fauxton
Open http://localhost:8000 in a browser. Run fauxton with
-p PORT
to use a port other than 8000.
-
-
-
Shared Directory
- If you are following the default instructions, move/merge the
sharedfiles
directory into the HQ root, otherwise do so into theSHARED_DRIVE_ROOT
directory referenced inlocalsettings.py
- If you are following the default instructions, move/merge the
Before running any of the commands below, you should have all of the following running: Postgres, CouchDB, Redis, and Elasticsearch. The easiest way to do this is using the Docker instructions above.
If you want to use a partitioned database, change
USE_PARTITIONED_DATABASE = False
to True
in localsettings.py
.
You will also need to create the additional databases manually:
$ psql -h localhost -p 5432 -U commcarehq
(assuming that "commcarehq" is the username in DATABASES
in
localsettings.py
). When prompted, use the password associated with the
username, of course.
commcarehq=# CREATE DATABASE commcarehq_proxy;
CREATE DATABASE
commcarehq=# CREATE DATABASE commcarehq_p1;
CREATE DATABASE
commcarehq=# CREATE DATABASE commcarehq_p2;
CREATE DATABASE
commcarehq=# \q
Populate your database:
$ ./manage.py sync_couch_views
$ ./manage.py create_kafka_topics
$ env CCHQ_IS_FRESH_INSTALL=1 ./manage.py migrate --noinput
If you are using a partitioned database, populate the additional databases too, and configure PL/Proxy:
$ env CCHQ_IS_FRESH_INSTALL=1 ./manage.py migrate_multi --noinput
$ ./manage.py configure_pl_proxy_cluster --create_only
You should run ./manage.py migrate
frequently, but only use the environment
variable CCHQ_IS_FRESH_INSTALL during your initial setup. It is used to skip a
few tricky migrations that aren't necessary for new installs.
If you are seeing errors from the CouchDB Docker container that include
database_does_not_exist
... "_users"
, it could be because CouchDB
is missing its three system databases, _users
, _replicator
and
_global_changes
. The _global_changes
database is not necessary if
you do not expect to be using the global changes feed. You can use
curl
to create the databases:
$ curl -X PUT http://username:password@127.0.0.1:5984/_users
$ curl -X PUT http://username:password@127.0.0.1:5984/_replicator
where "username" and "password" are the values of "COUCH_USERNAME"
and "COUCH_PASSWORD" given in COUCH_DATABASES
set in
dev_settings.py
.
Mac OS M1 Users: If you see the following error, check the Supplementary Guide.
ImportError: failed to find libmagic. Check your installation
If you have an authentication error running ./manage.py migrate
the first
time, open pg_hba.conf
(/etc/postgresql/9.1/main/pg_hba.conf
on Ubuntu)
and change the line "local all all peer" to "local all all md5".
If you have trouble with your first run of ./manage.py sync_couch_views
:
-
401 error related to nonexistent database:
curl -X PUT http://localhost:5984/commcarehq # create the database curl -X PUT http://localhost:5984/_config/admins/commcarehq -d '"commcarehq"' . # add admin user
-
Error stemming from any Python modules: the issue may be that your virtualenv is relying on the
site-packages
directory of your local Python installation for some of its requirements. (Creating your virtualenv with the--no-site-packages
flag should prevent this, but it seems that it does not always work). You can check if this is the case by runningpip show {name-of-module-that-is-erroring}
. This will show the location that your virtualenv is pulling that module from; if the location is somewhere other than the path to your virtualenv, then something is wrong. The easiest solution to this is to remove any conflicting modules from the location that your virtualenv is pulling them from (as long as you use virtualenvs for all of your Python projects, this won't cause you any issues). -
If you encounter an error stemming from an Incompatible Library Version of libxml2.2.dylib on Mac OS X, try running the following commands:
brew link libxml2 --force brew link libxslt --force
-
If you encounter an authorization error related to CouchDB, try going to your
localsettings.py
file and changeCOUCH_PASSWORD
to an empty string. -
If you get errors saying "Segmentation fault (core dumped)", with a warning like "RuntimeWarning: greenlet.greenlet size changed, may indicate binary incompatibility. Expected 144 from C header, got 152 from PyObject" check that your Python version is correct (3.9). Alternatively, you can try upgrading
gevent
(pip install --upgrade gevent
) to fix this error on Python 3.8, but you may run into other issues!
To set up elasticsearch indexes run the following:
./manage.py ptop_preindex [--reset]
This will create all the elasticsearch indexes (that don't already exist) and populate them with any data that's in the database.
We are transitioning to using sass
/scss
for our stylesheets. In order to compile *.scss
,
Dart Sass is required.
We recommend using npm
to install this globally with:
npm install -g sass
You can also follow the instructions here if you encounter issues with this method.
We use Yarn to manage our JavaScript dependencies. It is able to install older
bower
dependencies/repositories that we still need and npm
repositories.
Eventually we will move fully to npm
, but for now you will need yarn
to
manage js
repositories.
In order to download the required JavaScript packages, you'll need to install
yarn
and run yarn install
. Follow these steps to install:
-
Follow these steps to install Yarn.
-
Install dependencies with:
yarn install --frozen-lockfile
-
Ensure that
django
translations are compiled for javascript (or it will throw a JS error):./manage.py compilejsi18n
NOTE: if you are making changes to package.json
, please run yarn install
without the --frozen-lockfile
flag so that yarn.lock
will get updated.
Depending on your operating system, and what version of nodejs
and npm
you
have locally, you might run into issues. Here are minimum version requirements
for these packages.
$ npm --version
10.2.4
$ node --version
v20.11.1
On a clean Ubuntu 22.04 LTS install, the packaged nodejs version is expected to be v12. The easiest way to get onto the current nodejs v16 is
curl -sL https://deb.nodesource.com/setup_16.x | sudo -E bash -
sudo apt install -y nodejs
At present, we are undergoing a migration from Bootstrap 3 to 5. Bootstrap 3 uses LESS as its CSS precompiler, and Bootstrap 5 using SASS / SCSS. You will need both installed.
LESS is already taken care of by package.json
when you run yarn install
. In order to
compile SASS, we need Dart Sass. There is a sass
npm package that can be installed globally with
npm install -g sass
, however this installs the pure javascript version without a binary. For speed in a
development environment, it is recommended to install sass
with homebrew:
brew install sass/sass/sass
You can also view alternative installation instructions if homebrew doesn't work for you.
This is the setup most developers use. If you don't know which option to use,
use this one. It's the simplest to set up and the least painful way to develop:
just make sure your localsettings.py
does not contain COMPRESS_ENABLED
or
COMPRESS_OFFLINE
settings (or has them both set to False
).
The disadvantage is that this is a different setup than production, where LESS/SASS files are compressed.
This mirrors production's setup, but it's really only useful if you're trying to debug issues that mirror production that's related to staticfiles and compressor. For all practical uses, please use Option 1 to save yourself the headache.
Make sure your localsettings.py
file has the following set:
COMPRESS_ENABLED = True
COMPRESS_OFFLINE = True
For all STATICFILES changes (primarily LESS, SASS, and JavaScript), run:
./manage.py collectstatic
./manage.py compilejsi18n
./manage.py fix_less_imports_collectstatic
./manage.py compress
We recommend disabling the cache. In Chrome, go to Dev Tools > Settings > Preferences > Network and check the following:
- Disable cache (while DevTools is open)
To be able to use CommCare, you'll want to create a superuser, which you can do by running:
./manage.py make_superuser <email>
This can also be used to promote a user created by signing up to a superuser.
Note that promoting a user to superuser status using this command will also give them the ability to assign other users as superuser in the in-app Superuser Management page.
Make sure the required services are running (PostgreSQL, Redis, CouchDB, Kafka, Elasticsearch).
./manage.py check_services
Some of the services listed there aren't necessary for very basic operation, but it can give you a good idea of what's broken. If you're not running formplayer in docker, it will of course fail. Don't worry about celery for now.
Then run the django server with the following command:
./manage.py runserver localhost:8000
You should now be able to load CommCare HQ in a browser at http://localhost:8000.
If you can load the page, but either the styling is missing or you get JavaScript console errors trying to create an account, try running the JavaScript set up steps again. In particular you may need to run:
yarn install --frozen-lockfile
./manage.py compilejsi18n
./manage.py fix_less_imports_collectstatic
Formplayer is a Java service that allows us to use applications on the web instead of on a mobile device.
In localsettings.py
:
FORMPLAYER_URL = 'http://localhost:8080'
FORMPLAYER_INTERNAL_AUTH_KEY = "secretkey"
LOCAL_APPS += ('django_extensions',)
IMPORTANT: When running HQ, be sure to use runserver_plus
./manage.py runserver_plus localhost:8000
Then you need to have Formplayer running.
Before running Formplayer, you need to initialize the formplayer database.
The password for the "commcarehq" user is in the localsettings.py file in the
DATABASES
dictionary.
createdb formplayer -U commcarehq -h localhost
The fastest way to get Formplayer running outside of docker is to download the
application.properties
and formplayer.jar
files and run it directly. You may
download and run these in the commcare-hq repo root (these files are excluded
from git in the .gitignore
file).
curl https://raw.githubusercontent.com/dimagi/formplayer/master/config/application.example.properties -o application.properties
curl https://s3.amazonaws.com/dimagi-formplayer-jars/latest-successful/formplayer.jar -o formplayer.jar
Thereafter, to run Formplayer, navigate to the dir where you installed them above (probably the repo root), and run:
java -jar ./formplayer.jar
This starts a process in the foreground, so you'll need to keep it open as long as you plan on using Formplayer.
To keep Formplayer up to date with the version used in production, you can add
the curl
commands above to your hammer
command (see hammer.sh),
or whatever script you use for updating your dev environment.
The following additional processes are required for certain parts of the application to work. They can each be run in separate terminals:
Pillowtop is used to keep elasticsearch indices and configurable reports in sync.
Note Typically, you won't need to run these in a dev environment unless you are testing them. It is simpler to run the 'reindex' command to update ES with local changes when needed. See Populate Elasticsearch
Mac OS: run_ptop
Will likely not work for you.
See the Supplementary Guide for help.
It can be run as follows:
./manage.py run_ptop --all --processor-chunk-size=1
You can also run individual pillows with the following command
(Pillow names can be found in settings.py
):
./manage.py run_ptop --pillow-name=CaseSearchToElasticsearchPillow --processor-chunk-size=1
Alternatively, you can not run pillowtop and instead manually sync ES indices when needed, by calling the ptop_reindexer_v2
command.
See the command help for details, but it can be used to sync individual indices like this:
./manage.py ptop_reindexer_v2 user --reset
Celery is used for background jobs and scheduled tasks.
You can avoid running it by setting CELERY_TASK_ALWAYS_EAGER=False
in your localsettings.py
,
though some parts of HQ (especially those involving file uploads and exports) require running it.
This can be done using:
celery -A corehq worker -l info
This will run a worker that will listen to default queue which is named celery.
You may need to add a -Q
argument based on the queue you want to listen to.
For example, to use case importer with celery locally you need to run:
celery -A corehq worker -l info -Q case_import_queue
You can also run multiple queues on a single worker by passing multiple queue names separated by ,
celery -A corehq worker -l info -Q case_import_queue,background_queue
If you want to run periodic tasks you would need to start beat
service along with celery by running
celery -A corehq beat
By default, HQ uses Vellum minified build files to render form-designer. To use files from Vellum directly, do the following.
-
Make Django expect the Vellum source code in the
submodules/formdesigner
directory instead of using a minified build by enabling Vellum "dev mode" inlocalsettings.py
:VELLUM_DEBUG = "dev"
-
Clone (or symlink to repo elsewhere on disk) the Vellum repostory into/at the
submodules/formdesigner
directory under the commcare-hq repo root:# clone directly: git clone git@github.com:dimagi/Vellum.git ./submodules/formdesigner # -- OR -- # symlink existing Vellum code at submodules/formdesigner ln -s absolute/path/to/Vellum ./submodules/formdesigner
To run the standard tests for CommCare HQ, run
./manage.py test
These may not all pass in a local environment. It's often more practical, and faster, to just run tests for the django app where you're working.
To run a particular test or subset of tests:
./manage.py test <test.module.path>[:<TestClass>[.<test_name>]]
# examples
./manage.py test corehq.apps.app_manager
./manage.py test corehq.apps.app_manager.tests.test_suite:SuiteTest
./manage.py test corehq.apps.app_manager.tests.test_suite:SuiteTest.test_picture_format
# alternate: file system path
./manage.py test corehq/apps/app_manager
./manage.py test corehq/apps/app_manager/tests/test_suite.py:SuiteTest
./manage.py test corehq/apps/app_manager/tests/test_suite.py:SuiteTest.test_picture_format
To use the pdb
debugger in tests, include the s
flag:
./manage.py test -s <test.module.path>[:<TestClass>[.<test_name>]]
If database tests are failing because of a permission denied
error, give your
Postgres user permissions to create a database.
In the Postgres shell, run the following as a superuser:
ALTER USER commcarehq CREATEDB;
If you are on arm64 architecture using a non-Dimagi Docker Postgres image:
- If you run into a missing "hashlib.control" or "plproxy.control" file while trying to run tests, it is because you are not using the Dimagi Postgres Docker image that includes the pghashlib and plproxy extensions. You will need to change the USE_PARTITIONED_DATABASE variable in your localsettings.py to False so that you won't shard your test database and need the extensions
USE_PARTITIONED_DATABASE = False
To avoid having to run the database setup for each test run you can specify the
REUSE_DB
environment variable which will use an existing test database if one
exists:
REUSE_DB=1 ./manage.py test corehq.apps.app_manager
Or, to drop the current test DB and create a fresh one
./manage.py test corehq.apps.app_manager --reusedb=reset
See corehq.tests.nose.HqdbContext
(source) for full
description of REUSE_DB
and --reusedb
.
The CCHQ_TESTING
environment variable allows you to run management commands in
the context of your test environment rather than your dev environment.
This is most useful for shell or direct database access:
CCHQ_TESTING=1 ./manage.py dbshell
CCHQ_TESTING=1 ./manage.py shell
Deprecation warnings are converted to errors when running tests unless the
warning has been whitelisted (or unless PYTHONWARNINGS
is set with a value
that does not convert warnings to errors, more below). The warnings whitelist
can be found in corehq/warnings.py
.
The CCHQ_STRICT_WARNINGS
environment variable can be used to convert
non-whitelisted deprecation warnings into errors for all management commands
(in addition to when running tests). It is recommended to add this to your bash
profile or wherever you set environment variables for your shell:
export CCHQ_STRICT_WARNINGS=1
If you don't want strict warnings, but do want to ignore (or perform some other
action on) whitelisted warnings you can use the CCHQ_WHITELISTED_WARNINGS
environment variable instead. CCHQ_WHITELISTED_WARNINGS
accepts any of the
PYTHONWARNINGS
action values (ignore
, default
, error
, etc).
export CCHQ_WHITELISTED_WARNINGS=ignore
CCHQ_WHITELISTED_WARNINGS=ignore
is implied when CCHQ_STRICT_WARNINGS=1
is
set.
If PYTHONWARNINGS
is set it may override the default behavior of converting
warnings to errors. This allows additional local configuration of personal
whitelist entries, for example. Ensure there is an "error" item as the first
entry in the value to preserve the default behavior of converting warnings to
errors. For example:
export PYTHONWARNINGS='
error,
ignore:Using or importing the ABCs::kombu.utils.functional,
ignore:Using or importing the ABCs::celery.utils.text,
ignore:the imp module is deprecated::celery.utils.imports,
ignore:unclosed:ResourceWarning'
Personal whitelist items may also be added in localsettings.py.
You can run all tests with a certain tag as follows:
./manage.py test --attr=tag
Available tags:
- slow: especially slow tests
- sharded: tests that should get run on the sharded test runner
- es_test: Elasticsearch tests
See http://nose.readthedocs.io/en/latest/plugins/attrib.html for more details.
# only run tests that extend TestCase
./manage.py test --db=only
# skip all tests that extend TestCase but run everything else
./manage.py test --db=skip
See https://github.com/nose-devs/nose/blob/master/nose/plugins/testid.py
Make sure javascript packages are installed with the following. Please see the
section on installing yarn
above for more details.
It's recommended to install grunt globally (with yarn
) in order to use grunt
from the command line:
yarn global add grunt
yarn global add grunt-cli
You'll then need to add the yarn bin folder to your path:
export PATH="$(yarn global bin):$PATH"
More information can be found here
In order for the tests to run the development server needs to be running on port 8000.
To run all JavaScript tests in all the apps:
grunt test
To run the JavaScript tests for a particular app run:
grunt test:<app_name> # (e.g. grunt test:app_manager)
To list all the apps available to run:
grunt list
To run tests from the browser (useful for debugging) visit this url:
http://localhost:8000/mocha/<app_name>
Occasionally you will see an app specified with a #
, like app_manager#b3
.
The string after #
specifies that the test uses an alternate configuration. To
visit this suite in the browser go to:
http://localhost:8000/mocha/<app_name>/<config>
For example:
http://localhost:8000/mocha/app_manager/b3
To generate a JavaScript test coverage report, ensure the development server is active on port 8000 and run:
./scripts/coverage-js.sh
This script goes through the steps to prepare a report for test coverage of
JavaScript files that are touched by tests, i.e., apps and files with 0% test
coverage will not be shown. A coverage summary is output to the terminal and a
detailed html report is generated at coverage-js/index.html
.