diff --git a/.dockerignore b/.dockerignore index 50d39eda5..b60d80b2d 100644 --- a/.dockerignore +++ b/.dockerignore @@ -3,3 +3,4 @@ Dockerfile docker-compose*.yml .vagrant/ codecov/ +.dev/ diff --git a/.gitignore b/.gitignore index 81939ad6c..83f9d73ff 100644 --- a/.gitignore +++ b/.gitignore @@ -14,3 +14,4 @@ lib/ log/ .vagrant/ codecov/ +.dev/ diff --git a/Dockerfile b/Dockerfile index 1b2624a7f..fac5e24da 100644 --- a/Dockerfile +++ b/Dockerfile @@ -51,6 +51,8 @@ RUN sudo python3 prerequisites.py --yes --cmsuser=cmsuser install RUN sudo sed 's|/cmsuser:your_password_here@localhost:5432/cmsdb"|/postgres@testdb:5432/cmsdbfortesting"|' ./config/cms.conf.sample \ | sudo tee /usr/local/etc/cms-testdb.conf +RUN sudo sed 's|/cmsuser:your_password_here@localhost:5432/cmsdb"|/postgres@devdb:5432/cmsdb"|' ./config/cms.conf.sample \ + | sudo tee /usr/local/etc/cms-devdb.conf ENV LANG C.UTF-8 diff --git a/cms-dev.sh b/cms-dev.sh new file mode 100755 index 000000000..875c11b8a --- /dev/null +++ b/cms-dev.sh @@ -0,0 +1,6 @@ +#!/usr/bin/env bash +set -x + +GIT_BRANCH_NAME=$(git rev-parse --abbrev-ref HEAD) + +docker compose -p $GIT_BRANCH_NAME -f docker-compose.dev.yml run --build --rm --service-ports devcms diff --git a/cms-test.sh b/cms-test.sh new file mode 100755 index 000000000..1202d4f31 --- /dev/null +++ b/cms-test.sh @@ -0,0 +1,6 @@ +#!/usr/bin/env bash +set -x + +GIT_BRANCH_NAME=$(git rev-parse --abbrev-ref HEAD) + +docker compose -p $GIT_BRANCH_NAME -f docker-compose.test.yml run --build --rm testcms diff --git a/docker-compose.dev.yml b/docker-compose.dev.yml new file mode 100644 index 000000000..acd09af89 --- /dev/null +++ b/docker-compose.dev.yml @@ -0,0 +1,25 @@ +services: + devdb: + image: postgres + environment: + POSTGRES_HOST_AUTH_METHOD: trust + volumes: + - "./.dev/postgres-data:/var/lib/postgresql/data" + + devcms: + build: . + depends_on: + - "devdb" + environment: + CMS_CONFIG: /usr/local/etc/cms-devdb.conf + volumes: + - "./.dev/home:/home/cmsuser" + - ".:/home/cmsuser/cms" + privileged: true + cgroup: host + command: > + wait-for-it devdb:5432 -- bash + ports: + - 8888:8888 + - 8889:8889 + - 8890:8890 diff --git a/docs/Docker image.rst b/docs/Docker image.rst index d8058ea38..99d32051b 100644 --- a/docs/Docker image.rst +++ b/docs/Docker image.rst @@ -3,8 +3,19 @@ Docker image We provide a docker image (defined in :gh_blob:`Dockerfile`) that can be used to easily get an instance of CMS running locally with all the necessary -dependencies. We also provide a :gh_blob:`docker-compose.test.yml` files that -uses said docker image to run the tests. +dependencies. We also provide: + +* :gh_blob:`cms-test.sh`: This file uses :gh_blob:`docker-compose.test.yml` to + spawn a volatile database (not persisted on disk) as well as a CMS instance + that automatically runs all tests. + +* :gh_blob:`cms-dev.sh`: This file uses :gh_blob:`docker-compose.dev.yml` to + spawn a database (persisted in the local ``.dev/postgres-data`` folder + within the repository) as well as a CMS instance that only runs `bash`, + leaving you with a shell from where you can start cms services and servers. + The DB port and CMS server ports are also automatically forwarded on the + host machine (respectively to ``15432`` for the database, and ``8888-8890`` + for CMS). Make sure that you have a recent version of Docker installed, as well as Docker Compose. @@ -14,45 +25,109 @@ Compose. Running tests ============= -First you need to build the docker image, then you use it to run the tests. +You can simply run: + +.. sourcecode:: bash + + ./cms-test.sh + +Or, you can issue the full command (that is defined in ``cms-test.sh``) which +is similar to: + +.. sourcecode:: bash + + docker compose -p someprojectname -f docker-compose.test.yml run --build --rm testcms .. note:: + Some versions of docker require to specify ``-p``, some version will fill it + for you based on the current folder's name (which for us would be equivalent + to passing ``-p cms``). + The ``-p`` flag is used as a namespace for the containers that will be created. When you're running tests on a separate branch, it can be useful to - include the branch name there, to avoid any conflict. (You can also omit the - flag and specify the name via the ``COMPOSE_PROJECT_NAME`` environment - variable.) + include the branch name there, to avoid any conflict. The ``cms-test.sh`` + script uses the **name of the current git branch** and passes it to ``-p``. + + Note also that if you are not part of the ``docker`` group then you'll need + to run every docker command with ``sudo``, including ``sudo ./cms-test.sh``. + We recommend adding yourself to the ``docker`` group. + +What the ``cms-test.sh`` command does is: first build a fresh CMS image when +necessary, and then create (assuming you didn't specify the ``-p`` flag) a +``cms-testdb-1`` container for the database, and a +``cms-testcms-run-`` container for CMS. + +The database container **will not** be automatically deleted, while the CMS +container will be automatically deleted upon exiting (because of the ``--rm`` +flag). + +To delete the ``cms-testdb-1`` container after testing you can run: + +.. sourcecode:: bash - If you are not part of the ``docker`` group, then you need to run every - docker command with ``sudo``. + docker rm -f cms-testdb-1 -To build the image: +Developing CMS +============== + +To run a local development instance of CMS, you can simply run: .. sourcecode:: bash - docker compose -p cms -f docker-compose.test.yml build testcms + ./cms-dev.sh -To run the tests: +Or, you can issue the full command (that is defined in ``cms-dev.sh``) which is +similar to: .. sourcecode:: bash - docker compose -p cms -f docker-compose.test.yml run --rm testcms + docker compose -p someprojectname -f docker-compose.dev.yml run --build --rm --service-ports devcms + +The command will build a fresh CMS image when necessary, and drop you into a +bash prompt where the repository is mounted on ``~/cms`` for ease of +development. You can edit the code from the host (i.e. outside the container) +and then reinstall CMS (``python3 setup.py install``) directly from inside the +container, without having to rebuild the image every time. -Another option is to add the ``--build`` flag to the ``run`` command, to perform -a new build before running the tests: +Upon running ``cms-dev.sh`` for the first time, the database will initially be +empty. You need to initialize it (notice that the following commands are +indicated with a ``>>>`` prompt because they are meant to be executed **inside** +the container, from the prompt that you get to after running ``cms-dev.sh``) +like so: .. sourcecode:: bash - docker compose -p cms -f docker-compose.test.yml run --build --rm testcms + >>> createdb -h devdb -U postgres cmsdb + >>> cmsInitDB -This command will create (assuming you used ``-p cms``) a ``cms-testdb-1`` -container for the database which **will not** be automatically deleted, and a -``cms-testcms-run-`` container for CMS which will be -automatically deleted (because of the ``--rm`` flag) upon exiting. +Then you probably want to download a test contest and import it, for example +like this: -To delete the ``cms-testdb-1`` container after testing you can run: +.. sourcecode:: bash + + >>> git clone https://github.com/cms-dev/con_test.git + >>> cd con_test + >>> cmsImportUser --all + >>> cmsImportContest -i . + +If this succeeds, you can then run one of the servers, for example the +ContestWebServer, like so: .. sourcecode:: bash - docker rm -f cms-testdb-1 + >>> cmsContestWebServer + +When it prompts you to choose a contest ID, you can simply hit Enter. + +When the server is finally running, you can check (from the host machine) that +the server is reachable at http://localhost:8888/ + +You can also verify that upon exiting the container's bash shell and reentering +it (by running ``cms-dev.sh`` again) you won't need to re-import the contest, as +the database is persisted on disk on the host machine. Even manually destroying +and recreating the database container will retain the same data. If for some +reason you need to reset the database, we recommend using the ``dropdb -h devdb +-U postgres cmsdb`` command inside the container. To remove any trace of the +database data, you can delete the ``.dev/postgres-data`` folder within the git +repository.