Skip to content

Commit

Permalink
Merge pull request IQSS#9584 from poikilotherm/9540-ct-dev-docs
Browse files Browse the repository at this point in the history 
9540 - more docs for container usage
  • Loading branch information
@kcondon
kcondon committed Jun 23, 2023
2 parents 3d6eabe + eaa0ef5 commit 0cad39a
Show file tree
Hide file tree
Showing 4 changed files with 170 additions and 92 deletions.
156 changes: 145 additions & 11 deletions doc/sphinx-guides/source/container/dev-usage.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,8 @@ Please note! This Docker setup is not for production!
Quickstart
----------

First, install Java 11 and Maven.

After cloning the repo, try this:

``mvn -Pct clean package docker:run``
Expand All @@ -27,25 +29,157 @@ Assuming you have `Docker <https://docs.docker.com/engine/install/>`_, `Docker D
you have Java and Maven installed, as you are at least about to develop code changes.

To test drive these local changes to the Dataverse codebase in a containerized application server (and avoid the
setup described in :doc:`../developers/dev-environment`), you must a) build the application container and b)
run it in addition to the necessary dependencies.

Building and Running
--------------------
setup described in :doc:`../developers/dev-environment`), you must a) build the application and b) run it in addition
to the necessary dependencies. (Which might involve building a new local version of the :doc:`configbaker-image`.)

To build the application image, run the following command, as described in :doc:`app-image`:
.. _dev-build:

``mvn -Pct clean package``
Building
--------

Now, start all the containers with a single command:
To build the :doc:`application <app-image>` and :doc:`config baker image <configbaker-image>`, run the following command:

``mvn -Pct docker:run``
``mvn -Pct clean package``

(You could also concatenate both commands into one, as shown above in the quickstart.)
Once this is done, you will see images ``gdcc/dataverse:unstable`` and ``gdcc/configbaker:unstable`` available in your
Docker cache.

**Note:** This will skip any unit tests. If you have built the code before for testing, etc. you might omit the
``clean`` to avoid recompiling.

**Note:** Also we have a ``docker-compose-dev.yml`` file, it's currently not possible to build the images without
invoking Maven. This might change in the future.


.. _dev-run:

Running
-------

After building the app and config baker image containing your local changes to the Dataverse application, you want to
run it together with all dependencies. There are four ways to do this (commands executed at root of project directory):

.. list-table:: Cheatsheet: Running Containers
:widths: 15 40 45
:header-rows: 1
:stub-columns: 1
:align: left

* - \
- Using Maven
- Using Compose
* - In foreground
- ``mvn -Pct docker:run``
- ``docker compose -f docker-compose-dev.yml up``
* - In background
- ``mvn -Pct docker:start``
- ``docker compose -f docker-compose-dev.yml up -d``

Both ways have their pros and cons:

.. list-table:: Decision Helper: Fore- or Background?
:widths: 15 40 45
:header-rows: 1
:stub-columns: 1
:align: left

* - \
- Pros
- Cons
* - Foreground
- | Logs scroll by when interacting with API / UI
| To stop all containers simply hit ``Ctrl+C``
- | Lots and lots of logs scrolling by
| Must stop all containers to restart
* - Background
- | No logs scrolling by
| Easy to replace single containers
- | No logs scrolling by
| Stopping containers needs an extra command
In case you want to concatenate building and running, here's a cheatsheet for you:

.. list-table:: Cheatsheet: Building and Running Containers
:widths: 15 40 45
:header-rows: 1
:stub-columns: 1
:align: left

* - \
- Using Maven
- Using Compose
* - In foreground
- ``mvn -Pct package docker:run``
- ``mvn -Pct package && docker compose -f docker-compose-dev.yml up``
* - In background
- ``mvn -Pct package docker:start``
- ``mvn -Pct package && docker compose -f docker-compose-dev.yml up -d``

Once all containers have been started, you can check if the application was deployed correctly by checking the version
at http://localhost:8080/api/info/version or watch the logs.

**Note:** To stop all containers you started in background, invoke ``mvn -Pct docker:stop`` or
``docker compose -f docker-compose-dev.yml down``.

Check that you can log in to http://localhost:8080 using user ``dataverseAdmin`` and password ``admin1``.

You can also access the Payara Admin Console if needed, which is available at http://localhost:4848. To log in, use user ``admin`` and password ``admin``. As a reminder, the application container is for development use only, so we are exposing the admin console for testing purposes. In a production environment, it may be more convenient to leave this console unopened.
You can also access the Payara Admin Console if needed, which is available at http://localhost:4848. To log in, use
user ``admin`` and password ``admin``. As a reminder, the application container is for development use only, so we
are exposing the admin console for testing purposes. In a production environment, it may be more convenient to leave
this console unopened.

Note that data is persisted in ``./docker-dev-volumes`` in the root of the Git repo. For a clean start, you should
remove this directory before running the ``mvn`` commands above.


.. _dev-logs:

Viewing Logs
------------

In case you started containers in background mode (see :ref:`dev-run`), you can use the following commands to view and/or
watch logs from the containers.

The safe bet for any running container's logs is to lookup the container name via ``docker ps`` and use it in
``docker logs <name>``. You can tail logs by adding ``-n`` and follow them by adding ``-f`` (just like ``tail`` cmd).
See ``docker logs --help`` for more.

Alternatives:

- In case you used Maven for running, you may use ``mvn -Pct docker:logs -Ddocker.filter=<service name>``.
- If you used Docker Compose for running, you may use ``docker compose -f docker-compose-dev.yml logs <service name>``.
Options are the same.


Re-Deploying
------------

Currently, the only safe and tested way to re-deploy the Dataverse application after you applied code changes is
by recreating the container(s). In the future, more options may be added here.

If you started your containers in foreground, just stop them and follow the steps for building and running again.
The same goes for using Maven to start the containers in the background.

In case of using Docker Compose and starting the containers in the background, you can use a workaround to only
restart the application container:

.. code-block::
# First rebuild the container (will complain about an image still in use, this is fine.)
mvn -Pct package
# Then re-create the container (will automatically restart the container for you)
docker compose -f docker-compose-dev.yml create dev_dataverse
Using ``docker container inspect dev_dataverse | grep Image`` you can verify the changed checksums.

Using A Debugger
----------------

The :doc:`base-image` enables usage of the `Java Debugging Wire Protocol <https://dzone.com/articles/remote-debugging-java-applications-with-jdwp>`_
for remote debugging if you set ``ENABLE_JDWP=1`` as environment variable for the application container.
The default configuration when executing containers with the commands listed at :ref:`dev-run` already enables this.

There are a lot of tutorials how to connect your IDE's debugger to a remote endpoint. Please use ``localhost:9009``
as the endpoint. Here are links to the most common IDEs docs on remote debugging:
`Eclipse <https://help.eclipse.org/latest/topic/org.eclipse.jdt.doc.user/concepts/cremdbug.htm?cp=1_2_12>`_,
`IntelliJ <https://www.jetbrains.com/help/idea/tutorial-remote-debug.html#debugger_rc>`_
22 changes: 17 additions & 5 deletions doc/sphinx-guides/source/developers/dev-environment.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -7,12 +7,20 @@ These instructions are purposefully opinionated and terse to help you get your d
.. contents:: |toctitle|
:local:

Quick Start
-----------
Quick Start (Docker)
--------------------

The quickest way to get the Dataverse Software running is to use Vagrant as described in the :doc:`tools` section, or use Docker containers as described the :doc:`../container/dev-usage` section of the Container Guide.
The quickest way to get Dataverse running is in Docker as explained in :doc:`../container/dev-usage` section of the Container Guide.

For day to day development work, we recommended the following setup.

Classic Dev Environment
-----------------------

Since before Docker existed, we have encouraged installing Dataverse and all its dependencies directly on your development machine, as described below. This can be thought of as the "classic" development environment for Dataverse.

However, in 2023 we decided that we'd like to encourage all developers to start using Docker instead and opened https://github.com/IQSS/dataverse/issues/9616 to indicate that we plan to rewrite this page to recommend the use of Docker.

There's nothing wrong with the classic instructions below and we don't plan to simply delete them. They are a valid alternative to running Dataverse in Docker. We will likely move them to another page.

Set Up Dependencies
-------------------
Expand All @@ -22,7 +30,7 @@ Supported Operating Systems

Mac OS X or Linux is required because the setup scripts assume the presence of standard Unix utilities.

Windows is not well supported, unfortunately, but Vagrant and Minishift environments are described in the :doc:`windows` section.
Windows is gaining support through Docker as described in the :doc:`windows` section.

Install Java
~~~~~~~~~~~~
Expand Down Expand Up @@ -186,6 +194,10 @@ If you want to remove the containers, then run:

``docker-compose -f docker-compose-dev.yml down``

If you want to run a single container (the mail server, for example) then run:

``docker-compose -f docker-compose-dev.yml up dev_smtp``

For a fresh installation, and before running the Software Installer Script, it is recommended to delete the docker-dev-env folder to avoid installation problems due to existing data in the containers.

Run the Dataverse Software Installer Script
Expand Down
83 changes: 7 additions & 76 deletions doc/sphinx-guides/source/developers/windows.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,86 +2,17 @@
Windows Development
===================

Development on Windows is not well supported, unfortunately. You will have a much easier time if you develop on Mac or Linux as described under :doc:`dev-environment` section.

Vagrant commands appear below and were tested on Windows 10 but the Vagrant environment is currently broken. Please see https://github.com/IQSS/dataverse/issues/6849

There is a newer, experimental Docker option described under :doc:`/container/dev-usage` in the Container Guide.
Historically, development on Windows is `not well supported <https://groups.google.com/d/msg/dataverse-community/Hs9j5rIxqPI/-q54751aAgAJ>`_ but as of 2023 a container-based approach is recommended.

.. contents:: |toctitle|
:local:

Running the Dataverse Software in Vagrant
-----------------------------------------

Install Vagrant
~~~~~~~~~~~~~~~

Download and install Vagrant from https://www.vagrantup.com

Vagrant advises you to reboot but let's install VirtualBox first.

Install VirtualBox
~~~~~~~~~~~~~~~~~~

Download and install VirtualBox from https://www.virtualbox.org

Note that we saw an error saying "Oracle VM VirtualBox 5.2.8 Setup Wizard ended prematurely" but then we re-ran the installer and it seemed to work.

Reboot
~~~~~~

Again, Vagrant asks you to reboot, so go ahead.

Install Git
~~~~~~~~~~~

Download and install Git from https://git-scm.com

Configure Git to use Unix Line Endings
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Launch Git Bash and run the following commands:

``git config --global core.autocrlf input``

Pro tip: Use Shift-Insert to paste into Git Bash.

See also https://help.github.com/articles/dealing-with-line-endings/

If you skip this step you are likely to see the following error when you run ``vagrant up``.

``/tmp/vagrant-shell: ./install: /usr/bin/perl^M: bad interpreter: No such file or directory``

Clone Git Repo
~~~~~~~~~~~~~~

From Git Bash, run the following command:

``git clone https://github.com/IQSS/dataverse.git``

vagrant up
~~~~~~~~~~

From Git Bash, run the following commands:

``cd dataverse``

The ``dataverse`` directory you changed is the one you just cloned. Vagrant will operate on a file called ``Vagrantfile``.

``vagrant up``

After a long while you hopefully will have a Dataverse installation available at http://localhost:8888

Improving Windows Support
-------------------------

Windows Subsystem for Linux
~~~~~~~~~~~~~~~~~~~~~~~~~~~
Running Dataverse in Docker on Windows
--------------------------------------

We have been unable to get Windows Subsystem for Linux (WSL) to work. We tried following the steps at https://docs.microsoft.com/en-us/windows/wsl/install-win10 but the "Get" button was greyed out when we went to download Ubuntu.
See the `post <https://groups.google.com/g/dataverse-dev/c/utqkZ7gYsf4/m/4IDtsvKSAwAJ>`_ by Akio Sone for additional details, but please observe the following:

Discussion and Feedback
~~~~~~~~~~~~~~~~~~~~~~~
- In git, the line-ending setting should be set to always LF (line feed, ``core.autocrlf=input``)
- You must have jq installed: https://jqlang.github.io/jq/download/

For more discussion of Windows support for Dataverse Software development see our community list thread `"Do you want to develop on Windows?" <https://groups.google.com/d/msg/dataverse-community/Hs9j5rIxqPI/-q54751aAgAJ>`_ We would be happy to incorporate feedback from Windows developers into this page. The :doc:`documentation` section describes how.
One the above is all set you can move on to :doc:`/container/dev-usage` in the Container Guide.
1 change: 1 addition & 0 deletions docker-compose-dev.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ services:
- DATAVERSE_DB_HOST=postgres
- DATAVERSE_DB_PASSWORD=secret
- DATAVERSE_DB_USER=${DATAVERSE_DB_USER}
- ENABLE_JDWP=1
- DATAVERSE_FEATURE_API_BEARER_AUTH=1
ports:
- "8080:8080" # HTTP (Dataverse Application)
Expand Down

0 comments on commit 0cad39a

Please sign in to comment.