From 51114583888e151d9830c51c8b1109a5d72bf663 Mon Sep 17 00:00:00 2001 From: Oliver Bertuch Date: Wed, 10 May 2023 14:56:56 +0200 Subject: [PATCH 1/9] docs(ct): start extending dev usage docs --- .../source/container/dev-usage.rst | 99 +++++++++++++++++-- 1 file changed, 92 insertions(+), 7 deletions(-) diff --git a/doc/sphinx-guides/source/container/dev-usage.rst b/doc/sphinx-guides/source/container/dev-usage.rst index 4eba70eb23b..28ab7196926 100644 --- a/doc/sphinx-guides/source/container/dev-usage.rst +++ b/doc/sphinx-guides/source/container/dev-usage.rst @@ -17,21 +17,91 @@ To test drive these local changes to the Dataverse codebase in a containerized a setup described in :doc:`../developers/dev-environment`), you must a) build the application container and b) run it in addition to the necessary dependencies. -Building and Running --------------------- +Building +-------- To build the application image, run the following command, as described in :doc:`app-image`: ``mvn -Pct clean package`` -Now, start all the containers with a single command: +Once this is done, you will see an image ``gdcc/dataverse:unstable`` available in your Docker cache. + +**Note:** This will skip any unit tests. If you have built the code before for testing, etc. you might omit the ``clean`` to +avoid recompiling. + +**Note:** Also we have a ``docker-compose-dev.yml`` file, it's currently not possible to build the images without +invoking Maven. This might change in the future. + +Running +------- + +After building the app image containing your local changes to the Dataverse application, you want to run it together +with all dependencies. There are four ways to do this (commands executed at root of project directory): + +.. list-table:: Cheatsheet: Running Containers + :widths: 15 40 45 + :header-rows: 1 + :stub-columns: 1 + :align: left + + * - \ + - Using Maven + - Using Compose + * - In foreground + - ``mvn -Pct docker:run`` + - ``docker compose -f docker-compose-dev.yml up`` + * - In background + - ``mvn -Pct docker:start`` + - ``docker compose -f docker-compose-dev.yml up -d`` + +Both ways have their pros and cons: + +.. list-table:: Decision Helper: Fore- or Background? + :widths: 15 40 45 + :header-rows: 1 + :stub-columns: 1 + :align: left + + * - \ + - Pros + - Cons + * - Foreground + - | Logs scroll by when interacting with API / UI + | To stop all containers simply hit ``Ctrl+C`` + - | Lots and lots of logs scrolling by + | Must stop all containers to restart + * - Background + - | No logs scrolling by + | Easy to replace single containers + - | No logs scrolling by + | Stopping containers needs an extra command + +In case you want to concatenate building and running, here's a cheatsheet for you: + +.. list-table:: Cheatsheet: Building and Running Containers + :widths: 15 40 45 + :header-rows: 1 + :stub-columns: 1 + :align: left + + * - \ + - Using Maven + - Using Compose + * - In foreground + - ``mvn -Pct package docker:run`` + - ``mvn -Pct package && docker compose -f docker-compose-dev.yml up`` + * - In background + - ``mvn -Pct package docker:start`` + - ``mvn -Pct package && docker compose -f docker-compose-dev.yml up -d`` -``mvn -Pct docker:run`` +Once all containers have been started, you can check if the application was deployed correctly by checking the version +at http://localhost:8080/api/info/version. or watch the logs. -(You could also concatenate both commands into one.) +**Note:** To stop all containers you started in background, invoke ``mvn -Pct docker:stop`` or +``docker compose -f docker-compose-dev.yml down``. -Once all containers have been started, you can check if the application was deployed correctly by checking the version -at http://localhost:8080/api/info/version. +Bootstrapping New Instance +-------------------------- If all looks good, run the :download:`docker-final-setup.sh <../../../../scripts/dev/docker-final-setup.sh>` script below. (This is a simplified version of the script described in :ref:`rebuilding-dev-environment`.) @@ -49,3 +119,18 @@ You can also access the Payara Admin Console if needed, which is available at ht Note that data is persisted in ``./docker-dev-volumes`` in the root of the Git repo. For a clean start, you should remove this directory before running the ``mvn`` commands above. + +Viewing Logs +------------ + +TODO + +Re-Deploying +------------ + +TODO + +Using A Debugger +---------------- + +TODO From 78d035e348d029eb28f0e96e62036718995a8e75 Mon Sep 17 00:00:00 2001 From: Oliver Bertuch Date: Wed, 10 May 2023 15:21:14 +0200 Subject: [PATCH 2/9] docs(ct): add quickstart for dev usage --- doc/sphinx-guides/source/container/dev-usage.rst | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/doc/sphinx-guides/source/container/dev-usage.rst b/doc/sphinx-guides/source/container/dev-usage.rst index 28ab7196926..843ebef50b6 100644 --- a/doc/sphinx-guides/source/container/dev-usage.rst +++ b/doc/sphinx-guides/source/container/dev-usage.rst @@ -17,6 +17,12 @@ To test drive these local changes to the Dataverse codebase in a containerized a setup described in :doc:`../developers/dev-environment`), you must a) build the application container and b) run it in addition to the necessary dependencies. +| **TL;DR** *I have all that, just give me containers!* +| Execute ``mvn -Pct clean package docker:run``, wait and continue at :ref:`dev-bootstrap`. + + +.. _dev-build: + Building -------- @@ -32,6 +38,9 @@ avoid recompiling. **Note:** Also we have a ``docker-compose-dev.yml`` file, it's currently not possible to build the images without invoking Maven. This might change in the future. + +.. _dev-run: + Running ------- @@ -100,6 +109,10 @@ at http://localhost:8080/api/info/version. or watch the logs. **Note:** To stop all containers you started in background, invoke ``mvn -Pct docker:stop`` or ``docker compose -f docker-compose-dev.yml down``. + + +.. _dev-bootstrap: + Bootstrapping New Instance -------------------------- From bfd2e953514ed9efcf5f48931d929b6678e3bb89 Mon Sep 17 00:00:00 2001 From: Oliver Bertuch Date: Wed, 10 May 2023 15:42:40 +0200 Subject: [PATCH 3/9] docs(ct): add log viewing hints for dev usage --- .../source/container/dev-usage.rst | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-) diff --git a/doc/sphinx-guides/source/container/dev-usage.rst b/doc/sphinx-guides/source/container/dev-usage.rst index 843ebef50b6..bf0629b8d75 100644 --- a/doc/sphinx-guides/source/container/dev-usage.rst +++ b/doc/sphinx-guides/source/container/dev-usage.rst @@ -133,10 +133,25 @@ You can also access the Payara Admin Console if needed, which is available at ht Note that data is persisted in ``./docker-dev-volumes`` in the root of the Git repo. For a clean start, you should remove this directory before running the ``mvn`` commands above. + +.. _dev-logs: + Viewing Logs ------------ -TODO +In case you started containers in background mode (see :ref:`dev-run`), you can use the following commands to view and/or +watch logs from the containers. + +The safe bet for any running container's logs is to lookup the container name via ``docker ps`` and use it in +``docker logs ``. You can tail logs by adding ``-n`` and follow them by adding ``-f`` (just like ``tail`` cmd). +See ``docker logs --help`` for more. + +Alternatives: + +- In case you used Maven for running, you may use ``mvn -Pct docker:logs -Ddocker.filter=``. +- If you used Docker Compose for running, you may use ``docker compose -f docker-compose-dev.yml logs ``. + Options are the same. + Re-Deploying ------------ From 149d50030926c41ee0af5a59eefdcd1e9c69984a Mon Sep 17 00:00:00 2001 From: Oliver Bertuch Date: Wed, 10 May 2023 16:12:36 +0200 Subject: [PATCH 4/9] docs,feat(ct): enable and document JDWP usage for dev usage --- doc/sphinx-guides/source/container/dev-usage.rst | 9 ++++++++- docker-compose-dev.yml | 1 + 2 files changed, 9 insertions(+), 1 deletion(-) diff --git a/doc/sphinx-guides/source/container/dev-usage.rst b/doc/sphinx-guides/source/container/dev-usage.rst index bf0629b8d75..2f918ecd551 100644 --- a/doc/sphinx-guides/source/container/dev-usage.rst +++ b/doc/sphinx-guides/source/container/dev-usage.rst @@ -161,4 +161,11 @@ TODO Using A Debugger ---------------- -TODO +The :doc:`base-image` enables usage of the `Java Debugging Wire Protocol `_ +for remote debugging if you set ``ENABLE_JDWP=1`` as environment variable for the application container. +The default configuration when executing containers with the commands listed at :ref:`dev-run` already enables this. + +There are a lot of tutorials how to connect your IDE's debugger to a remote endpoint. Please use ``localhost:9009`` +as the endpoint. Here are links to the most common IDEs docs on remote debugging: +`Eclipse `_, +`IntelliJ `_ diff --git a/docker-compose-dev.yml b/docker-compose-dev.yml index 2b079cf6029..2fe9ba71926 100644 --- a/docker-compose-dev.yml +++ b/docker-compose-dev.yml @@ -12,6 +12,7 @@ services: - DATAVERSE_DB_HOST=postgres - DATAVERSE_DB_PASSWORD=secret - DATAVERSE_DB_USER=${DATAVERSE_DB_USER} + - ENABLE_JDWP=1 ports: - "8080:8080" # HTTP (Dataverse Application) - "4848:4848" # HTTP (Payara Admin Console) From 1386f1e77d864c18b86deeb9bcac678e51e289a4 Mon Sep 17 00:00:00 2001 From: Oliver Bertuch Date: Wed, 10 May 2023 17:08:26 +0200 Subject: [PATCH 5/9] docs(ct): add notes about redeploying in dev usage --- .../source/container/dev-usage.rst | 18 +++++++++++++++++- 1 file changed, 17 insertions(+), 1 deletion(-) diff --git a/doc/sphinx-guides/source/container/dev-usage.rst b/doc/sphinx-guides/source/container/dev-usage.rst index 2f918ecd551..3eebb10cee5 100644 --- a/doc/sphinx-guides/source/container/dev-usage.rst +++ b/doc/sphinx-guides/source/container/dev-usage.rst @@ -156,7 +156,23 @@ Alternatives: Re-Deploying ------------ -TODO +Currently, the only safe and tested way to re-deploy the Dataverse application after you applied code changes is +by recreating the container(s). In the future, more options may be added here. + +If you started your containers in foreground, just stop them and follow the steps for building and running again. +The same goes for using Maven to start the containers in the background. + +In case of using Docker Compose and starting the containers in the background, you can use a workaround to only +restart the application container: + +.. code-block:: + + # First rebuild the container (will complain about an image still in use, this is fine.) + mvn -Pct package + # Then re-create the container (will automatically restart the container for you) + docker compose -f docker-compose-dev.yml create dev_dataverse + +Using ``docker container inspect dev_dataverse | grep Image`` you can verify the changed checksums. Using A Debugger ---------------- From a8cfdd7a9572f7bc6342d0c9f0614d764b83c9d8 Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Fri, 23 Jun 2023 09:06:00 -0400 Subject: [PATCH 6/9] update windows dev page #9540 --- .../source/developers/dev-environment.rst | 2 +- .../source/developers/windows.rst | 83 ++----------------- 2 files changed, 8 insertions(+), 77 deletions(-) diff --git a/doc/sphinx-guides/source/developers/dev-environment.rst b/doc/sphinx-guides/source/developers/dev-environment.rst index 58b25d8b941..14087650eb4 100755 --- a/doc/sphinx-guides/source/developers/dev-environment.rst +++ b/doc/sphinx-guides/source/developers/dev-environment.rst @@ -22,7 +22,7 @@ Supported Operating Systems Mac OS X or Linux is required because the setup scripts assume the presence of standard Unix utilities. -Windows is not well supported, unfortunately, but Vagrant and Minishift environments are described in the :doc:`windows` section. +Windows is gaining support through Docker as described in the :doc:`windows` section. Install Java ~~~~~~~~~~~~ diff --git a/doc/sphinx-guides/source/developers/windows.rst b/doc/sphinx-guides/source/developers/windows.rst index e278b193e12..53578fe980c 100755 --- a/doc/sphinx-guides/source/developers/windows.rst +++ b/doc/sphinx-guides/source/developers/windows.rst @@ -2,86 +2,17 @@ Windows Development =================== -Development on Windows is not well supported, unfortunately. You will have a much easier time if you develop on Mac or Linux as described under :doc:`dev-environment` section. - -Vagrant commands appear below and were tested on Windows 10 but the Vagrant environment is currently broken. Please see https://github.com/IQSS/dataverse/issues/6849 - -There is a newer, experimental Docker option described under :doc:`/container/dev-usage` in the Container Guide. +Historically, development on Windows is `not well supported `_ but as of 2023 a container-based approach is recommended. .. contents:: |toctitle| :local: -Running the Dataverse Software in Vagrant ------------------------------------------ - -Install Vagrant -~~~~~~~~~~~~~~~ - -Download and install Vagrant from https://www.vagrantup.com - -Vagrant advises you to reboot but let's install VirtualBox first. - -Install VirtualBox -~~~~~~~~~~~~~~~~~~ - -Download and install VirtualBox from https://www.virtualbox.org - -Note that we saw an error saying "Oracle VM VirtualBox 5.2.8 Setup Wizard ended prematurely" but then we re-ran the installer and it seemed to work. - -Reboot -~~~~~~ - -Again, Vagrant asks you to reboot, so go ahead. - -Install Git -~~~~~~~~~~~ - -Download and install Git from https://git-scm.com - -Configure Git to use Unix Line Endings -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -Launch Git Bash and run the following commands: - -``git config --global core.autocrlf input`` - -Pro tip: Use Shift-Insert to paste into Git Bash. - -See also https://help.github.com/articles/dealing-with-line-endings/ - -If you skip this step you are likely to see the following error when you run ``vagrant up``. - -``/tmp/vagrant-shell: ./install: /usr/bin/perl^M: bad interpreter: No such file or directory`` - -Clone Git Repo -~~~~~~~~~~~~~~ - -From Git Bash, run the following command: - -``git clone https://github.com/IQSS/dataverse.git`` - -vagrant up -~~~~~~~~~~ - -From Git Bash, run the following commands: - -``cd dataverse`` - -The ``dataverse`` directory you changed is the one you just cloned. Vagrant will operate on a file called ``Vagrantfile``. - -``vagrant up`` - -After a long while you hopefully will have a Dataverse installation available at http://localhost:8888 - -Improving Windows Support -------------------------- - -Windows Subsystem for Linux -~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Running Dataverse in Docker on Windows +-------------------------------------- -We have been unable to get Windows Subsystem for Linux (WSL) to work. We tried following the steps at https://docs.microsoft.com/en-us/windows/wsl/install-win10 but the "Get" button was greyed out when we went to download Ubuntu. +See the `post `_ by Akio Sone for additional details, but please observe the following: -Discussion and Feedback -~~~~~~~~~~~~~~~~~~~~~~~ +- In git, the line-ending setting should be set to always LF (line feed, ``core.autocrlf=input``) +- You must have jq installed: https://jqlang.github.io/jq/download/ -For more discussion of Windows support for Dataverse Software development see our community list thread `"Do you want to develop on Windows?" `_ We would be happy to incorporate feedback from Windows developers into this page. The :doc:`documentation` section describes how. +One the above is all set you can move on to :doc:`/container/dev-usage` in the Container Guide. From acb1a52f2611b54d5732c100816c42c94785dbdf Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Fri, 23 Jun 2023 09:11:07 -0400 Subject: [PATCH 7/9] how to run a single container #9540 --- doc/sphinx-guides/source/developers/dev-environment.rst | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/doc/sphinx-guides/source/developers/dev-environment.rst b/doc/sphinx-guides/source/developers/dev-environment.rst index 14087650eb4..1159121f491 100755 --- a/doc/sphinx-guides/source/developers/dev-environment.rst +++ b/doc/sphinx-guides/source/developers/dev-environment.rst @@ -184,6 +184,10 @@ If you want to remove the containers, then run: ``docker-compose -f docker-compose-dev.yml down`` +If you want to run a single container (the mail server, for example) then run: + +``docker-compose -f docker-compose-dev.yml up dev_smtp`` + For a fresh installation, and before running the Software Installer Script, it is recommended to delete the docker-dev-env folder to avoid installation problems due to existing data in the containers. Run the Dataverse Software Installer Script From 6bed5a33df0a033f8ebf904bb3f14b016324efa4 Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Fri, 23 Jun 2023 09:25:58 -0400 Subject: [PATCH 8/9] encourage Docker, remove Vagrant #9540 #9616 --- .../source/developers/dev-environment.rst | 16 ++++++++++++---- 1 file changed, 12 insertions(+), 4 deletions(-) diff --git a/doc/sphinx-guides/source/developers/dev-environment.rst b/doc/sphinx-guides/source/developers/dev-environment.rst index 1159121f491..214d5da8192 100755 --- a/doc/sphinx-guides/source/developers/dev-environment.rst +++ b/doc/sphinx-guides/source/developers/dev-environment.rst @@ -7,12 +7,20 @@ These instructions are purposefully opinionated and terse to help you get your d .. contents:: |toctitle| :local: -Quick Start ------------ +Quick Start (Docker) +-------------------- -The quickest way to get the Dataverse Software running is to use Vagrant as described in the :doc:`tools` section, or use Docker containers as described the :doc:`../container/dev-usage` section of the Container Guide. +The quickest way to get Dataverse running is in Docker as explained in :doc:`../container/dev-usage` section of the Container Guide. -For day to day development work, we recommended the following setup. + +Classic Dev Environment +----------------------- + +Since before Docker existed, we have encouraged installing Dataverse and all its dependencies directly on your development machine, as described below. This can be thought of as the "classic" development environment for Dataverse. + +However, in 2023 we decided that we'd like to encourage all developers to start using Docker instead and opened https://github.com/IQSS/dataverse/issues/9616 to indicate that we plan to rewrite this page to recommend the use of Docker. + +There's nothing wrong with the classic instructions below and we don't plan to simply delete them. They are a valid alternative to running Dataverse in Docker. We will likely move them to another page. Set Up Dependencies ------------------- From eaa0ef5a4acf147fbc50b6dd5b136fd1ebc1200a Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Fri, 23 Jun 2023 09:31:33 -0400 Subject: [PATCH 9/9] improve quickstart #9540 --- .../source/container/dev-usage.rst | 20 ++++++++++++++----- 1 file changed, 15 insertions(+), 5 deletions(-) diff --git a/doc/sphinx-guides/source/container/dev-usage.rst b/doc/sphinx-guides/source/container/dev-usage.rst index 15c4a244f39..3fbe55766d5 100644 --- a/doc/sphinx-guides/source/container/dev-usage.rst +++ b/doc/sphinx-guides/source/container/dev-usage.rst @@ -6,6 +6,21 @@ Please note! This Docker setup is not for production! .. contents:: |toctitle| :local: +Quickstart +---------- + +First, install Java 11 and Maven. + +After cloning the repo, try this: + +``mvn -Pct clean package docker:run`` + +After some time you should be able to log in: + +- url: http://localhost:8080 +- username: dataverseAdmin +- password: admin1 + Intro ----- @@ -17,11 +32,6 @@ To test drive these local changes to the Dataverse codebase in a containerized a setup described in :doc:`../developers/dev-environment`), you must a) build the application and b) run it in addition to the necessary dependencies. (Which might involve building a new local version of the :doc:`configbaker-image`.) -| **TL;DR** *I have all that, just give me containers!* -| Execute ``mvn -Pct clean package docker:run``, wait for "done" message and log in at http://localhost:8080. -| (Username: ``dataverseAdmin``, Password: ``admin1``) - - .. _dev-build: Building