Launch additional services in separate container

[csadorf]

The additional services (like Postgres or RabbitMQ) should be started as additional Docker services. This could for example be achieved via docker-compose.

AiiDAlab-launch would likely need some major revision to then support this, since it currently uses docker directly to manage the container(s).

Sub-tasks:

• Create proof-of-concept docker-compose file that demonstrates the approach
• Evaluate how we want to manage multiple containers within the context of aiidalab-launch (docker-compose, manually, external tool ?)
• Revise aiidalab-launch to manage multiple containers based on approach identified in previous step

Dependencies

• Base AiiDAlab image on jupyter/base-notebook image and support AiiDA 2.0 aiidalab-docker-stack#254

[csadorf]

It looks like a similar approach for AiiDA only was implemented here: https://github.com/aiidateam/aiida_docker_compose, however the repository appears unmaintained.

[csadorf]

@yakutovicha I've made a proposal on how to break up this task into multiple sub-steps in the description above. Please feel free to propose revisions to those. Thank you!

[yakutovicha]

I think I have the first somehow working version.

Here is a bunch of PRs that I am preparing in order to switch to the docker compose approach:

• Update to the docker container Base AiiDAlab image on jupyter/base-notebook image and support AiiDA 2.0 aiidalab-docker-stack#254
• Update to the aiidalab-home app Switch to AiiDA kernel aiidalab-home#90
• Switch to docker-compose approach in aiidalab-launch Switch to docker-compose approach. #127

More information will follow.

[csadorf]

@yakutovicha Could you please briefly document the remaining tasks for this?

[yakutovicha]

So, here is the report of what has been done and what needs to be done:

[DONE]
To my knowledge, all the features that are available in the current stable version of the container are implemented in the aforementioned PRs. So from the user perspective, it is almost the same as before with one exception - AiiDA is now installed in a separate conda environment named aiida.

[TODO]

• Implement handling of the new docker-compose approach within the aiidalab-launch package.
• Create a rough estimate about how much of the API will need to be changed (which concepts are invalidated)
• Decide whether to keep the old approach of single-docker-image and the new docker compose approach to coexist at least for the transition period.
• Identify the API changes and ensure a smooth transition from a single-docker-image to the docker-compose.
• Update the tests, making sure they pass.

[csadorf]

I would suggest to add a TODO item "Create rough estimate about how much of the API will need to be changed (which concepts are invalidated)". I would like to see some planning of those changes before we move into the implementation phase (even prototyping).

[yakutovicha]

Edit by @csadorf : this task list is no longer valid.

Here is what needs to be done:

• In all the apps - switch to the AiiDA kernel. It can be done now, as it works even if the selected kernel is absent. Jupyter switches to the default kernel. (Switch to AiiDA kernel aiidalab-home#90)
  • Add an explicit check to the home app that checks the notebook kernel. The problem is that it is not well defined which notebook will be executed, so we need to check every notebook within the app. (Add an explicit check of the notebook kernel. aiidalab#293)
• Make sure the PSQL migration is implemented (see Enable automated migration of the PostgreSQL database. aiidalab-docker-stack#25)
  • Implement a migration in the base docker image (Enable automated migration of PostgreSQL database. aiidateam/aiida-prerequisites#41)
  • Integrate into a visual interface that allows users to perform the data migration (Provide an interface to perform the PostgreSQL database migration aiidalab-home#94)
• Update and release the new aiidalab-docker-stack container based on jupyter-notebooks (Base AiiDAlab image on jupyter/base-notebook image and support AiiDA 2.0 aiidalab-docker-stack#254)
• Write documentation that guides through the update process.
  • That also involves either automation or description of steps to migrate users' data from folder to Docker volume.

[csadorf]

@yakutovicha Can you please create an issue for each individual unit of work and link them here as well as in the roadmap?

[unkcpz]

I add the stack image aiidalab/aiidalab-docker-stack#319 and tested that it can be started with the services.