Implement automated PGSQL DB migration. #63

yakutovicha · 2022-10-31T16:20:00Z

fixes #41

TODO:

Use pg_dumpall, not pg_dump.

opt/migrate_pgsql_db.sh

danielhollas · 2022-11-02T12:06:06Z

@yakutovicha I am a bit confused here. I believe that @unkcpz mentioned that aiida-prerequisites is no longer used to build the new AiiDA-2.0 images. So presumably the migration script needs to live in aiidalab-docker-stack repo?

yakutovicha · 2022-11-02T12:17:42Z

@yakutovicha I am a bit confused here. I believe that @unkcpz mentioned that aiida-prerequisites is no longer used to build the new AiiDA-2.0 images. So presumably the migration script needs to live in aiidalab-docker-stack repo?

Maybe I missed that part, sorry. For the moment aiida-core images are still based on aiida-prerequisites. We will at some point replace them, but we haven't agreed on the strategy yet.

Anyways, I am going to implement the migration here and then we port it elsewhere.

Co-authored-by: Daniel Hollas <danekhollas@gmail.com>

* Use pg_dumpall to dump the whole server. * Automatically determine the old and the new DB version. * Copy the migration script to the /opt folder.

yakutovicha · 2022-11-29T13:00:57Z

opt/start-postgres.sh

@@ -29,6 +29,11 @@ else
    if ! $running ; then
       echo "" > /home/${SYSTEM_USER}/.postgresql/logfile # empty log files
       rm -vf /home/${SYSTEM_USER}/.postgresql/postmaster.pid
-       ${PSQL_START_CMD}
+       ${PSQL_START_CMD} || cant_start=true


As @danielhollas suggested, we should distinguish the "migration required" case from other failures and properly handle it here.

danielhollas

Hi @yakutovicha,

Thanks for doing this annoying work. Just a couple of quick comments. I don't think I'll have time for a more thorough review, unfortunately.

danielhollas · 2022-11-30T13:54:30Z

opt/migrate-database.sh

+# This script is intended to be run in a container. Its job is to migrate the PGSQL database to a newer version.
+
+# The following variables are required:
+DB_FOLDER=/home/${SYSTEM_USER}/.postgresql


Just noting for future reference that in the new AiiDAlab docker stack we use ${NB_USER}

danielhollas · 2022-11-30T13:58:19Z

opt/migrate-database.sh

+# Part 1: dump the old database.
+
+# Make a backup of the database.
+mv ${DB_FOLDER} ${BACKUP_DB_FOLDER}


Where would this script show it's output?
I would put an echo here to show where the backup is

(although when the script is executed in the debug mode (set -x), each command is echoed, so I guess and extra echo make it a bit more clear what is happening)

Suggested change

mv ${DB_FOLDER} ${BACKUP_DB_FOLDER}

echo "Creating a DB backup to ${BACKUP_DB_FOLDER}"

mv ${DB_FOLDER} ${BACKUP_DB_FOLDER}

If we can retrieve the debug information, yes, I agree it is more clear to print this massage. But I don't know where to check these printed-out messages. Is it write to some log files?

danielhollas · 2022-11-30T13:59:33Z

Dockerfile

@@ -93,7 +93,7 @@ RUN cd /tmp && \
    conda clean --all -f -y

 # Install PostgreSQL in a dedicated conda environment.
-RUN conda create -c conda-forge -n pgsql postgresql=10 && conda clean --all -f -y
+RUN conda create -c conda-forge -n pgsql postgresql=12 && conda clean --all -f -y


Any reason why not jump straight to version 14? (just asking, I think it is probably better to do it incrementally)

danielhollas · 2022-11-30T14:05:49Z

opt/migrate-database.sh

+fi
+
+# Start the old version of PostgreSQL.
+conda run -n ${OLD_DB_CONDA_ENV_NAME} pg_ctl -D ${BACKUP_DB_FOLDER} -l ${BACKUP_DB_FOLDER}/logfile start


Using the ${BACKUP_DB_FOLDER} for this makes me a bit nervous, I don't think we should touch it at all during the migration. But perhaps I am misunderstanding....

Suggested change

conda run -n ${OLD_DB_CONDA_ENV_NAME} pg_ctl -D ${BACKUP_DB_FOLDER} -l ${BACKUP_DB_FOLDER}/logfile start

conda run -n ${OLD_DB_CONDA_ENV_NAME} pg_ctl -D ${DB_FOLDER} -l ${DB_FOLDER}/logfile start

Never mind, I misunderstood how the backup folder works.

danielhollas · 2022-11-30T14:06:53Z

opt/migrate-database.sh

+conda run -n ${OLD_DB_CONDA_ENV_NAME} pg_ctl -D ${BACKUP_DB_FOLDER} stop
+
+# Delete the environment with old version of PostgreSQL.
+conda env remove -n ${OLD_DB_CONDA_ENV_NAME}


Should this command be run only at the very and once we're sure the migration was successfull. Otherwise might be good to keep around for debugging?

unkcpz · 2023-01-10T09:19:04Z

I test using aiidalab-launch with this change. This is how I did the test:

(new image) build aiida-prerequisites image based on this PR -> build aiida-core 1.6.9 -> build aiidalab-docker-stack support-1.x.
(old image) Start an aiidalab instance of legacy version (running aiida-core 1.6.9 from the image aiidalab/aiidalab-docker-stack:22.8.1) and install QeApp and running a calculation to store more data to the database.
Create a new profile using the volume from old image but container using the new image to apply the change made in this PR.

I check the psql version and it is all correct as expected. However, verdi status show the error:

(base) aiida@fb1488a6d2fe:~$ verdi status
 ✔ config dir:  /home/aiida/.aiida
 ✔ profile:     On profile default
 ✔ repository:  /home/aiida/.aiida/repository/default
 ✘ postgres:    Unable to connect as aiida_qs_aiida_477d3dfc78a2042156110cb00ae3618f@localhost:5432
Error: OperationalError: could not connect to server: Connection refused
        Is the server running on host "localhost" (127.0.0.1) and accepting
        TCP/IP connections on port 5432?
could not connect to server: Cannot assign requested address
        Is the server running on host "localhost" (::1) and accepting
        TCP/IP connections on port 5432?

 ✔ rabbitmq:    Connected to RabbitMQ v3.8.14 as amqp://guest:guest@127.0.0.1:5672?heartbeat=600

I did anything wrong? How I confirm the migrate script is executed and the database is successfully migrated?

unkcpz · 2023-01-10T13:44:49Z

As mentioned by @yakutovicha in the meeting, the migration script needs to be run by hand.

I manually trigger the migration and it seems failed at last step with:

(1 row)

SET
SET
SET
COMMENT
CREATE EXTENSION
COMMENT
You are now connected to database "template1" as user "aiida".
SET
SET
SET
SET
SET
SET
 set_config
------------

(1 row)

SET
SET
SET
COMMENT
CREATE EXTENSION
COMMENT

psql:/home/aiida/pgsql-dump-10-20230110134114.sql:14: ERROR:  role "aiida" already exists

danielhollas · 2023-01-31T13:54:33Z

@unkcpz I am wondering if the error that you so could simply be ignored (e.g. user aiida already exist, never mind). Per
https://dba.stackexchange.com/questions/318379/role-postgres-already-exists-when-restoring-cluster

Implement automated PGSQL DB migration.

b14daf6

yakutovicha marked this pull request as draft October 31, 2022 16:21

danielhollas reviewed Nov 1, 2022

View reviewed changes

opt/migrate_pgsql_db.sh Outdated Show resolved Hide resolved

yakutovicha and others added 2 commits November 28, 2022 11:52

Update opt/migrate_pgsql_db.sh

66a1add

Co-authored-by: Daniel Hollas <danekhollas@gmail.com>

Working version of the PGSQL db migration script:

1d8b178

* Use pg_dumpall to dump the whole server. * Automatically determine the old and the new DB version. * Copy the migration script to the /opt folder.

yakutovicha requested review from unkcpz and danielhollas November 28, 2022 16:03

yakutovicha marked this pull request as ready for review November 28, 2022 16:03

yakutovicha commented Nov 29, 2022

View reviewed changes

danielhollas reviewed Nov 30, 2022

View reviewed changes

unkcpz mentioned this pull request Jan 23, 2023

Enable automated migration of the PostgreSQL database. aiidalab/aiidalab-docker-stack#25

Open

7 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement automated PGSQL DB migration. #63

Implement automated PGSQL DB migration. #63

yakutovicha commented Oct 31, 2022 •

edited

Loading

danielhollas commented Nov 2, 2022

yakutovicha commented Nov 2, 2022

yakutovicha Nov 29, 2022

danielhollas left a comment

danielhollas Nov 30, 2022

danielhollas Nov 30, 2022

unkcpz Jan 10, 2023

danielhollas Nov 30, 2022

danielhollas Nov 30, 2022

danielhollas Jan 31, 2023

danielhollas Nov 30, 2022

unkcpz commented Jan 10, 2023

unkcpz commented Jan 10, 2023 •

edited

Loading

danielhollas commented Jan 31, 2023

	mv ${DB_FOLDER} ${BACKUP_DB_FOLDER}
	echo "Creating a DB backup to ${BACKUP_DB_FOLDER}"
	mv ${DB_FOLDER} ${BACKUP_DB_FOLDER}

	conda run -n ${OLD_DB_CONDA_ENV_NAME} pg_ctl -D ${BACKUP_DB_FOLDER} -l ${BACKUP_DB_FOLDER}/logfile start
	conda run -n ${OLD_DB_CONDA_ENV_NAME} pg_ctl -D ${DB_FOLDER} -l ${DB_FOLDER}/logfile start

Implement automated PGSQL DB migration. #63

Are you sure you want to change the base?

Implement automated PGSQL DB migration. #63

Conversation

yakutovicha commented Oct 31, 2022 • edited Loading

danielhollas commented Nov 2, 2022

yakutovicha commented Nov 2, 2022

Choose a reason for hiding this comment

danielhollas left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

unkcpz commented Jan 10, 2023

unkcpz commented Jan 10, 2023 • edited Loading

danielhollas commented Jan 31, 2023

yakutovicha commented Oct 31, 2022 •

edited

Loading

unkcpz commented Jan 10, 2023 •

edited

Loading