Pull upstream changes #7
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Align loading Nodesource keyring with musicbrainz-server repository. Enable Yarn 4 directly from Node.js 20 package as it is now part of it.
Upgrade MusicBrainz Server to Node 20 & Perl 5.34
Co-authored-by: Nicolás Tamargo <reosarevok@metabrainz.org>
It follows changes that have already been applied in production.
Or fix building db service with pg_amqp and new base image
The official Docker images for Postgres updated their base image to more
recent versions of Debian, breaking the compilation of pg_amqp.
This patch sets image variant, to prevent further breakage,
and compilation flag, to address the three following errors:
2.857 /usr/bin/clang-16 -Wno-ignored-attributes -fno-strict-aliasing -fwrapv -fexcess-precision=standard -Xclang -no-opaque-pointers -Wno-unused-command-line-argument -Wno-compound-token-split-by-macro -Wno-deprecated-non-prototype -O2 -I. -I./ -I/usr/include/postgresql/12/server -I/usr/include/postgresql/internal -Wdate-time -D_FORTIFY_SOURCE=2 -D_GNU_SOURCE -I/usr/include/libxml2 -flto=thin -emit-llvm -c -o src/pg_amqp.bc src/pg_amqp.c
2.989 src/pg_amqp.c:99:10: warning: 5 enumeration values not handled in switch: 'XACT_EVENT_PARALLEL_COMMIT', 'XACT_EVENT_PARALLEL_ABORT', 'XACT_EVENT_PRE_COMMIT'... [-Wswitch]
2.989 switch(event) {
2.989 ^~~~~
2.989 src/pg_amqp.c:140:21: error: parameter 'broker_id' was not declared, defaults to 'int'; ISO C99 and later do not support implicit int [-Wimplicit-int]
2.989 local_amqp_get_a_bs(broker_id) {
2.989 ^
2.989 src/pg_amqp.c:152:19: error: parameter 'broker_id' was not declared, defaults to 'int'; ISO C99 and later do not support implicit int [-Wimplicit-int]
2.989 local_amqp_get_bs(broker_id) {
2.989 ^
2.990 src/pg_amqp.c:239:23: error: parameter 'broker_id' was not declared, defaults to 'int'; ISO C99 and later do not support implicit int [-Wimplicit-int]
2.991 local_amqp_disconnect(broker_id) {
2.991 ^
2.994 1 warning and 3 errors generated.
2.997 make: *** [/usr/lib/postgresql/12/lib/pgxs/src/makefiles/../../src/Makefile.global:1076: src/pg_amqp.bc] Error 1
Even though some helper scripts won’t work anymore, these are not strictly needed. One can either adapt these or run commands manually.
chore: Make redis, postgres and rabbitmq hosts configurable via env variables
This will make it easier to compare/merge code in the future.
This method will work even if installing a more recent Perl from source.
This method will work even if installing a more recent Perl from source.
This method will work even if installing a more recent Perl from source.
Our (MetaBrainz) base image is based on Phusion base image which in turn is based on Ubuntu Jammy that comes with Perl 5.34.0 (using apt). To upgrade Perl with the same base image, we have to install from source. This patch is for development setup only. The same series of patches will be applied to mirrors when deploying Perl 5.38.2 in production. In deployment setup only, Perl modules have to be rebuilt as follows: ```bash docker-compose build musicbrainz docker-compose stop musicbrainz sudo rm -fr ../musicbrainz-server/perl_modules docker-compose up -d musicbrainz ```
Use the new instructions provided by `pg_upgrade`. Reference: https://www.postgresql.org/docs/14/release-14.html#id-1.11.6.17.5.12
Use the file `update_extensions.sql` provided by `pg_upgrade`.
MBS-13361 (mirror): Upgrade to PostgreSQL 16
It brings MusicBrainz database schema 29 and requires manual upgrade. See release notes for detailed upgrade steps.
The last service to still be fully built from source was musicbrainz. It was taking some time (as Perl modules were installed sequentially, and now even Perl itself is compiled from source), and happened to break after release time on unexpected upstream changes. This patch makes deploying mirror to use the prebuilt image `metabrainz/musicbrainz-docker-musicbrainz:<tag>` for the service musicbrainz by default instead of building it from the source. This image has to be built and pushed by the repository maintainers, using the newly added script, before each release. It is always trying to pull the latest upstream Docker image for security updates. For backward compatibility, the image tag isn’t set in the compose files directly, it is set in a Dockerfile instead. It makes it incidentally easier to customize the image for particular needs. It will be revisited when merging into musicbrainz-server repository.
The pg_amqp extension was still built from source for the db service. It was taking some time for compilation, and happened to break after release time on unexpected upstream changes. This patch makes deploying mirror to use the prebuilt image `metabrainz/musicbrainz-docker-db:<tag>` for the service db by default instead of building pg_amqp from the source. This image has to be built and pushed by the repository maintainers, using the newly added script, before each release. It is always trying to pull the latest upstream Docker image for security updates. For backward compatibility, the image tag isn’t set in the compose files directly, it is set in a Dockerfile instead. It makes it incidentally easier to customize the image for particular needs. It will be revisited when merging into musicbrainz-server repository.
It replaces previous builds 0 & 1 which were missing some changes.
MBVM-42: Prebuild images for the services db and musicbrainz
* schema-change-2024-q2: Bump musicbrainz build sequence to 2 MBVM-42 (2/2): Prebuild db service image MBVM-42 (1/2): Prebuild musicbrainz service image Upgrade Server to v-2024-05-13-schema-change (#278) Pass POSTGRES_VERSION ARG in development setup too Update installed extensions after Postgres upgrade Replace analyze_new_cluster with vacuumdb (PG 14) Fix copying Postgres upgrade script to container Fix showing server version during Postgres upgrade Fix compiling pg_amqp during Postgres upgrade Hide expected and harmless error message MBS-13361 (mirror): Upgrade to PostgreSQL 16 Install Perl module dependencies with Carton MBS-13358 (III): Upgrade Perl version to 5.38.2 Install cpanm from source instead of using apt-get Install local::lib using cpanm instead of apt-get
Also add notes to hopefully keep language lists in sync. It follows similar changes in musicbrainz-server for MBS-13584.
MBVM-98: Fix enabling mirror website translations
Previously, `SIR_LOCAL_ROOT` had to be set in the `.env` file, mandatorily. The default path is now `../sir` which assumes that `musicbrainz-docker` and `sir` have been cloned under the same parent directory.
Previously, running a different version of SIR was requiring either to modify the Dockerfile or to add a local compose file setting build arguments. This patch allows specifying the version of SIR, the Python version and the date of its base image, through build arguments and Docker Compose environment variables for SIR development setup only as needed: - `SIR_DEV_PYTHON_VERSION` (matching a `metabrainz/python` image tag) - `SIR_DEV_BASE_IMAGE_DATE` (matching a `metabrainz/python` image tag) - `SIR_DEV_VERSION` (informative only) Rebuilding the image is required after changing build arguments.
Prefixing variables with `SIR_DEV_` has two purposes: - It clarifies that it is used for SIR development setup only. - It allows to keep two separate configuration files for SIR, one for the current version, one for the development version, thus making it quicker to switch from one setup to the other.
It resolves the following error when running SIR with Python 3:
Unsupported scheme ''
From @amCap1712:
> the DSN is being read literally as "" by the configparser,
> not as an empty string as one would expect. I am guessing new
> versions of sentry are do stricter checking on DSN format
I tested it with Python 3.11 and then the still used Python 2.7.
Set up SIR development via Docker Compose environment variables
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.