Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add MariaDB migration testing #2170

Merged
merged 45 commits into from
Jan 9, 2024
Merged
Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
45 commits
Select commit Hold shift + click to select a range
01e80b9
update script to handle mariadb migration testing
strickvl Dec 20, 2023
996479b
WIP
strickvl Dec 20, 2023
07b11d3
Merge remote-tracking branch 'origin/develop' into feature/support-ma…
strickvl Dec 20, 2023
75d27ee
reformat temporary script
strickvl Dec 20, 2023
3875549
parameterize DB startup delay
strickvl Dec 20, 2023
84e9438
Merge branch 'develop' into feature/support-mariadb-db-migrations
strickvl Dec 20, 2023
040cec3
Merge remote-tracking branch 'origin/develop' into feature/support-ma…
strickvl Dec 20, 2023
eddf463
add a check for mysql vs mariadb
strickvl Dec 20, 2023
40deb3a
try another way to isolate mariadb vs mysql
strickvl Dec 20, 2023
e835ca4
try a different mariadb connection path
strickvl Dec 20, 2023
d422dc2
Merge branch 'develop' into feature/support-mariadb-db-migrations
strickvl Dec 20, 2023
7fb0e0e
Merge branch 'develop' into feature/support-mariadb-db-migrations
strickvl Jan 5, 2024
efb2663
Fix database connection URL in test-migrations.sh script
strickvl Jan 5, 2024
915ccb5
Adjust settings for MySQL and MariaDB engines in migration script
strickvl Jan 5, 2024
f02471f
log values
strickvl Jan 5, 2024
4337882
temporarily disable full CI runs for testing
strickvl Jan 5, 2024
0b6190f
print values
strickvl Jan 5, 2024
26e2950
Merge branch 'develop' into feature/support-mariadb-db-migrations
strickvl Jan 5, 2024
9b8ee69
trying a different connection approach
strickvl Jan 5, 2024
fed45a4
trying innodb
strickvl Jan 5, 2024
d4e6171
Add support for MariaDB in primary key modification
strickvl Jan 8, 2024
8b628ce
split mariadb from mysql db migration testing
strickvl Jan 8, 2024
dcc52af
Merge remote-tracking branch 'origin/develop' into feature/support-ma…
strickvl Jan 8, 2024
1bf185e
split mariadb from mysql db migration testing (core)
strickvl Jan 8, 2024
68b004f
Remove support for MariaDB and add version 0.53.1
strickvl Jan 8, 2024
e7a4e3e
Split out mariadb from mysql in migration testing
strickvl Jan 8, 2024
063733a
Merge remote-tracking branch 'origin/develop' into feature/support-ma…
strickvl Jan 8, 2024
0baaed9
add latest zenml version
strickvl Jan 8, 2024
5b6d084
Update migration testing scripts for different databases
strickvl Jan 8, 2024
7472dd9
restore the CI to working state
strickvl Jan 8, 2024
98d9e83
formatting
strickvl Jan 8, 2024
39c23d5
Manually merge develop
strickvl Jan 9, 2024
1f06225
use zenml init
strickvl Jan 9, 2024
c16c653
reformat yml
strickvl Jan 9, 2024
6d6df66
revert to copier instantiation
strickvl Jan 9, 2024
f24392d
add conditional check for version >= 0.43.0
strickvl Jan 9, 2024
aba3ee0
compare set of vals instead of complicated semantic check
strickvl Jan 9, 2024
15b2f92
remove extra cd
strickvl Jan 9, 2024
60ab65a
update mariadb script
strickvl Jan 9, 2024
cd0d854
add test flag to init command
strickvl Jan 9, 2024
d359e4d
remove (extra) analytics opt-out
strickvl Jan 9, 2024
00f8b9f
Add additional template versions for testing migrations
strickvl Jan 9, 2024
1f62ba3
enable the CI again
strickvl Jan 9, 2024
f84fab5
Merge branch 'develop' into feature/support-mariadb-db-migrations
strickvl Jan 9, 2024
36b8b64
Add comment explaining pre-template-versions
strickvl Jan 9, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 15 additions & 0 deletions .github/workflows/release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -66,6 +66,21 @@ jobs:
- name: Test migrations across versions
run: bash scripts/test-migrations.sh sqlite

mariadb-db-migration-testing:
runs-on: ubuntu-dind-runners

steps:
- name: Checkout code
uses: actions/checkout@v4.1.1

- name: Set up Python 3.9
uses: actions/setup-python@v4.8.0
with:
python-version: "3.9"

- name: Test migrations across versions
run: bash scripts/test-migrations.sh mariadb

publish-python-package:
needs: [setup-and-test, mlstacks-compatibility-check, sqlite-db-migration-testing, mysql-db-migration-testing]
uses: ./.github/workflows/publish_to_pypi.yml
Expand Down
18 changes: 18 additions & 0 deletions .github/workflows/testing_this_now.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
---
strickvl marked this conversation as resolved.
Show resolved Hide resolved
name: TEST MariaDB DB migrations
on:
pull_request:
types: [opened, synchronize, ready_for_review]
paths-ignore: [docs/**, docker/**, '*', '!pyproject.toml', '**.md']
jobs:
mariadb-db-migration-testing:
runs-on: ubuntu-dind-runners
steps:
- name: Checkout code
uses: actions/checkout@v4.1.1
- name: Set up Python 3.9
uses: actions/setup-python@v4.8.0
with:
python-version: '3.9'
- name: Test migrations across versions
run: bash scripts/test-migrations.sh mariadb
14 changes: 13 additions & 1 deletion scripts/test-migrations.sh
Original file line number Diff line number Diff line change
Expand Up @@ -41,9 +41,14 @@ if [ "$1" == "mysql" ]; then
echo "===== Testing MySQL ====="
# run a mysql instance in docker
docker run --name mysql -d -p 3306:3306 -e MYSQL_ROOT_PASSWORD=password mysql:latest

# mysql takes a while to start up
sleep 30
elif [ "$1" == "mariadb" ]; then
echo "===== Testing MariaDB ====="
# run a mariadb instance in docker
docker run --name mariadb -d -p 3306:3306 -e MYSQL_ROOT_PASSWORD=password mariadb:10.6
# mariadb takes a while to start up
sleep 30
strickvl marked this conversation as resolved.
Show resolved Hide resolved
fi

# List of versions to test
Expand Down Expand Up @@ -75,6 +80,8 @@ do

if [ "$1" == "mysql" ]; then
zenml connect --url mysql://127.0.0.1/zenml --username root --password password
elif [ "$1" == "mariadb" ]; then
zenml connect --url mysql://127.0.0.1/zenml --username root --password password
fi

# Run the tests for this version
Expand All @@ -100,13 +107,18 @@ pip3 install importlib_metadata

if [ "$1" == "mysql" ]; then
zenml connect --url mysql://127.0.0.1/zenml --username root --password password
elif [ "$1" == "mariadb" ]; then
zenml connect --url mysql://127.0.0.1/zenml --username root --password password
fi

run_tests_for_version current_branch_mysql

if [ "$1" == "mysql" ]; then
zenml disconnect
docker rm -f mysql
elif [ "$1" == "mariadb" ]; then
zenml disconnect
docker rm -f mariadb
fi

deactivate
Expand Down
Loading