Move towards MySQL 8.0 as the default template generation (vitessio#1…

…1153) (vitessio#11619) * Move towards MySQL 8.0 as the default template generation (vitessio#11153) * Move towards MySQL 8.0 as the default template generation This upgrades the remaining things to Ubuntu 20.04 and makes MySQL 8.0 the default we run tests against. We still have tests for MySQL 5.7 but those are now explicitly opted into. This should finish up the Ubuntu 20.04 upgrade and also makes things easier for the future when we need to upgrade again. Signed-off-by: Dirkjan Bussink <d.bussink@gmail.com> * CI: rename shard vtorc_8.0 to vtorc_5.7, change expected test output for 8.0 Signed-off-by: deepthi <deepthi@planetscale.com> * CI: increase timeout for 8.0 tests on the actual test step from 30 to 45 mins Signed-off-by: deepthi <deepthi@planetscale.com> * CI: increase timeout to 45 minutes for mysql57 tests too. We really only need this for vtorc, but I've made the change to the template so all tests get it. Signed-off-by: deepthi <deepthi@planetscale.com> * CI: fix vtorc test to work with both 5.7 and 8.0 Signed-off-by: deepthi <deepthi@planetscale.com> * CI: missed docker flag in mysql57 template, one more fix to vtorc test Signed-off-by: deepthi <deepthi@planetscale.com> * removing spaces from pb file Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com> * removing spaces in pb file part 2 Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com> Signed-off-by: Dirkjan Bussink <d.bussink@gmail.com> Signed-off-by: deepthi <deepthi@planetscale.com> Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com> Co-authored-by: deepthi <deepthi@planetscale.com> Co-authored-by: Rameez Sajwani <rameezwazirali@hotmail.com> * fixing template files to default to mysql80 Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com> * changing update statement to alter Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com> * Avoid deadlocks related to 0 receiver behavior (vitessio#10132) * Prevent deadlocks related to 0 receiver behavior Signed-off-by: Matt Lord <mattalord@gmail.com> * Update test tables to use poller_idx Signed-off-by: Matt Lord <mattalord@gmail.com> * Minor changes after mutex usage review in message manager + cache Signed-off-by: Matt Lord <mattalord@gmail.com> * Use atomics for receiver count and messages pending Signed-off-by: Matt Lord <mattalord@gmail.com> * Don't take exclusive lock when in fast path Signed-off-by: Matt Lord <mattalord@gmail.com> * Update tests to use the new recommended message table structure See: vitessio/website#1015 Signed-off-by: Matt Lord <mattalord@gmail.com> * Correct tests Signed-off-by: Matt Lord <mattalord@gmail.com> * Update e2e test to use new recommended table structure Signed-off-by: Matt Lord <mattalord@gmail.com> * Fix TestMessageStreamingPlan test Signed-off-by: Matt Lord <mattalord@gmail.com> * Fix godriver/TestStreamMessaging test Signed-off-by: Matt Lord <mattalord@gmail.com> * Split streamMu into streamProcessingMu and lastPollPositionMu Signed-off-by: Matt Lord <mattalord@gmail.com> * Poller cannot take main lock w/o having X stream processing lock Signed-off-by: Matt Lord <mattalord@gmail.com> * Improve the comments a bit Signed-off-by: Matt Lord <mattalord@gmail.com> * Hold the main mutex during Add This is for safe concurrency with the last receiver unsubscribing Signed-off-by: Matt Lord <mattalord@gmail.com> * Changes after pair reviewing with Sugu Signed-off-by: Matt Lord <mattalord@gmail.com> * Use my GitHub handle for the self reference Signed-off-by: Matt Lord <mattalord@gmail.com> Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com> * remove unwanted sleep Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com> * Fix failures Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com> * Fix backup tests Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com> * fix vtgate_gen4 error Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com> * upgrading to mysql 8.0 using vinalla mysql Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com> * pin to specific version 8.0.25 Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com> * Try to pin mysql version to 8.0.25 using tar file Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com> * Fix bug in last commit for 8.0.25 Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com> * fixing template files to use mysql8.0.25 Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com> * fixing community version Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com> * removing all mysql version before installation Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com> * moving cluster 11 to self host Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com> * trying different combination since mysql is not getting installed correctly Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com> * use tar instead of deb file Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com> * remove mysql stop statement Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com> * setting vt_mysql_root Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com> * moving export to right place Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com> * change template to accomodate tar file download Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com> * fixing mysql80 cluster Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com> * Adding mysql to the path env variable Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com> * fixing shardedpitr_tls test Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com> * fix upgrade-downgrade test Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com> * adjust epected AUTO_INCREMENT value Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * removed 'expected_table_structure' files because there are different outputs in mysql57 and in mysql80 Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * adding mysql version in workflow logs Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com> * move to utuntu 18 for upgrade-downgrade test Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com> Signed-off-by: Dirkjan Bussink <d.bussink@gmail.com> Signed-off-by: deepthi <deepthi@planetscale.com> Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com> Signed-off-by: Matt Lord <mattalord@gmail.com> Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> Co-authored-by: Dirkjan Bussink <d.bussink@gmail.com> Co-authored-by: deepthi <deepthi@planetscale.com> Co-authored-by: Matt Lord <mattalord@gmail.com> Co-authored-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
rohit-nayak-ps · Nov 22, 2022 · 8bc1fbc · 8bc1fbc
1 parent f197542
commit 8bc1fbc
Show file tree

Hide file tree

Showing 137 changed files with 6,785 additions and 968 deletions.
diff --git a/.github/workflows/cluster_endtoend_11.yml b/.github/workflows/cluster_endtoend_11.yml
@@ -9,7 +9,7 @@ concurrency:
 jobs:
   build:
     name: Run endtoend tests on Cluster (11)
-    runs-on: ubuntu-18.04
+    runs-on: ubuntu-20.04
 
     steps:
     - name: Check if workflow needs to be skipped
@@ -44,30 +44,54 @@ jobs:
     - name: Get dependencies
       if: steps.skip-workflow.outputs.skip-workflow == 'false'
       run: |
+        
+        # Get key to latest MySQL repo
+        sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 467B942D3A79BD29
+
         sudo apt-get update
-        sudo apt-get install -y mysql-server mysql-client make unzip g++ etcd curl git wget eatmydata
+        # stop any existing running instance of mysql
         sudo service mysql stop
-        sudo service etcd stop
         sudo ln -s /etc/apparmor.d/usr.sbin.mysqld /etc/apparmor.d/disable/
         sudo apparmor_parser -R /etc/apparmor.d/usr.sbin.mysqld
+        sudo systemctl stop apparmor
+
+        # Uninstall any previously installed MySQL first
+        sudo DEBIAN_FRONTEND="noninteractive" apt-get remove -y --purge mysql-\*
+        sudo apt-get -y autoremove
+        sudo apt-get -y autoclean
+        sudo deluser mysql
+        sudo rm -rf /var/lib/mysql
+        sudo rm -rf /etc/mysql
+
+        # install necessary tools
+        sudo apt-get install -y make unzip g++ etcd curl git wget eatmydata xz-utils libmecab2
+        sudo service etcd stop
+
+        # Download mysql8.0.25 (pin it)
+        wget -c https://downloads.mysql.com/archives/get/p/23/file/mysql-8.0.25-linux-glibc2.17-x86_64-minimal.tar.xz
+        # Untar the bundle. It will have all necessary deb
+        sudo tar xf mysql-8.0.25-linux-glibc2.17-x86_64-minimal.tar.xz -v -C /usr
+        echo 'mysql version: '
+        sudo /usr/mysql-8.0.25-linux-glibc2.17-x86_64-minimal/bin/mysql --version
+        
+        # configure rest
+        sudo service etcd stop
         go mod download
 
         # install JUnit report formatter
         go get -u github.com/vitessio/go-junit-report@HEAD
 
-        wget https://repo.percona.com/apt/percona-release_latest.$(lsb_release -sc)_all.deb
-        sudo apt-get install -y gnupg2
-        sudo dpkg -i percona-release_latest.$(lsb_release -sc)_all.deb
-        sudo apt-get update
-        sudo apt-get install percona-xtrabackup-24
-
     - name: Run cluster endtoend test
       if: steps.skip-workflow.outputs.skip-workflow == 'false'
-      timeout-minutes: 30
+      timeout-minutes: 45
       run: |
+        
+        export VT_MYSQL_ROOT="/usr/mysql-8.0.25-linux-glibc2.17-x86_64-minimal"
+        export PATH="/usr/mysql-8.0.25-linux-glibc2.17-x86_64-minimal/bin:$PATH"
+        
         source build.env
 
         set -x
 
         # run the tests however you normally do, then produce a JUnit XML file
-        eatmydata -- go run test.go -docker=false -follow -shard 11
+        eatmydata -- go run test.go -docker=false -follow -shard 11
diff --git a/.github/workflows/cluster_endtoend_12.yml b/.github/workflows/cluster_endtoend_12.yml
@@ -9,7 +9,7 @@ concurrency:
 jobs:
   build:
     name: Run endtoend tests on Cluster (12)
-    runs-on: ubuntu-18.04
+    runs-on: ubuntu-20.04
 
     steps:
     - name: Check if workflow needs to be skipped
@@ -44,30 +44,54 @@ jobs:
     - name: Get dependencies
       if: steps.skip-workflow.outputs.skip-workflow == 'false'
       run: |
+        
+        # Get key to latest MySQL repo
+        sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 467B942D3A79BD29
+
         sudo apt-get update
-        sudo apt-get install -y mysql-server mysql-client make unzip g++ etcd curl git wget eatmydata
+        # stop any existing running instance of mysql
         sudo service mysql stop
-        sudo service etcd stop
         sudo ln -s /etc/apparmor.d/usr.sbin.mysqld /etc/apparmor.d/disable/
         sudo apparmor_parser -R /etc/apparmor.d/usr.sbin.mysqld
+        sudo systemctl stop apparmor
+
+        # Uninstall any previously installed MySQL first
+        sudo DEBIAN_FRONTEND="noninteractive" apt-get remove -y --purge mysql-\*
+        sudo apt-get -y autoremove
+        sudo apt-get -y autoclean
+        sudo deluser mysql
+        sudo rm -rf /var/lib/mysql
+        sudo rm -rf /etc/mysql
+
+        # install necessary tools
+        sudo apt-get install -y make unzip g++ etcd curl git wget eatmydata xz-utils libmecab2
+        sudo service etcd stop
+
+        # Download mysql8.0.25 (pin it)
+        wget -c https://downloads.mysql.com/archives/get/p/23/file/mysql-8.0.25-linux-glibc2.17-x86_64-minimal.tar.xz
+        # Untar the bundle. It will have all necessary deb
+        sudo tar xf mysql-8.0.25-linux-glibc2.17-x86_64-minimal.tar.xz -v -C /usr
+        echo 'mysql version: '
+        sudo /usr/mysql-8.0.25-linux-glibc2.17-x86_64-minimal/bin/mysql --version
+        
+        # configure rest
+        sudo service etcd stop
         go mod download
 
         # install JUnit report formatter
         go get -u github.com/vitessio/go-junit-report@HEAD
 
-        wget https://repo.percona.com/apt/percona-release_latest.$(lsb_release -sc)_all.deb
-        sudo apt-get install -y gnupg2
-        sudo dpkg -i percona-release_latest.$(lsb_release -sc)_all.deb
-        sudo apt-get update
-        sudo apt-get install percona-xtrabackup-24
-
     - name: Run cluster endtoend test
       if: steps.skip-workflow.outputs.skip-workflow == 'false'
-      timeout-minutes: 30
+      timeout-minutes: 45
       run: |
+        
+        export VT_MYSQL_ROOT="/usr/mysql-8.0.25-linux-glibc2.17-x86_64-minimal"
+        export PATH="/usr/mysql-8.0.25-linux-glibc2.17-x86_64-minimal/bin:$PATH"
+        
         source build.env
 
         set -x
 
         # run the tests however you normally do, then produce a JUnit XML file
-        eatmydata -- go run test.go -docker=false -follow -shard 12
+        eatmydata -- go run test.go -docker=false -follow -shard 12
diff --git a/.github/workflows/cluster_endtoend_13.yml b/.github/workflows/cluster_endtoend_13.yml
@@ -9,7 +9,7 @@ concurrency:
 jobs:
   build:
     name: Run endtoend tests on Cluster (13)
-    runs-on: ubuntu-18.04
+    runs-on: ubuntu-20.04
 
     steps:
     - name: Check if workflow needs to be skipped
@@ -44,30 +44,54 @@ jobs:
     - name: Get dependencies
       if: steps.skip-workflow.outputs.skip-workflow == 'false'
       run: |
+        
+        # Get key to latest MySQL repo
+        sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 467B942D3A79BD29
+
         sudo apt-get update
-        sudo apt-get install -y mysql-server mysql-client make unzip g++ etcd curl git wget eatmydata
+        # stop any existing running instance of mysql
         sudo service mysql stop
-        sudo service etcd stop
         sudo ln -s /etc/apparmor.d/usr.sbin.mysqld /etc/apparmor.d/disable/
         sudo apparmor_parser -R /etc/apparmor.d/usr.sbin.mysqld
+        sudo systemctl stop apparmor
+
+        # Uninstall any previously installed MySQL first
+        sudo DEBIAN_FRONTEND="noninteractive" apt-get remove -y --purge mysql-\*
+        sudo apt-get -y autoremove
+        sudo apt-get -y autoclean
+        sudo deluser mysql
+        sudo rm -rf /var/lib/mysql
+        sudo rm -rf /etc/mysql
+
+        # install necessary tools
+        sudo apt-get install -y make unzip g++ etcd curl git wget eatmydata xz-utils libmecab2
+        sudo service etcd stop
+
+        # Download mysql8.0.25 (pin it)
+        wget -c https://downloads.mysql.com/archives/get/p/23/file/mysql-8.0.25-linux-glibc2.17-x86_64-minimal.tar.xz
+        # Untar the bundle. It will have all necessary deb
+        sudo tar xf mysql-8.0.25-linux-glibc2.17-x86_64-minimal.tar.xz -v -C /usr
+        echo 'mysql version: '
+        sudo /usr/mysql-8.0.25-linux-glibc2.17-x86_64-minimal/bin/mysql --version
+        
+        # configure rest
+        sudo service etcd stop
         go mod download
 
         # install JUnit report formatter
         go get -u github.com/vitessio/go-junit-report@HEAD
 
-        wget https://repo.percona.com/apt/percona-release_latest.$(lsb_release -sc)_all.deb
-        sudo apt-get install -y gnupg2
-        sudo dpkg -i percona-release_latest.$(lsb_release -sc)_all.deb
-        sudo apt-get update
-        sudo apt-get install percona-xtrabackup-24
-
     - name: Run cluster endtoend test
       if: steps.skip-workflow.outputs.skip-workflow == 'false'
-      timeout-minutes: 30
+      timeout-minutes: 45
       run: |
+        
+        export VT_MYSQL_ROOT="/usr/mysql-8.0.25-linux-glibc2.17-x86_64-minimal"
+        export PATH="/usr/mysql-8.0.25-linux-glibc2.17-x86_64-minimal/bin:$PATH"
+        
         source build.env
 
         set -x
 
         # run the tests however you normally do, then produce a JUnit XML file
-        eatmydata -- go run test.go -docker=false -follow -shard 13
+        eatmydata -- go run test.go -docker=false -follow -shard 13
diff --git a/.github/workflows/cluster_endtoend_14.yml b/.github/workflows/cluster_endtoend_14.yml
@@ -9,7 +9,7 @@ concurrency:
 jobs:
   build:
     name: Run endtoend tests on Cluster (14)
-    runs-on: ubuntu-18.04
+    runs-on: ubuntu-20.04
 
     steps:
     - name: Check if workflow needs to be skipped
@@ -44,30 +44,54 @@ jobs:
     - name: Get dependencies
       if: steps.skip-workflow.outputs.skip-workflow == 'false'
       run: |
+        
+        # Get key to latest MySQL repo
+        sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 467B942D3A79BD29
+
         sudo apt-get update
-        sudo apt-get install -y mysql-server mysql-client make unzip g++ etcd curl git wget eatmydata
+        # stop any existing running instance of mysql
         sudo service mysql stop
-        sudo service etcd stop
         sudo ln -s /etc/apparmor.d/usr.sbin.mysqld /etc/apparmor.d/disable/
         sudo apparmor_parser -R /etc/apparmor.d/usr.sbin.mysqld
+        sudo systemctl stop apparmor
+
+        # Uninstall any previously installed MySQL first
+        sudo DEBIAN_FRONTEND="noninteractive" apt-get remove -y --purge mysql-\*
+        sudo apt-get -y autoremove
+        sudo apt-get -y autoclean
+        sudo deluser mysql
+        sudo rm -rf /var/lib/mysql
+        sudo rm -rf /etc/mysql
+
+        # install necessary tools
+        sudo apt-get install -y make unzip g++ etcd curl git wget eatmydata xz-utils libmecab2
+        sudo service etcd stop
+
+        # Download mysql8.0.25 (pin it)
+        wget -c https://downloads.mysql.com/archives/get/p/23/file/mysql-8.0.25-linux-glibc2.17-x86_64-minimal.tar.xz
+        # Untar the bundle. It will have all necessary deb
+        sudo tar xf mysql-8.0.25-linux-glibc2.17-x86_64-minimal.tar.xz -v -C /usr
+        echo 'mysql version: '
+        sudo /usr/mysql-8.0.25-linux-glibc2.17-x86_64-minimal/bin/mysql --version
+        
+        # configure rest
+        sudo service etcd stop
         go mod download
 
         # install JUnit report formatter
         go get -u github.com/vitessio/go-junit-report@HEAD
 
-        wget https://repo.percona.com/apt/percona-release_latest.$(lsb_release -sc)_all.deb
-        sudo apt-get install -y gnupg2
-        sudo dpkg -i percona-release_latest.$(lsb_release -sc)_all.deb
-        sudo apt-get update
-        sudo apt-get install percona-xtrabackup-24
-
     - name: Run cluster endtoend test
       if: steps.skip-workflow.outputs.skip-workflow == 'false'
-      timeout-minutes: 30
+      timeout-minutes: 45
       run: |
+        
+        export VT_MYSQL_ROOT="/usr/mysql-8.0.25-linux-glibc2.17-x86_64-minimal"
+        export PATH="/usr/mysql-8.0.25-linux-glibc2.17-x86_64-minimal/bin:$PATH"
+        
         source build.env
 
         set -x
 
         # run the tests however you normally do, then produce a JUnit XML file
-        eatmydata -- go run test.go -docker=false -follow -shard 14
+        eatmydata -- go run test.go -docker=false -follow -shard 14
diff --git a/.github/workflows/cluster_endtoend_15.yml b/.github/workflows/cluster_endtoend_15.yml
@@ -9,7 +9,7 @@ concurrency:
 jobs:
   build:
     name: Run endtoend tests on Cluster (15)
-    runs-on: ubuntu-18.04
+    runs-on: ubuntu-20.04
 
     steps:
     - name: Check if workflow needs to be skipped
@@ -44,30 +44,54 @@ jobs:
     - name: Get dependencies
       if: steps.skip-workflow.outputs.skip-workflow == 'false'
       run: |
+        
+        # Get key to latest MySQL repo
+        sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 467B942D3A79BD29
+
         sudo apt-get update
-        sudo apt-get install -y mysql-server mysql-client make unzip g++ etcd curl git wget eatmydata
+        # stop any existing running instance of mysql
         sudo service mysql stop
-        sudo service etcd stop
         sudo ln -s /etc/apparmor.d/usr.sbin.mysqld /etc/apparmor.d/disable/
         sudo apparmor_parser -R /etc/apparmor.d/usr.sbin.mysqld
+        sudo systemctl stop apparmor
+
+        # Uninstall any previously installed MySQL first
+        sudo DEBIAN_FRONTEND="noninteractive" apt-get remove -y --purge mysql-\*
+        sudo apt-get -y autoremove
+        sudo apt-get -y autoclean
+        sudo deluser mysql
+        sudo rm -rf /var/lib/mysql
+        sudo rm -rf /etc/mysql
+
+        # install necessary tools
+        sudo apt-get install -y make unzip g++ etcd curl git wget eatmydata xz-utils libmecab2
+        sudo service etcd stop
+
+        # Download mysql8.0.25 (pin it)
+        wget -c https://downloads.mysql.com/archives/get/p/23/file/mysql-8.0.25-linux-glibc2.17-x86_64-minimal.tar.xz
+        # Untar the bundle. It will have all necessary deb
+        sudo tar xf mysql-8.0.25-linux-glibc2.17-x86_64-minimal.tar.xz -v -C /usr
+        echo 'mysql version: '
+        sudo /usr/mysql-8.0.25-linux-glibc2.17-x86_64-minimal/bin/mysql --version
+        
+        # configure rest
+        sudo service etcd stop
         go mod download
 
         # install JUnit report formatter
         go get -u github.com/vitessio/go-junit-report@HEAD
 
-        wget https://repo.percona.com/apt/percona-release_latest.$(lsb_release -sc)_all.deb
-        sudo apt-get install -y gnupg2
-        sudo dpkg -i percona-release_latest.$(lsb_release -sc)_all.deb
-        sudo apt-get update
-        sudo apt-get install percona-xtrabackup-24
-
     - name: Run cluster endtoend test
       if: steps.skip-workflow.outputs.skip-workflow == 'false'
-      timeout-minutes: 30
+      timeout-minutes: 45
       run: |
+        
+        export VT_MYSQL_ROOT="/usr/mysql-8.0.25-linux-glibc2.17-x86_64-minimal"
+        export PATH="/usr/mysql-8.0.25-linux-glibc2.17-x86_64-minimal/bin:$PATH"
+        
         source build.env
 
         set -x
 
         # run the tests however you normally do, then produce a JUnit XML file
-        eatmydata -- go run test.go -docker=false -follow -shard 15
+        eatmydata -- go run test.go -docker=false -follow -shard 15