fix bugs of stoneatom#309 and stoneatom#310 (stoneatom#313)

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> fix(mtr):mtr result stonedb change to tianmu(stoneatom#359) (stoneatom#368) * fix(mtr):mysqlpump*.test sys_stoendb to sys_tianmu fix(mtr):fix main result stonedb to tianmu fix(mtr):mv innodb_fts.ngram_1 to test/innodb_fts fix(mtr):fix mtr funcs_1 stonedb to tianmu fix(mtr):fix innodb mtr result stonedb change to tianmu fix(mtr):fix perfschema mtr result stonedb change to tianmu * fix(mtr):fix mtr stoendb change to tianmu Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> fix(tianmu): fix group_concat_max_len warning (stoneatom#338) (stoneatom#363) Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> docs(stonedb): update docs and fix bugs (stoneatom#373) * docs(Quick Deployment and Compiling Methods): update docs Make the document more user-friendly for new users * fix(docs): Let the official website pictures can be adaptive * Update compile-using-ubuntu2004.md In the install make step, add the go to directory：cd make-3.82 * Update compile-using-redhat7.md In the install make step, add the go to directory：cd make-3.82 * Update compile-using-centos7.md In the install make step, add the go to directory：cd make-3.82 Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: haitaoguan <105625912+haitaoguan@users.noreply.github.com> feat(doc):doc add innodb backup to stonedb demonstrate（stoneatom#353） (stoneatom#355) * feat(doc):doc add innodb backup to stonedb demonstrate * fix(doc):fix mydumper and mysqldump doc Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> feat(innodb): To support compiling innodb as C++ 17(stoneatom#375) (stoneatom#376) To support innodb is compiled as C++ 17. For now, tianmu is compiling as C++17, and C++17 has became mainstream of C++ standard. Therefore, now, we star to support to compile Innodb as C++17 in stonedb. This is backport from MySQL, which commit id is: 1a8a111d8f855a31d0aeffc8f02309b2b82dd410 and bug# is #32907274 Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> fix stonedb for optimize exists subquery and deep memory copy stoneatom#369 (stoneatom#330, stoneatom#329, stoneatom#340, stoneatom#332) (stoneatom#369) * fix stonedb for optimize exists subquery and deep memcopy copy * Update parameterized_filter.cpp fix ParameterizedFilter deep copy mind memory leak * fix subquery when this is in sql * fix code readability and defensive * use shallow memory when paral hash join by class DimensionGroupMultiMaterialized Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> feat(stonedb): To support gcc 9 and adds `WITH_ROCKSDB` and `WITH_MARISA` to specify where (stoneatom#377) rocksdb and marisa installed, respectively. Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> fix(mtr): Add test cases for NULL value select on UNIQUE constraint(stoneatom#281) (stoneatom#379) * fix(core): Double prepare on lex unit for cts with union(stoneatom#226) fix(core): MTR + Double prepare on lex unit for ctas with union(stoneatom#226) * fix(mtr): Add test cases for NULL value select on UNIQUE constraint(stoneatom#281) Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> fix bug stoneatom#342 (stoneatom#346) Co-authored-by: dfx <duanfuxiang> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Doc(stonedb): Added extensions deccriptions in topics related to backup and recovery for O&M Guide(stoneatom#360) (stoneatom#361) * updated some code snippets in Install RocksDB step Signed-off-by: xuejiao-joy <821682573@qq.com> * some description here Signed-off-by: xuejiao-joy <821682573@qq.com> Co-authored-by: Yuting <34496563+Nliver@users.noreply.github.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> fix(mtr): Add more ctas queries for the same bug-fix from code-analysis perspectives(stoneatom#226)
xiaoguangye · Aug 11, 2022 · 6069120 · 6069120
1 parent 0b334e0
commit 6069120
Show file tree

Hide file tree

Showing 131 changed files with 2,728 additions and 1,327 deletions.
diff --git a/.github/workflows/integration-compile-mtr.yml b/.github/workflows/integration-compile-mtr.yml
@@ -32,7 +32,7 @@ env:
 
 jobs:
   stonedb-build:
-    runs-on: ubuntu-18.04
+    runs-on: ubuntu-latest
 
     steps:
       - uses: actions/checkout@v3
@@ -49,34 +49,14 @@ jobs:
           
       - name: install libs
         run: |
-          sudo git clone https://github.com/hustjieke/stonedb-libs.git
+          sudo git clone https://github.com/stoneatom/stonedb-libs.git
           cd stonedb-libs
           sudo tar -zxvf stonedb-boost.tar.gz
           sudo tar -zxvf stonedb-gcc-rocksdb.tar.gz
           sudo tar -zxvf stonedb-marisa.tar.gz
-          cd gcc7.3
-          sudo tar -zxvf bin.tar.gz
-          sudo tar -zxvf include.tar.gz
-          sudo tar -zxvf lib64.tar.gz
-          sudo tar -zxvf lib.tar.gz
-          sudo tar -zxvf share.tar.gz
-          cd libexec/gcc/x86_64-pc-linux-gnu/7.3.0
-          sudo tar -zxvf all.tar.gz
-          sudo tar -zxvf cc1plus.tar.gz
-          sudo tar -zxvf cc1.tar.gz
-          cd - && cd ..
           sudo cp -r stonedb-boost /usr/local/stonedb-boost
-          sudo cp -r stonedb-boost/lib /usr/local/stonedb-boost/include
           sudo cp -r stonedb-gcc-rocksdb /usr/local/stonedb-gcc-rocksdb
           sudo cp -r stonedb-marisa /usr/local/stonedb-marisa
-          sudo cp -r gcc7.3 /usr/local/gcc7.3
-          sudo rm /usr/bin/gcc
-          sudo ln -s /usr/local/gcc7.3/bin/gcc /usr/bin/gcc
-          sudo rm /usr/bin/g++
-          sudo ln -s /usr/local/gcc7.3/bin/g++ /usr/bin/g++
-          sudo rm /usr/bin/c++
-          sudo ln -s /usr/local/gcc7.3/bin/c++ /usr/bin/c++
-          cd ..
        
       - name: compile stonedb
         run: |
@@ -96,7 +76,9 @@ jobs:
           -DEXTRA_CHARSETS=all \
           -DDEFAULT_CHARSET=utf8 \
           -DDEFAULT_COLLATION=utf8_general_ci \
-          -DWITH_BOOST=/usr/local/stonedb-boost/include
+          -DWITH_BOOST=/usr/local/stonedb-boost \
+          -DWITH_ROCKSDB=/usr/local/stonedb-gcc-rocksdb \
+          -DWITH_MARISA=/usr/local/stonedb-marisa
           sudo make VERBOSE=1 -j`nproc`
           sudo make install -j`nproc`
 

diff --git a/.gitignore b/.gitignore
@@ -113,3 +113,4 @@ CMakeFiles
 CMakeFiles/*
 CTestTestfile.cmake
 COPYING
+.vs/
diff --git a/CMakeLists.txt b/CMakeLists.txt
@@ -303,6 +303,29 @@ OPTION(WITH_AUTHENTICATION_LDAP
 OPTION(FORCE_UNSUPPORTED_COMPILER "Disable compiler version checks" OFF)
 MARK_AS_ADVANCED(WITHOUT_SERVER DISABLE_SHARED FORCE_UNSUPPORTED_COMPILER)
 
+#for boost, to check boost params.
+INCLUDE(cmake/boost.cmake)
+#Link directories under boost, etc.
+IF (WITH_BOOST)
+  INCLUDE_DIRECTORIES("${WITH_BOOST}/include")
+  LINK_DIRECTORIES("${WITH_BOOST}/lib/")
+ENDIF ()
+
+#for rocksdb, to check rocksdb
+INCLUDE(cmake/rocksdb.cmake)
+#Link directories under rocksdb, etc.
+IF (WITH_ROCKSDB)
+  INCLUDE_DIRECTORIES("${WITH_ROCKSDB}" "${WITH_ROCKSDB}/include")
+  LINK_DIRECTORIES("${WITH_ROCKSDB}")
+ENDIF ()
+
+#for marisa, to check marisa
+INCLUDE(cmake/marisa.cmake)
+#Link directories under marisa, etc.
+IF (WITH_MARISA)
+  INCLUDE_DIRECTORIES("${WITH_MARISA}" "${WITH_MARISA}/include")
+  LINK_DIRECTORIES("${WITH_MARISA}" "${WITH_MARISA}/lib")
+ENDIF ()
 
 include(CheckCSourceCompiles)
 include(CheckCXXSourceCompiles)
@@ -481,6 +504,13 @@ IF (USE_CXX11 AND CMAKE_COMPILER_IS_GNUCC AND CMAKE_COMPILER_IS_GNUCXX)
           "${CMAKE_EXE_LINKER_FLAGS_DEBUG} -static-libstdc++ -static-libgcc")
 ENDIF()
 
+IF(WITH_BOOST AND WITH_ROCKSDB AND WITH_MARISA)
+SET(CMAKE_INSTALL_RPATH "${CMAKE_INSTALL_RPATH};${WITH_BOOST}/lib;${WITH_ROCKSDB};${WITH_MARISA}/lib")
+ELSE()
+SET(CMAKE_INSTALL_RPATH "${CMAKE_INSTALL_RPATH};${BOOST_INCLUDE_DIR};${ROCKSDB_INCLUDE_DIR};${MARISA_INCLUDE_DIR}")
+ENDIF()
+MESSAGE(STATUS "TO SET INSTALL RPATH FOR STONEDB: ${CMAKE_INSTALL_RPATH}")
+
 OPTION(ENABLED_LOCAL_INFILE
  "If we should enable LOAD DATA LOCAL by default" ${IF_WIN})
 MARK_AS_ADVANCED(ENABLED_LOCAL_INFILE)
@@ -565,8 +595,6 @@ ELSE()
   SET(DEFAULT_TMPDIR "\"${TMPDIR}\"")
 ENDIF()
 
-INCLUDE(cmake/boost.cmake)
-
 IF (CMAKE_SYSTEM_NAME MATCHES "Linux")
   OPTION(WITH_SYSTEMD "Enable installation of systemd support files" OFF)
   IF (WITH_SYSTEMD)
@@ -584,6 +612,7 @@ INCLUDE(configure.cmake)
 # Common defines and includes
 ADD_DEFINITIONS(-DHAVE_CONFIG_H)
 ADD_DEFINITIONS(-DTIANMU)
+ADD_DEFINITIONS(-D_GLIBCXX_USE_CXX11_ABI=1)
 INCLUDE_DIRECTORIES(
   ${CMAKE_CURRENT_BINARY_DIR}/include
   ${CMAKE_SOURCE_DIR}/extra/rapidjson/include

diff --git a/Docs/02-getting-started/quick-deployment.md b/Docs/02-getting-started/quick-deployment.md
@@ -5,40 +5,43 @@ sidebar_position: 3.1
 
 # Quick Deployment
 
-## 1. Download the latest package
+## **1. Download the latest package**
 Click [here](https://static.stoneatom.com/stonedb-ce-5.6-v1.0.0.el7.x86_64.tar.gz) to download the latest installation package of StoneDB.
-## 2. Upload and decompress the TAR package
+## **2. Upload and decompress the TAR package**
 ```shell
 tar -zxvf stonedb-ce-5.6-v1.0.0.el7.x86_64.tar.gz
 ```
 Upload the installation package to the directory. The name of the folder extracted from the package is **stonedb56**.
-## 3. Check dependencies
+## **3. Check dependencies**
 ```bash
 cd /stonedb56/install/bin
 ldd mysqld
 ldd mysql
 ```
-If the command output contains keywords **not found**, some dependencies are missing and must be installed.
-## 4. Modify the configuration file
+If the command output contains keywords **not found**, some dependencies are missing and must be installed. <br />For example, `libsnappy.so.1 => not found` is returned:
+
+- If your OS is Ubuntu, run the `sudo apt search libsnappy` command. The command output will inform you to install `libsnappy-dev`. 
+- If your OS is RHEL or CentOS, run the `yum search all snappy` command. The command output will inform you to install **snappy-devel** and **snappy**.
+## **4. Modify the configuration file**
 ```bash
 cd /stonedb56/install/
 cp stonedb.cnf stonedb.cnf.bak
 vi stonedb.cnf
 ```
 Modify the path and parameters. If the installation folder is **stonedb**, you only need to modify the parameters.
-## 5. Create an account
+## **5. Create an account**
 ```bash
 groupadd mysql
 useradd -g mysql mysql
 passwd mysql
 ```
-## 6. Execute reinstall.sh
+## **6. Execute reinstall.sh**
 ```bash
 cd /stonedb56/install
 ./reinstall.sh
 ```
 The process of executing the script is to initialize and start the StoneDB.
-## 7. Log in to StoneDB
+## **7. Log in to StoneDB**
 ```shell
 /stonedb56/install/bin/mysql -uroot -p -S /stonedb56/install/tmp/mysql.sock 
 Enter password: 
@@ -65,7 +68,7 @@ mysql> show databases;
 +--------------------+
 7 rows in set (0.00 sec)
 ```
-## 8. Stop StoneDB
+## **8. Stop StoneDB**
 ```shell
 /stonedb56/install/bin/mysqladmin -uroot -p -S /stonedb56/install/tmp/mysql.sock shutdown
-```
+```
diff --git a/Docs/03-O&M-Guide/01-backup-and-recovery/use-mydumper-full-backup.md b/Docs/03-O&M-Guide/01-backup-and-recovery/use-mydumper-full-backup.md
@@ -99,7 +99,7 @@ Application Options:
 --load-data
 --fields-terminated-by
 --fields-enclosed-by
---fields-escaped-by             Single character that is going to be used to escape characters in theLOAD DATA stament, default: '\'
+--fields-escaped-by             Single character that is going to be used to escape characters in the LOAD DATA stament, default: '\'
 --lines-starting-by             Adds the string at the begining of each row. When --load-data is usedit is added to the LOAD DATA statement. Its affects INSERT INTO statementsalso when it is used.
 --lines-terminated-by           Adds the string at the end of each row. When --load-data is used it isadded to the LOAD DATA statement. Its affects INSERT INTO statementsalso when it is used.
 --statement-terminated-by       This might never be used, unless you know what are you doing
@@ -249,6 +249,24 @@ Each table has two backup files:
 - **database.table.00000.sql**: records table data.
 - **database.table-metadata**: records table metadata.
 
+***Extensions***
+
+If you want to import data to StoneDB, you must replace **engine=innodb** with **engine=stonedb** in table schema file **database.table-schema.sql** and check whether the syntax of the table schema is compatible with StoneDB. For example, if the syntax contain keyword **unsigned**, it is incompatible. Following is a schema example after modification:
+```
+[root@dev-myos dumper]# cat zz.t_user-schema.sql
+/*!40101 SET NAMES binary*/;
+/*!40014 SET FOREIGN_KEY_CHECKS=0*/;
+/*!40103 SET TIME_ZONE='+00:00' */;
+CREATE TABLE `t_user` (
+  `id` int(11) NOT NULL AUTO_INCREMENT,
+  `c_user_id` varchar(36) NOT NULL DEFAULT '',
+  `c_name` varchar(22) NOT NULL DEFAULT '',
+  `c_province_id` int(11) NOT NULL,
+  `c_city_id` int(11) NOT NULL,
+  `create_time` datetime NOT NULL,
+  PRIMARY KEY (`id`)
+) ENGINE=STONEDB AUTO_INCREMENT=10001 DEFAULT CHARSET=utf8;
+```
 
 ### Backup principles
 

diff --git a/Docs/03-O&M-Guide/01-backup-and-recovery/use-mysqldump-backup-and-restore.md b/Docs/03-O&M-Guide/01-backup-and-recovery/use-mysqldump-backup-and-restore.md
@@ -414,11 +414,30 @@ mysql> select * from dumpdb.dumptb;
 /stonedb56/install/bin/mysqldump  -uroot -p***** -P3306 --skip-opt --master-data=2 --single-transaction --set-gtid-purged=off  --databases dumpdb > /tmp/dumpdb.sql
 ```
 
+#### Back up the table schema of a specific database
+```
+/stonedb56/install/bin/mysqldump  -uroot -p***** -P3306   -d --databases dumpdb > /tmp/dumpdb_table.sql
+```
+
+#### Back up data in a table of a specific database, excluding the schema
+```
+/stonedb56/install/bin/mysqldump  -uroot -p***** -P3306 --skip-opt --master-data=2 --single-transaction --set-gtid-purged=off  -t dumpdb > /tmp/dumpdb_table.sql
+```
+
 #### Use mysqldump to back up all databases, except system databases mysql, performation_schema, and information_schema
 ```bash
 /stonedb56/install/bin/mysql  -uroot -p****** -P3306 -e "show databases;" | grep -Ev "sys|performance_schema|information_schema|Database|test" | xargs /stonedb56/install/bin/mysqldump  -uroot -p****** -P3306 --master-data=1 --skip-opt --databases > /tmp/ig_sysdb.sql
 ```
 
+***Extensions***
+
+The previous method is suitable for backing up small tables from InnoDB. If you want to back up a large table, we recommend that you back up the schema and the data file separately, and then run the `sed -i 's/<Original character string>/<New character string>/g' <File name>` to change the engine setting in the backup file. For example:
+```
+sed -i 's/ENGINE=InnoDB/ENGINE=STONEDB/g' <File name>
+```
+Then, perform recovery to import the table to StoneDB.
+
+
 ### Recovery
 
 #### Import data to StoneDB
-Original file line number
+Diff line change
@@ Expand Up / @@ -113,3 +113,4 @@ CMakeFiles @@
     CMakeFiles/*
     CTestTestfile.cmake
     COPYING
+    .vs/