Make mysqldump work with gitbase #361

ajnavarro · 2018-07-02T09:26:51Z

Right now if you try to do a mysqldump, you will have the next error:

mysqldump --all-databases --port=3306 --host=localhost --protocol=tcp --user=root

mysqldump: Couldn't execute '/*!40100 SET @@SQL_MODE='' */': unknown error: syntax error at position 30 (1105)

The text was updated successfully, but these errors were encountered:

erizocosmico · 2018-07-02T15:12:49Z

We need to support the following things:

Ignore comment lines, for example, /*!40101 SET @OLD_CHARACTER_SET_CLIENT=@@CHARACTER_SET_CLIENT */;
Support SHOW VARIABLES (vitess sqlparser tells you it's a SHOW VARIABLES, but it does not give you anything else, so it would have to be a handcrafted parser).
SET statements with @@SESSION and @@GLOBAL, which ideally should be able to access global and session scoped configurations. We do not have such things right now. We could just ignore them, though.
SELECT @@GLOBAL..., as the previous point, we need some way of accessing @@GLOBAL and @@SESSION as tables. These are a bit more complicated to ignore, though, as just returning nothing in those case does not work.
Probably more, that's how far I got just ignoring stuff.

I remember the discussion around supporting MySQL Workbench and we ended up closing it as it was not prioritary. Should we keep working on this, then, given it's the same constraints we had in that issue?

smola · 2018-07-02T15:36:32Z

@erizocosmico gitbase interoperability is a mid-priority (P1) objective for Q3, but the tools that should be initially included in this interoperability are still not defined.

gitbase interoperability with 3rd party tools

erizocosmico · 2018-07-03T13:15:03Z

Supported so far:

Remove comments.
Empty queries do not crash (see the /* SOME QUERY INSIDE COMMENT*/).
SHOW VARIABLES, which returns a dummy result.
SET, which returns an dummy result.
@@GLOBAL.FOO and @@SESSION.FOO always return nil and can be resolved.
SHOW DATABASES.
LOCK, which returns a dummy result.
USE, which returns a dummy result

However, these are way harder to "hack" (right now I just have a special case for them to get past them, but that should not go into the code):

SELECT LOGFILE_GROUP_NAME, FILE_NAME, TOTAL_EXTENTS, INITIAL_SIZE, ENGINE, EXTRA FROM INFORMATION_SCHEMA.FILES WHERE FILE_TYPE = 'UNDO LOG' AND FILE_NAME IS NOT NULL AND LOGFILE_GROUP_NAME IS NOT NULL GROUP BY LOGFILE_GROUP_NAME, FILE_NAME, ENGINE, TOTAL_EXTENTS, INITIAL_SIZE ORDER BY LOGFILE_GROUP_NAME```

SELECT DISTINCT TABLESPACE_NAME, FILE_NAME, LOGFILE_GROUP_NAME, EXTENT_SIZE, INITIAL_SIZE, ENGINE FROM INFORMATION_SCHEMA.FILES WHERE FILE_TYPE = 'DATAFILE' ORDER BY TABLESPACE_NAME, LOGFILE_GROUP_NAME

These are all the queries being run:

/*!40100 SET @@SQL_MODE='' */
/*!40103 SET TIME_ZONE='+00:00' */
SHOW VARIABLES LIKE 'gtid\_mode'
SELECT LOGFILE_GROUP_NAME, FILE_NAME, TOTAL_EXTENTS, INITIAL_SIZE, ENGINE, EXTRA FROM INFORMATION_SCHEMA.FILES WHERE FILE_TYPE = 'UNDO LOG' AND FILE_NAME IS NOT NULL AND LOGFILE_GROUP_NAME IS NOT NULL GROUP BY LOGFILE_GROUP_NAME, FILE_NAME, ENGINE, TOTAL_EXTENTS, INITIAL_SIZE ORDER BY LOGFILE_GROUP_NAME
SELECT DISTINCT TABLESPACE_NAME, FILE_NAME, LOGFILE_GROUP_NAME, EXTENT_SIZE, INITIAL_SIZE, ENGINE FROM INFORMATION_SCHEMA.FILES WHERE FILE_TYPE = 'DATAFILE' ORDER BY TABLESPACE_NAME, LOGFILE_GROUP_NAME
SHOW DATABASES
SHOW VARIABLES LIKE 'ndbinfo\_version'
SHOW CREATE DATABASE IF NOT EXISTS ``
show tables
LOCK TABLES `blobs` READ /*!32311 LOCAL */,`commit_blobs` READ /*!32311 LOCAL */,`commit_files` READ /*!32311 LOCAL */,`commit_trees` READ /*!32311 LOCAL */,`commits` READ /*!32311 LOCAL */,`files` READ /*!32311 LOCAL */,`ref_commits` READ /*!32311 LOCAL */,`refs` READ /*!32311 LOCAL */,`remotes` READ /*!32311 LOCAL */,`repositories` READ /*!32311 LOCAL */,`tree_entries` READ /*!32311 LOCAL */

And then mysqldump exits with

mysqldump: Couldn't execute 'show table status like 'blobs'': Commands out of sync; you can't run this command now (2014)

Although that query has never been executed in the server (at least it never got to the engine).

According to the manual, this is the reason this error could happen, which is very very helpful https://dev.mysql.com/doc/refman/8.0/en/commands-out-of-sync.html

So, I'm kind of in a dead end here.

Any thoughts? @smola @ajnavarro

smola · 2018-07-03T14:29:09Z

I suggest using a more dumb thing such as mysqldump --skip-opt --no-create-db --force

erizocosmico · 2018-07-03T14:43:25Z

I will try with that. I tried with --ignore-errors=2014 with no luck so far

erizocosmico · 2018-07-03T14:45:57Z

UPDATE: same errors. Dump is completed, but the result is riddled with stuff like this:

mysqldump: Couldn't execute 'show table status like 'tree\_entries'': Commands out of sync; you can't run this command now (2014)
mysqldump: Couldn't execute 'SET SQL_QUOTE_SHOW_CREATE=1/*!40102 ,SQL_MODE=concat(@@sql_mode, _utf8 ',NO_KEY_OPTIONS,NO_TABLE_OPTIONS,NO_FIELD_OPTIONS') */': Commands out of sync; you can't run this command now (2014)
mysqldump: Couldn't execute 'SELECT `COLUMN_NAME` AS `Field`, `COLUMN_TYPE` AS `Type`, `IS_NULLABLE` AS `Null`, `COLUMN_KEY` AS `Key`, `COLUMN_DEFAULT` AS `Default`, `EXTRA` AS `Extra`, `COLUMN_COMMENT` AS `Comment` FROM `INFORMATION_SCHEMA`.`COLUMNS` WHERE TABLE_SCHEMA = 'foo' AND TABLE_NAME = 'tree_entries'': Commands out of sync; you can't run this command now (2014)
/*!40103 SET TIME_ZONE=@OLD_TIME_ZONE */;

mcuadros · 2018-09-20T09:46:28Z

@ajnavarro just a friendly ping that this is an OKR for this Q.

ajnavarro · 2018-09-20T10:03:09Z

Actually, the description is really broad:

Ensure gitbase interoperability with 3rd party tools

We discovered that each tool is doing totally different queries, We focused to be compatible with MariaDB JDBC driver, to make it compatible with JVM applications like Spark.

Anyways, we'll have a look to see if we can make it work without implementing a lot of new statements.

erizocosmico · 2018-10-04T14:24:34Z

Ran a mysql server with the log on just to get all the queries we will need to support for mysqldump to work. This is the full list of queries executed by mysqldump:

/*!40100 SET @@SQL_MODE='' */
/*!40103 SET TIME_ZONE='+00:00' */
/*!80000 SET SESSION information_schema_stats_expiry=0 */
SET SESSION NET_READ_TIMEOUT= 700, SESSION NET_WRITE_TIMEOUT= 700
SHOW VARIABLES LIKE 'gtid\_mode'
SELECT LOGFILE_GROUP_NAME, FILE_NAME, TOTAL_EXTENTS, INITIAL_SIZE, ENGINE, EXTRA FROM INFORMATION_SCHEMA.FILES WHERE FILE_TYPE = 'UNDO LOG' AND FILE_NAME IS NOT NULL AND LOGFILE_GROUP_NAME IS NOT NULL GROUP BY LOGFILE_GROUP_NAME, FILE_NAME, ENGINE, TOTAL_EXTENTS, INITIAL_SIZE ORDER BY LOGFILE_GROUP_NAME
SELECT DISTINCT TABLESPACE_NAME, FILE_NAME, LOGFILE_GROUP_NAME, EXTENT_SIZE, INITIAL_SIZE, ENGINE FROM INFORMATION_SCHEMA.FILES WHERE FILE_TYPE = 'DATAFILE' ORDER BY TABLESPACE_NAME, LOGFILE_GROUP_NAME
SHOW DATABASES
SHOW VARIABLES LIKE 'ndbinfo\_version'
SHOW CREATE DATABASE IF NOT EXISTS `foo`
show tables
LOCK TABLES `bar` READ /*!32311 LOCAL */
show table status like 'bar'
SET SQL_QUOTE_SHOW_CREATE=1
SET SESSION character_set_results = 'binary'
show create table `bar`
SET SESSION character_set_results = 'utf8mb4'
show fields from `bar`
show fields from `bar`
SELECT /*!40001 SQL_NO_CACHE */ * FROM `bar`
SET SESSION character_set_results = 'binary'
use `foo`
select @@collation_database
SHOW TRIGGERS LIKE 'bar'
SET SESSION character_set_results = 'utf8mb4'
SET SESSION character_set_results = 'binary'
SELECT COLUMN_NAME,                       JSON_EXTRACT(HISTOGRAM, '$."number-of-buckets-specified"')                FROM information_schema.COLUMN_STATISTICS                WHERE SCHEMA_NAME = 'foo' AND TABLE_NAME = 'bar'
SET SESSION character_set_results = 'utf8mb4'
UNLOCK TABLES

Maybe we can reduce those queries with some flags.

ajnavarro · 2018-10-04T14:29:16Z

would be great to find some flags that reduce the number of executed queries.

erizocosmico · 2018-10-04T15:25:45Z

The most I've been able to reduce it is by using --skip-triggers to skip getting triggers. What the other flags reduce is the output file.

erizocosmico · 2018-10-05T08:23:30Z

Queries performed by mysqldump with outputs

These are the queries mysqldump performs and the outputs a real mysql server would output.

/*!40100 SET @@SQL_MODE='' */

Output: no rows

/*!40103 SET TIME_ZONE='+00:00' */

Output: no rows

/*!80000 SET SESSION information_schema_stats_expiry=0 */

Output: no rows

SET SESSION NET_READ_TIMEOUT= 700, SESSION NET_WRITE_TIMEOUT= 700

Output: no rows

SHOW VARIABLES LIKE 'gtid\_mode'

Output:

+---------------+-------+
| Variable_name | Value |
+---------------+-------+
| gtid_mode     | OFF   |
+---------------+-------+

SELECT LOGFILE_GROUP_NAME, FILE_NAME, TOTAL_EXTENTS, INITIAL_SIZE, ENGINE, EXTRA FROM INFORMATION_SCHEMA.FILES WHERE FILE_TYPE = 'UNDO LOG' AND FILE_NAME IS NOT NULL AND LOGFILE_GROUP_NAME IS NOT NULL GROUP BY LOGFILE_GROUP_NAME, FILE_NAME, ENGINE, TOTAL_EXTENTS, INITIAL_SIZE ORDER BY LOGFILE_GROUP_NAME

Output: no rows

SELECT DISTINCT TABLESPACE_NAME, FILE_NAME, LOGFILE_GROUP_NAME, EXTENT_SIZE, INITIAL_SIZE, ENGINE FROM INFORMATION_SCHEMA.FILES WHERE FILE_TYPE = 'DATAFILE' ORDER BY TABLESPACE_NAME, LOGFILE_GROUP_NAME

Output: no rows

SHOW DATABASES

Output:

+--------------------+
| Database           |
+--------------------+
| foo                |
| information_schema |
| mysql              |
| performance_schema |
| sys                |
+--------------------+

SHOW VARIABLES LIKE 'ndbinfo\_version'

Output: no rows

SHOW CREATE DATABASE IF NOT EXISTS `foo`

Output:

+----------+---------------------------------------------------------------------------------------------------------------------+
| Database | Create Database                                                                                                     |
+----------+---------------------------------------------------------------------------------------------------------------------+
| foo      | CREATE DATABASE /*!32312 IF NOT EXISTS*/ `foo` /*!40100 DEFAULT CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci */ |
+----------+---------------------------------------------------------------------------------------------------------------------+

show tables

Output: tables

LOCK TABLES `bar` READ /*!32311 LOCAL */

Output: 0 rows

show table status like 'bar'

Output:

+------+--------+---------+------------+------+----------------+-------------+-----------------+--------------+-----------+----------------+---------------------+---------------------+------------+--------------------+----------+----------------+---------+
| Name | Engine | Version | Row_format | Rows | Avg_row_length | Data_length | Max_data_length | Index_length | Data_free | Auto_increment | Create_time         | Update_time         | Check_time | Collation          | Checksum | Create_options | Comment |
+------+--------+---------+------------+------+----------------+-------------+-----------------+--------------+-----------+----------------+---------------------+---------------------+------------+--------------------+----------+----------------+---------+
| bar  | InnoDB |      10 | Dynamic    |    3 |           5461 |       16384 |               0 |            0 |         0 |              4 | 2018-10-05 08:05:02 | 2018-10-05 08:05:20 | NULL       | utf8mb4_0900_ai_ci |     NULL |                |         |
+------+--------+---------+------------+------+----------------+-------------+-----------------+--------------+-----------+----------------+---------------------+---------------------+------------+--------------------+----------+----------------+---------+

SET SQL_QUOTE_SHOW_CREATE=1

Output: no rows

SET SESSION character_set_results = 'binary'

Output: no rows

show create table `bar`

Output:

+-------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Table | Create Table                                                                                                                                                                      |
+-------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| bar   | CREATE TABLE `bar` (
  `id` int(11) NOT NULL AUTO_INCREMENT,
  `a` text,
  PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=4 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci |
+-------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

SET SESSION character_set_results = 'utf8mb4'

Output: no rows

show fields from `bar`

Output:

+-------+---------+------+-----+---------+----------------+
| Field | Type    | Null | Key | Default | Extra          |
+-------+---------+------+-----+---------+----------------+
| id    | int(11) | NO   | PRI | NULL    | auto_increment |
| a     | text    | YES  |     | NULL    |                |
+-------+---------+------+-----+---------+----------------+

SELECT /*!40001 SQL_NO_CACHE */ * FROM `bar`

Output: everything in the table

SET SESSION character_set_results = 'binary'

Output: 0 rows

use `foo`

Output: 0 rows

select @@collation_database

Output:

+----------------------+
| @@collation_database |
+----------------------+
| utf8mb4_0900_ai_ci   |
+----------------------+

SET SESSION character_set_results = 'utf8mb4'

Output: no rows

SET SESSION character_set_results = 'binary'

Output: no rows

SELECT COLUMN_NAME,                       JSON_EXTRACT(HISTOGRAM, '$."number-of-buckets-specified"')                FROM information_schema.COLUMN_STATISTICS                WHERE SCHEMA_NAME = 'foo' AND TABLE_NAME = 'bar'

Output: no rows

SET SESSION character_set_results = 'utf8mb4'

Output: no rows

UNLOCK TABLES

Output: no rows

Things that need to be implemented

@ajnavarro this is the full list of queries with their outputs and from them all the things we would need to implement for this to, in theory, be able to work correctly. Should we move forward with this, then?

ajnavarro · 2018-10-05T08:39:56Z

@erizocosmico totally. Could you open several issues to be able to parallelize work? (some of that issues can be marked as hacktoberfest too)

erizocosmico · 2018-10-05T08:41:08Z

Sure

erizocosmico · 2018-11-02T15:25:02Z

Closing, this was already merged

ajnavarro added the feature label Jul 2, 2018

erizocosmico self-assigned this Jul 2, 2018

erizocosmico mentioned this issue Jul 3, 2018

sql/*: add some more compatibility with mysqldump src-d/go-mysql-server#258

Closed

erizocosmico mentioned this issue Aug 16, 2018

SQL global & session variables #425

Closed

erizocosmico added the EPIC label Oct 5, 2018

erizocosmico closed this as completed Nov 2, 2018

dpordomingo mentioned this issue Mar 6, 2019

Compatibility with MySQL Workbench #722

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Make mysqldump work with gitbase #361

Make mysqldump work with gitbase #361

ajnavarro commented Jul 2, 2018

erizocosmico commented Jul 2, 2018 •

edited

Loading

smola commented Jul 2, 2018

erizocosmico commented Jul 3, 2018 •

edited

Loading

smola commented Jul 3, 2018

erizocosmico commented Jul 3, 2018

erizocosmico commented Jul 3, 2018 •

edited

Loading

mcuadros commented Sep 20, 2018

ajnavarro commented Sep 20, 2018

erizocosmico commented Oct 4, 2018

ajnavarro commented Oct 4, 2018

erizocosmico commented Oct 4, 2018

erizocosmico commented Oct 5, 2018 •

edited

Loading

ajnavarro commented Oct 5, 2018

erizocosmico commented Oct 5, 2018

erizocosmico commented Nov 2, 2018

Make mysqldump work with gitbase #361

Make mysqldump work with gitbase #361

Comments

ajnavarro commented Jul 2, 2018

erizocosmico commented Jul 2, 2018 • edited Loading

smola commented Jul 2, 2018

erizocosmico commented Jul 3, 2018 • edited Loading

smola commented Jul 3, 2018

erizocosmico commented Jul 3, 2018

erizocosmico commented Jul 3, 2018 • edited Loading

mcuadros commented Sep 20, 2018

ajnavarro commented Sep 20, 2018

erizocosmico commented Oct 4, 2018

ajnavarro commented Oct 4, 2018

erizocosmico commented Oct 4, 2018

erizocosmico commented Oct 5, 2018 • edited Loading

Queries performed by mysqldump with outputs

Things that need to be implemented

ajnavarro commented Oct 5, 2018

erizocosmico commented Oct 5, 2018

erizocosmico commented Nov 2, 2018

erizocosmico commented Jul 2, 2018 •

edited

Loading

erizocosmico commented Jul 3, 2018 •

edited

Loading

erizocosmico commented Jul 3, 2018 •

edited

Loading

erizocosmico commented Oct 5, 2018 •

edited

Loading