mysql: figure out why MySqlInitialLoad feature-benchmark scenario spuriously fails

[rjobanp]

What version of Materialize are you using?

v0.88.0

What is the issue?

https://buildkite.com/materialize/release-qualification/builds/428#018daf74-d96d-4583-92c0-04c7db281b1f

==> Running scenario MySqlInitialLoad, scale = 7.0, N = 10000000
--
  | 2024-02-16 01:34:09 UTC | Running the shared() section for scenario MySqlInitialLoad with v0.88.0 ...
  | 2024-02-16 01:34:09 UTC |  
  | 2024-02-16 01:34:09 UTC | $ mysql-connect name=mysql url=mysql://root@mysql password=${arg.mysql-root-password}
  | 2024-02-16 01:34:09 UTC |  
  | 2024-02-16 01:34:09 UTC | $ mysql-execute name=mysql
  | 2024-02-16 01:34:09 UTC | DROP DATABASE IF EXISTS public;
  | 2024-02-16 01:34:09 UTC | CREATE DATABASE public;
  | 2024-02-16 01:34:09 UTC | USE public;
  | 2024-02-16 01:34:09 UTC |  
  | 2024-02-16 01:34:09 UTC | SET @i:=0;
  | 2024-02-16 01:34:09 UTC | CREATE TABLE pk_table (pk BIGINT PRIMARY KEY, f2 BIGINT);
  | 2024-02-16 01:34:09 UTC | INSERT INTO pk_table SELECT @i:=@i+1, @i*@i FROM mysql.time_zone t1, mysql.time_zone t2 LIMIT 10000000;
  | 2024-02-16 01:34:09 UTC |  
  | 2024-02-16 01:34:22 UTC | shared() done
  | 2024-02-16 01:34:22 UTC | Running the before() section for scenario MySqlInitialLoad with v0.88.0 ...
  | 2024-02-16 01:34:22 UTC |  
  | 2024-02-16 01:34:22 UTC | > DROP SOURCE IF EXISTS mz_source_mysqlcdc;
  | 2024-02-16 01:34:22 UTC | > DROP CLUSTER IF EXISTS source_cluster CASCADE
  | 2024-02-16 01:34:22 UTC |  
  | 2024-02-16 01:34:22 UTC | Running the benchmark for scenario MySqlInitialLoad with v0.88.0 ...
  | 2024-02-16 01:34:22 UTC |  
  | 2024-02-16 01:34:22 UTC | > CREATE SECRET IF NOT EXISTS mysqlpass AS '${arg.mysql-root-password}'
  | 2024-02-16 01:34:22 UTC | > CREATE CONNECTION IF NOT EXISTS mysql_conn TO MYSQL (
  | 2024-02-16 01:34:22 UTC | HOST mysql,
  | 2024-02-16 01:34:22 UTC | USER root,
  | 2024-02-16 01:34:22 UTC | PASSWORD SECRET mysqlpass
  | 2024-02-16 01:34:22 UTC | )
  | 2024-02-16 01:34:22 UTC |  
  | 2024-02-16 01:34:22 UTC | > CREATE CLUSTER source_cluster SIZE '4', REPLICATION FACTOR 1;
  | 2024-02-16 01:34:22 UTC |  
  | 2024-02-16 01:34:22 UTC | > CREATE SOURCE mz_source_mysqlcdc
  | 2024-02-16 01:34:22 UTC | IN CLUSTER source_cluster
  | 2024-02-16 01:34:22 UTC | FROM MYSQL CONNECTION mysql_conn
  | 2024-02-16 01:34:22 UTC | FOR TABLES (public.pk_table)
  | 2024-02-16 01:34:22 UTC | /* A */
  | 2024-02-16 01:34:22 UTC |  
  | 2024-02-16 01:34:22 UTC | > SELECT count(*) FROM pk_table
  | 2024-02-16 01:34:22 UTC | /* B */
  | 2024-02-16 01:34:22 UTC | 10000000
  | 2024-02-16 01:34:22 UTC |  
  | 2024-02-16 02:04:23 UTC | 17:1: error: non-matching rows: expected:
  | 2024-02-16 02:04:23 UTC | [["10000000"]]
  | 2024-02-16 02:04:23 UTC | got:
  | 2024-02-16 02:04:23 UTC | [["3200521"]]
  | 2024-02-16 02:04:23 UTC | Poor diff:
  | 2024-02-16 02:04:23 UTC | - 10000000
  | 2024-02-16 02:04:23 UTC | + 3200521
  | 2024-02-16 02:04:23 UTC |  
  | 2024-02-16 02:04:23 UTC | \|
  | 2024-02-16 02:04:23 UTC | 2 \| > CREATE SECRET IF N ... [rest of line truncated for security]
  | 2024-02-16 02:04:23 UTC | 6 \|     PASSWORD SECRET  ... [rest of line truncated for security]
  | 2024-02-16 02:04:23 UTC | 16 \|
  | 2024-02-16 02:04:23 UTC | 17 \| > SELECT count(*) FROM pk_table
  | 2024-02-16 02:04:23 UTC | \| ^
  | 2024-02-16 02:04:23 UTC | !!! Error Report

This feature-benchmark scenario was failing due to the release qualification build for 0.88.0.

This PR #25309 disabled the scenario until we figure out what went wrong.

[rjobanp]

Closing until we see this re-occur, now that it is reenabled

[def-]

Occurred again: #25684 (comment)

[def-]

@rjobanp This is still happening commonly: https://buildkite.com/materialize/release-qualification/builds/466#018eae09-91c5-4fe3-88eb-49a07d5cd777 I will try to disable MySqlInitialLoad in the feature-benchmark for release qualification temporarily. But I'm concerned that having more data causes wrong results.