Prepared statement - Frequent crash on ProxySQL

[s4sukan]

Hi @renecannao ,

ProxySQL is crashing every now and then, I've processed coredumps and each core dump is pointing to below line in MySQL_Session.cpp.

I've tested with 1.4.16,2.0.14 and even 2.1.1. All versions having the same crash,

Errorlog:

proxysql: MySQL_Session.cpp:3297: int MySQL_Session::handler(): Assertion `0' failed.
Error: signal 6:
/usr/bin/proxysql(_Z13crash_handleri+0x1a)[0x4d5fda]
/lib64/libc.so.6(+0x35270)[0x7f9262f09270]
/lib64/libc.so.6(gsignal+0x37)[0x7f9262f091f7]
/lib64/libc.so.6(abort+0x148)[0x7f9262f0a8e8]
/lib64/libc.so.6(+0x2e266)[0x7f9262f02266]
/lib64/libc.so.6(+0x2e312)[0x7f9262f02312]
/usr/bin/proxysql(_ZN13MySQL_Session7handlerEv+0x395f)[0x5386cf]
/usr/bin/proxysql(_ZN12MySQL_Thread20process_all_sessionsEv+0x3cc)[0x51695c]
/usr/bin/proxysql(_ZN12MySQL_Thread3runEv+0xc4c)[0x51d68c]
/usr/bin/proxysql(_Z24mysql_worker_thread_funcPv+0x6c)[0x4ceeec]
/lib64/libpthread.so.0(+0x7e25)[0x7f92640e0e25]
/lib64/libc.so.6(clone+0x6d)[0x7f9262fcc34d]
2021-03-25 09:22:41 main.cpp:1573:ProxySQL_daemonize_phase3(): [ERROR] ProxySQL crashed. Restarting!
2021-03-25 09:22:41 [INFO] ProxySQL version 2.0.14-70-g7b02b19

#0  0x00007f9262f091f7 in raise () from /lib64/libc.so.6
No symbol table info available.
#1  0x00007f9262f0a8e8 in abort () from /lib64/libc.so.6
No symbol table info available.
#2  0x00007f9262f02266 in __assert_fail_base () from /lib64/libc.so.6
No symbol table info available.
#3  0x00007f9262f02312 in __assert_fail () from /lib64/libc.so.6
No symbol table info available.
#4  0x00000000005386cf in MySQL_Session::handler (this=this@entry=0x7f925192a300) at MySQL_Session.cpp:3297
        client_stmt_id = <optimized out>
        stmt_global_id = <optimized out>
        stmt_info = <optimized out>
        stmt_meta_found = <optimized out>
        stmt_meta = 0x7f9200000000
        prepared_stmt_with_no_params = false
        pkt = {ptr = 0x7f925e652940, size = 15}
        c = <optimized out>
        __PRETTY_FUNCTION__ = "int MySQL_Session::handler()"
        __func__ = "handler"
        wrong_pass = false
#5  0x000000000051695c in MySQL_Thread::process_all_sessions (this=this@entry=0x7f925e5fe000) at MySQL_Thread.cpp:4396
        sess = 0x7f925192a300
        total_active_transactions_ = 0
        idle_maintenance_thread = false
        sess_sort = <optimized out>
        __func__ = "process_all_sessions"
        n = 0
        rc = <optimized out>
#6  0x000000000051d68c in MySQL_Thread::run (this=this@entry=0x7f925e5fe000) at MySQL_Thread.cpp:4118
        num_idles = <optimized out>
        ttw = <optimized out>
        maintenance_interval = <optimized out>
        idle_maintenance_thread = <optimized out>
        __PRETTY_FUNCTION__ = "void MySQL_Thread::run()"
        __func__ = "run"
        n = <optimized out>
        rc = <optimized out>
#7  0x00000000004ceeec in mysql_worker_thread_func (arg=0x7f9261c8a440) at main.cpp:760
        thread_attr = {__size = '\000' <repeats 17 times>, "\020", '\000' <repeats 37 times>, __align = 0}
        tmp_stack_size = 8388608
---Type <return> to continue, or q <return> to quit---
        mysql_thread = 0x7f9261c8a440
        worker = 0x7f925e5fe000
#8  0x00007f92640e0e25 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#9  0x00007f9262fcc34d in clone () from /lib64/libc.so.6

If needed, I can send the full core dumps. Any help is appreciated as we are facing severe stability issues.

[renecannao]

This is probably caused by an application trying to execute a prepared statement that doesn't exist.
A core dump, together with details on which OS and which proxysql binary are being used, can be useful to confirm this hypothesis.

[s4sukan]

Version : 2.0.14-70-g7b02b19
Package : https://github.com/sysown/proxysql/releases/download/v2.0.14/proxysql-2.0.14-1-centos7.x86_64.rpm
OS : CentOS Linux release 7.4.1708 (Core)

Coredump shared separately

[s4sukan]

@renecannao Did you have any inputs here? Is there any plan to add a exception instead of triggering assertion failure?

[bskllzh]

@s4sukan , please provide the coredump and proxysql file, and use to analyze this problem.

[inavy]

I encounter the same problem...

proxysql: MySQL_Session.cpp:3690: int MySQL_Session::handler(): Assertion `0' failed.
Error: signal 6:
/usr/bin/proxysql(_Z13crash_handleri+0x1a)[0x4ae55a]
/lib64/libc.so.6(+0x363b0)[0x7fbd94f603b0]
/lib64/libc.so.6(gsignal+0x37)[0x7fbd94f60337]
/lib64/libc.so.6(abort+0x148)[0x7fbd94f61a28]
/lib64/libc.so.6(+0x2f156)[0x7fbd94f59156]
/lib64/libc.so.6(+0x2f202)[0x7fbd94f59202]
/usr/bin/proxysql(_ZN13MySQL_Session7handlerEv+0x3957)[0x512bf7]
/usr/bin/proxysql(_ZN12MySQL_Thread20process_all_sessionsEv+0x2fc)[0x4f009c]
/usr/bin/proxysql(_ZN12MySQL_Thread3runEv+0xbb4)[0x4f72c4]
/usr/bin/proxysql(_Z24mysql_worker_thread_funcPv+0x6c)[0x4a791c]
/lib64/libpthread.so.0(+0x7e65)[0x7fbd96140e65]
/lib64/libc.so.6(clone+0x6d)[0x7fbd9502888d]
2022-02-28 10:23:50 main.cpp:1538:ProxySQL_daemonize_phase3(): [ERROR] ProxySQL crashed. Restarting!
2022-02-28 10:23:50 [INFO] ProxySQL version 2.0.10-27-g5b31997
2022-02-28 10:23:50 [INFO] ProxySQL SHA1 checksum: e1982906ea874ad66ba381b267abd085720dd934
2022-02-28 10:23:50 [INFO] Angel process started ProxySQL process 3580

[jaredev]

Hello, @inavy and @s4sukan, I'm working on a PR to resolve this. If you'd like to try it, branch v2.x-3371 removes the assert and instead returns an error and so won't result in a crash.