Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MyRocks crashes when using tmpfs for data directory & --rocksdb-use-direct-reads=1 #778

Closed
RoelVdP opened this issue Dec 31, 2017 · 9 comments

Comments

@RoelVdP
Copy link

RoelVdP commented Dec 31, 2017

Program terminated with signal 6, Aborted.
#0  0x00007f0db7cbb9b1 in __pthread_kill (threadid=<optimized out>, signo=6)
    at ../nptl/sysdeps/unix/sysv/linux/pthread_kill.c:61
61	  val = INTERNAL_SYSCALL (tgkill, err, 3, THREAD_GETMEM (THREAD_SELF, pid),
(gdb) bt
#0  0x00007f0db7cbb9b1 in __pthread_kill (threadid=<optimized out>, signo=6)
    at ../nptl/sysdeps/unix/sysv/linux/pthread_kill.c:61
#1  0x00000000012473d6 in my_write_core (sig=6) at /git/facebook-mysql-5.6_dbg/mysys/stacktrace.c:424
#2  0x0000000000e4a7e4 in handle_fatal_signal (sig=6) at /git/facebook-mysql-5.6_dbg/sql/signal_handler.cc:239
#3  <signal handler called>
#4  0x00007f0db61e91f7 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#5  0x00007f0db61ea8e8 in __GI_abort () at abort.c:90
#6  0x00007f0db61e2266 in __assert_fail_base (fmt=0x7f0db6334e68 "%s%s%s:%u: %s%sAssertion '%s' failed.\n%n", 
    assertion=assertion@entry=0x1c608a7 "log.getting_synced", 
    file=file@entry=0x1c60050 "/git/facebook-mysql-5.6_dbg/rocksdb/db/db_impl.cc", line=line@entry=710, 
    function=function@entry=0x1c64fc0 <rocksdb::DBImpl::MarkLogsSynced(unsigned long, bool, rocksdb::Status const&)::__PRETTY_FUNCTION__> "void rocksdb::DBImpl::MarkLogsSynced(uint64_t, bool, const rocksdb::Status&)")
    at assert.c:92
#7  0x00007f0db61e2312 in __GI___assert_fail (assertion=0x1c608a7 "log.getting_synced", 
    file=0x1c60050 "/git/facebook-mysql-5.6_dbg/rocksdb/db/db_impl.cc", line=710, 
    function=0x1c64fc0 <rocksdb::DBImpl::MarkLogsSynced(unsigned long, bool, rocksdb::Status const&)::__PRETTY_FUNCTION__> "void rocksdb::DBImpl::MarkLogsSynced(uint64_t, bool, const rocksdb::Status&)") at assert.c:101
#8  0x000000000131a1b2 in rocksdb::DBImpl::MarkLogsSynced (this=0x7f0db407b800, up_to=11, synced_dir=false, 
    status=...) at /git/facebook-mysql-5.6_dbg/rocksdb/db/db_impl.cc:710
#9  0x000000000135ef22 in rocksdb::DBImpl::WriteImpl (this=0x7f0db407b800, write_options=..., my_batch=
    0x7f0d8f03e140, callback=0x0, log_used=0x0, log_ref=0, disable_memtable=false)
    at /git/facebook-mysql-5.6_dbg/rocksdb/db/db_impl_write.cc:297
#10 0x000000000135ddc6 in rocksdb::DBImpl::Write (this=0x7f0db407b800, write_options=..., 
    my_batch=0x7f0d8f03e140) at /git/facebook-mysql-5.6_dbg/rocksdb/db/db_impl_write.cc:51
#11 0x00000000012bd333 in myrocks::Rdb_dict_manager::commit (this=0x26b5840 <myrocks::dict_manager>, 
    batch=0x7f0d8f03e140, sync=@0x7f0db82ad91a: true)
    at /git/facebook-mysql-5.6_dbg/storage/rocksdb/rdb_datadic.cc:4211
#12 0x0000000001272369 in myrocks::ha_rocksdb::delete_table (this=0x7f0d8f147ec0, 
    tablename=0x7f0db82aebf0 "./test/t1") at /git/facebook-mysql-5.6_dbg/storage/rocksdb/ha_rocksdb.cc:9476
#13 0x0000000000d36c4c in handler::ha_delete_table (this=0x7f0d8f147ec0, name=0x7f0db82aebf0 "./test/t1")
    at /git/facebook-mysql-5.6_dbg/sql/handler.cc:4885
#14 0x0000000000d3167d in ha_delete_table (thd=0x7f0d9779f000, table_type=0x7f0db40c1c00, 
    path=0x7f0db82aebf0 "./test/t1", db=0x7f0d8f147928 "test", alias=0x7f0d8f14792d "t1", 
    generate_warning=false) at /git/facebook-mysql-5.6_dbg/sql/handler.cc:2575
#15 0x0000000000f7a747 in mysql_rm_table_no_locks (thd=0x7f0d9779f000, tables=0x7f0d8f1473c0, if_exists=true, 
    drop_temporary=false, drop_view=true, dont_log_query=true)
    at /git/facebook-mysql-5.6_dbg/sql/sql_table.cc:2545
#16 0x0000000000ec5f35 in mysql_rm_db (thd=0x7f0d9779f000, db=0x7f0d8f0310b0 "test", if_exists=false, 
    silent=false) at /git/facebook-mysql-5.6_dbg/sql/sql_db.cc:1062
#17 0x0000000000f08ebe in mysql_execute_command (thd=0x7f0d9779f000, statement_start_time=0x7f0db82b1818, 
    post_parse=0x7f0db82b2960) at /git/facebook-mysql-5.6_dbg/sql/sql_parse.cc:4952
#18 0x0000000000f103c3 in mysql_parse (thd=0x7f0d9779f000, rawbuf=0x7f0d8f031010 "DROP DATABASE test", 
    length=18, parser_state=0x7f0db82b2230, last_timer=0x7f0db82b2960, async_commit=0x7f0db82b295f "")
    at /git/facebook-mysql-5.6_dbg/sql/sql_parse.cc:7770
#19 0x0000000000f00f4f in dispatch_command (command=COM_QUERY, thd=0x7f0d9779f000, 
    packet=0x7f0d977a4001 "DROP DATABASE test", packet_length=18, srv_session=0x0)
    at /git/facebook-mysql-5.6_dbg/sql/sql_parse.cc:1862
#20 0x0000000000eff36c in do_command (thd=0x7f0d9779f000) at /git/facebook-mysql-5.6_dbg/sql/sql_parse.cc:1207
#21 0x0000000000ebee74 in do_handle_one_connection (thd_arg=0x7f0d97690000)
    at /git/facebook-mysql-5.6_dbg/sql/sql_connect.cc:1142
#22 0x0000000000ebe907 in handle_one_connection (arg=0x7f0d97690000)
    at /git/facebook-mysql-5.6_dbg/sql/sql_connect.cc:963
#23 0x00000000016874af in pfs_spawn_thread (arg=0x7f0db40c2680)
    at /git/facebook-mysql-5.6_dbg/storage/perfschema/pfs.cc:1860
#24 0x00007f0db7cb6e25 in start_thread (arg=0x7f0db82b3700) at pthread_create.c:308
#25 0x00007f0db62ac34d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

2018-01-01 09:29:52 16513 [Note] /sda/FB200717-mysql-5.6.35-linux-x86_64-debug/bin/mysqld: ready for connections.
Version: '5.6.35-debug'  socket: '/sda/FB200717-mysql-5.6.35-linux-x86_64-debug/socket.sock'  port: 18240  MySQL Community Server (GPL)
2018-01-01 09:30:04 16513 [ERROR] LibRocksDB:[/git/facebook-mysql-5.6_dbg/rocksdb/db/db_impl_compaction_flush.cc:1257] Waiting after background flush error: IO error: While open a file for random read: ./.rocksdb/000012.sst: Invalid argumentAccumulated background error counts: 1
2018-01-01 09:30:04 16513 [ERROR] RocksDB: BG thread failed to write to RocksDB, Status Code: 5, Status: IO error: While open a file for random read: ./.rocksdb/000012.sst: Invalid argument
mysqld: /git/facebook-mysql-5.6_dbg/rocksdb/db/db_impl.cc:710: void rocksdb::DBImpl::MarkLogsSynced(uint64_t, bool, const rocksdb::Status&): Assertion 'log.getting_synced' failed.
22:30:05 UTC - mysqld got signal 6 ;

1) Make sure to run this with the data directory on /dev/shm (tmpfs) 
2) Testcase:
# mysqld options required for replay:  --no-defaults --default-tmp-storage-engine=MyISAM --rocksdb --skip-innodb --default-storage-engine=RocksDB --rocksdb-use-direct-reads=1
DROP DATABASE test;
CREATE DATABASE test;
USE test;
CREATE TABLE t1(c1 INT) ENGINE=RocksDB;
CREATE TABLE t2(c1 INT) ENGINE=RocksDB;
INSERT INTO t1 VALUES(1);
ALTER TABLE t1 CHANGE c1 c1 FLOAT;
INSERT INTO t2 VALUES(1);

I suspect that the issue is rocksdb-use-direct-reads=1 in combination with /dev/shm? i.e. similar to InnoDB's;

2017-12-31T20:37:30.084316Z 0 [Note] InnoDB: Failed to set O_DIRECT on file /dev/shm/809118/76/data/xb_doublewrite;create: Invalid argument, continuing anyway. O_DIRECT is known to result in 'Invalid argument' on Linux on tmpfs, see MySQL Bug#26662.

Seems like this situation needs to be covered in MyRocks also. This can also be made to crash Percona Server with MyRocks, using the same testcase and these mysqld options;

# mysqld options required for replay:  --plugin-load-add=rocksdb=ha_rocksdb.so --rocksdb-use-direct-reads=1  

Core was generated by '/sda/PS281217-percona-server-5.7.20-18-linux-x86_64-debug/bin/mysqld --no-defau'.
Program terminated with signal 6, Aborted.
#0  0x00007f05f758e9b1 in __pthread_kill (threadid=<optimized out>, signo=6)
    at ../nptl/sysdeps/unix/sysv/linux/pthread_kill.c:61
61	  val = INTERNAL_SYSCALL (tgkill, err, 3, THREAD_GETMEM (THREAD_SELF, pid),
(gdb) bt
#0  0x00007f05f758e9b1 in __pthread_kill (threadid=<optimized out>, signo=6)
    at ../nptl/sysdeps/unix/sysv/linux/pthread_kill.c:61
#1  0x000000000186b89d in my_write_core (sig=6) at /git/PS-5.7-new/PS-5.7-trunk_dbg/mysys/stacktrace.c:249
#2  0x0000000000e8e8b9 in handle_fatal_signal (sig=6)
    at /git/PS-5.7-new/PS-5.7-trunk_dbg/sql/signal_handler.cc:223
#3  <signal handler called>
#4  0x00007f05f58a61f7 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#5  0x00007f05f58a78e8 in __GI_abort () at abort.c:90
#6  0x00007f05f589f266 in __assert_fail_base (fmt=0x7f05f59f1e68 "%s%s%s:%u: %s%sAssertion '%s' failed.\n%n", 
    assertion=assertion@entry=0x7f05b8a0e577 "log.getting_synced", 
    file=file@entry=0x7f05b8a0db80 "/git/PS-5.7-new/PS-5.7-trunk_dbg/storage/rocksdb/rocksdb/db/db_impl.cc", 
    line=line@entry=731, 
    function=function@entry=0x7f05b8a12dc0 <rocksdb::DBImpl::MarkLogsSynced(unsigned long, bool, rocksdb::Status const&)::__PRETTY_FUNCTION__> "void rocksdb::DBImpl::MarkLogsSynced(uint64_t, bool, const rocksdb::Status&)")
    at assert.c:92
#7  0x00007f05f589f312 in __GI___assert_fail (assertion=0x7f05b8a0e577 "log.getting_synced", 
    file=0x7f05b8a0db80 "/git/PS-5.7-new/PS-5.7-trunk_dbg/storage/rocksdb/rocksdb/db/db_impl.cc", line=731, 
    function=0x7f05b8a12dc0 <rocksdb::DBImpl::MarkLogsSynced(unsigned long, bool, rocksdb::Status const&)::__PRETTY_FUNCTION__> "void rocksdb::DBImpl::MarkLogsSynced(uint64_t, bool, const rocksdb::Status&)") at assert.c:101
#8  0x00007f05b85623c0 in rocksdb::DBImpl::MarkLogsSynced (this=0x7f05f389ea00, up_to=11, synced_dir=true, 
    status=...) at /git/PS-5.7-new/PS-5.7-trunk_dbg/storage/rocksdb/rocksdb/db/db_impl.cc:731
#9  0x00007f05b858692c in rocksdb::DBImpl::WriteImpl (this=0x7f05f389ea00, write_options=..., my_batch=
    0x7f057d50f200, callback=0x0, log_used=0x0, log_ref=0, disable_memtable=false, seq_used=0x0)
    at /git/PS-5.7-new/PS-5.7-trunk_dbg/storage/rocksdb/rocksdb/db/db_impl_write.cc:305
#10 0x00007f05b8585641 in rocksdb::DBImpl::Write (this=0x7f05f389ea00, write_options=..., 
    my_batch=0x7f057d50f200) at /git/PS-5.7-new/PS-5.7-trunk_dbg/storage/rocksdb/rocksdb/db/db_impl_write.cc:49
#11 0x00007f05b89537ee in rocksdb::WriteCommittedTxn::CommitWithoutPrepareInternal (this=0x7f057d8b7400)
    at /git/PS-5.7-new/PS-5.7-trunk_dbg/storage/rocksdb/rocksdb/utilities/transactions/pessimistic_transaction.cc:304
#12 0x00007f05b895332f in rocksdb::PessimisticTransaction::Commit (this=0x7f057d8b7400)
    at /git/PS-5.7-new/PS-5.7-trunk_dbg/storage/rocksdb/rocksdb/utilities/transactions/pessimistic_transaction.cc:264
#13 0x00007f05b841f2a6 in myrocks::Rdb_transaction_impl::commit_no_binlog (this=0x7f057d478100)
    at /git/PS-5.7-new/PS-5.7-trunk_dbg/storage/rocksdb/ha_rocksdb.cc:2147
#14 0x00007f05b841e6b7 in myrocks::Rdb_transaction::commit (this=0x7f057d478100)
    at /git/PS-5.7-new/PS-5.7-trunk_dbg/storage/rocksdb/ha_rocksdb.cc:1880
#15 0x00007f05b83f3a2f in myrocks::rocksdb_commit (hton=0x7f05eb7f0f20, thd=0x7f057d412000, commit_tx=false)
    at /git/PS-5.7-new/PS-5.7-trunk_dbg/storage/rocksdb/ha_rocksdb.cc:2879
#16 0x0000000000f1db15 in ha_commit_low (thd=0x7f057d412000, all=false, run_after_commit=true)
    at /git/PS-5.7-new/PS-5.7-trunk_dbg/sql/handler.cc:1933
#17 0x0000000001637819 in TC_LOG_MMAP::commit (this=0x2ce2040 <tc_log_mmap>, thd=0x7f057d412000, all=false)
    at /git/PS-5.7-new/PS-5.7-trunk_dbg/sql/tc_log.cc:306
#18 0x0000000000f1d7f4 in ha_commit_trans (thd=0x7f057d412000, all=false, ignore_global_read_lock=false)
    at /git/PS-5.7-new/PS-5.7-trunk_dbg/sql/handler.cc:1829
#19 0x000000000163a379 in trans_commit_stmt (thd=0x7f057d412000)
    at /git/PS-5.7-new/PS-5.7-trunk_dbg/sql/transaction.cc:458
#20 0x0000000001535661 in mysql_execute_command (thd=0x7f057d412000, first_level=true)
    at /git/PS-5.7-new/PS-5.7-trunk_dbg/sql/sql_parse.cc:5297
#21 0x0000000001536dec in mysql_parse (thd=0x7f057d412000, parser_state=0x7f05f7b7b4b0)
    at /git/PS-5.7-new/PS-5.7-trunk_dbg/sql/sql_parse.cc:5897
#22 0x000000000152b973 in dispatch_command (thd=0x7f057d412000, com_data=0x7f05f7b7bc90, command=COM_QUERY)
    at /git/PS-5.7-new/PS-5.7-trunk_dbg/sql/sql_parse.cc:1493
#23 0x000000000152a7b9 in do_command (thd=0x7f057d412000)
    at /git/PS-5.7-new/PS-5.7-trunk_dbg/sql/sql_parse.cc:1021
#24 0x000000000166a3b0 in handle_connection (arg=0x7f057d411040)
    at /git/PS-5.7-new/PS-5.7-trunk_dbg/sql/conn_handler/connection_handler_per_thread.cc:312
#25 0x000000000189aee7 in pfs_spawn_thread (arg=0x7f05e9bebe20)
    at /git/PS-5.7-new/PS-5.7-trunk_dbg/storage/perfschema/pfs.cc:2190
#26 0x00007f05f7589e25 in start_thread (arg=0x7f05f7b7c700) at pthread_create.c:308
#27 0x00007f05f596934d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
@RoelVdP
Copy link
Author

RoelVdP commented Jan 1, 2018

A secondary testcase using --rocksdb-use-direct-io-for-flush-and-compaction=1;

Core was generated by `/sda/FB200717-mysql-5.6.35-linux-x86_64-debug/bin/mysqld --no-defaults --core -'.
Program terminated with signal 6, Aborted.
#0  0x00007f5b189909b1 in __pthread_kill (threadid=<optimized out>, signo=6)
    at ../nptl/sysdeps/unix/sysv/linux/pthread_kill.c:61
61	  val = INTERNAL_SYSCALL (tgkill, err, 3, THREAD_GETMEM (THREAD_SELF, pid),
(gdb) bt
#0  0x00007f5b189909b1 in __pthread_kill (threadid=<optimized out>, signo=6)
    at ../nptl/sysdeps/unix/sysv/linux/pthread_kill.c:61
#1  0x00000000012473d6 in my_write_core (sig=6) at /git/facebook-mysql-5.6_dbg/mysys/stacktrace.c:424
#2  0x0000000000e4a7e4 in handle_fatal_signal (sig=6) at /git/facebook-mysql-5.6_dbg/sql/signal_handler.cc:239
#3  <signal handler called>
#4  0x00007f5b16ebe1f7 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#5  0x00007f5b16ebf8e8 in __GI_abort () at abort.c:90
#6  0x00007f5b16eb7266 in __assert_fail_base (fmt=0x7f5b17009e68 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", 
    assertion=assertion@entry=0x1c608a7 "log.getting_synced", 
    file=file@entry=0x1c60050 "/git/facebook-mysql-5.6_dbg/rocksdb/db/db_impl.cc", line=line@entry=710, 
    function=function@entry=0x1c64fc0 <rocksdb::DBImpl::MarkLogsSynced(unsigned long, bool, rocksdb::Status const&)::__PRETTY_FUNCTION__> "void rocksdb::DBImpl::MarkLogsSynced(uint64_t, bool, const rocksdb::Status&)")
    at assert.c:92
#7  0x00007f5b16eb7312 in __GI___assert_fail (assertion=0x1c608a7 "log.getting_synced", 
    file=0x1c60050 "/git/facebook-mysql-5.6_dbg/rocksdb/db/db_impl.cc", line=710, 
    function=0x1c64fc0 <rocksdb::DBImpl::MarkLogsSynced(unsigned long, bool, rocksdb::Status const&)::__PRETTY_FUNCTION__> "void rocksdb::DBImpl::MarkLogsSynced(uint64_t, bool, const rocksdb::Status&)") at assert.c:101
#8  0x000000000131a1b2 in rocksdb::DBImpl::MarkLogsSynced (this=0x7f5b14c7b800, up_to=11, synced_dir=true, 
    status=...) at /git/facebook-mysql-5.6_dbg/rocksdb/db/db_impl.cc:710
#9  0x000000000135ef22 in rocksdb::DBImpl::WriteImpl (this=0x7f5b14c7b800, write_options=..., my_batch=
    0x7f5afbf3b380, callback=0x0, log_used=0x0, log_ref=0, disable_memtable=false)
    at /git/facebook-mysql-5.6_dbg/rocksdb/db/db_impl_write.cc:297
#10 0x000000000135ddc6 in rocksdb::DBImpl::Write (this=0x7f5b14c7b800, write_options=..., 
    my_batch=0x7f5afbf3b380) at /git/facebook-mysql-5.6_dbg/rocksdb/db/db_impl_write.cc:51
#11 0x00000000012bd333 in myrocks::Rdb_dict_manager::commit (this=0x26b5840 <myrocks::dict_manager>, 
    batch=0x7f5afbf3b380, sync=@0x7f5af27fead7: true)
    at /git/facebook-mysql-5.6_dbg/storage/rocksdb/rdb_datadic.cc:4211
#12 0x00000000012be55f in myrocks::Rdb_dict_manager::finish_indexes_operation (
    this=0x26b5840 <myrocks::dict_manager>, gl_index_ids=std::unordered_set with 2 elements = {...}, 
    dd_type=myrocks::Rdb_key_def::DDL_DROP_INDEX_ONGOING)
    at /git/facebook-mysql-5.6_dbg/storage/rocksdb/rdb_datadic.cc:4622
#13 0x0000000001282298 in myrocks::Rdb_dict_manager::finish_drop_indexes (
    this=0x26b5840 <myrocks::dict_manager>, gl_index_ids=std::unordered_set with 2 elements = {...})
    at /git/facebook-mysql-5.6_dbg/storage/rocksdb/./rdb_datadic.h:1268
#14 0x0000000001272028 in myrocks::Rdb_drop_index_thread::run (this=0x26c6560 <myrocks::rdb_drop_idx_thread>)
    at /git/facebook-mysql-5.6_dbg/storage/rocksdb/ha_rocksdb.cc:9424
#15 0x000000000130c630 in myrocks::Rdb_thread::thread_func (thread_ptr=0x26c6560 <myrocks::rdb_drop_idx_thread>)
    at /git/facebook-mysql-5.6_dbg/storage/rocksdb/rdb_threads.cc:32
#16 0x00000000016874af in pfs_spawn_thread (arg=0x7f5b14cc2200)
    at /git/facebook-mysql-5.6_dbg/storage/perfschema/pfs.cc:1860
#17 0x00007f5b1898be25 in start_thread (arg=0x7f5af27ff700) at pthread_create.c:308
#18 0x00007f5b16f8134d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

# mysqld options required for replay:  --no-defaults --default-tmp-storage-engine=MyISAM --rocksdb --skip-innodb --default-storage-engine=RocksDB --rocksdb-use-direct-io-for-flush-and-compaction=1

CREATE TABLE t1(c1 INT,c2 CHAR)PARTITION BY LINEAR KEY(c1) PARTITIONS 5;
insert into t1 values(0,0);
drop TABLE t1;

@RoelVdP
Copy link
Author

RoelVdP commented Jan 1, 2018

Slightly altering the original testcase to this (same SQL, different mysqld options);

# mysqld options required for replay:  --default-tmp-storage-engine=MyISAM --rocksdb --skip-innodb --default-storage-engine=RocksDB --rocksdb-use-direct-reads=1 --log-bin --server-id=1 --binlog-format=row
DROP DATABASE test;
CREATE DATABASE test;
USE test;
CREATE TABLE t1(c1 INT) ENGINE=RocksDB;
CREATE TABLE t2(c1 INT) ENGINE=RocksDB;
INSERT INTO t1 VALUES(1);
ALTER TABLE t1 CHANGE c1 c1 FLOAT;
INSERT INTO t2 VALUES(1);

Crashes in a different way;

Core was generated by `/sda/FB200717-mysql-5.6.35-linux-x86_64-debug/bin/mysqld --no-defaults --core -'.
Program terminated with signal 6, Aborted.
#0  0x00007f3f7b0589b1 in __pthread_kill (threadid=<optimized out>, signo=6)
    at ../nptl/sysdeps/unix/sysv/linux/pthread_kill.c:61
61	  val = INTERNAL_SYSCALL (tgkill, err, 3, THREAD_GETMEM (THREAD_SELF, pid),
(gdb) bt
#0  0x00007f3f7b0589b1 in __pthread_kill (threadid=<optimized out>, signo=6)
    at ../nptl/sysdeps/unix/sysv/linux/pthread_kill.c:61
#1  0x00000000012473d6 in my_write_core (sig=6) at /git/facebook-mysql-5.6_dbg/mysys/stacktrace.c:424
#2  0x0000000000e4a7e4 in handle_fatal_signal (sig=6) at /git/facebook-mysql-5.6_dbg/sql/signal_handler.cc:239
#3  <signal handler called>
#4  0x00007f3f795861f7 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#5  0x00007f3f795878e8 in __GI_abort () at abort.c:90
#6  0x0000000001277bcd in myrocks::rdb_handle_io_error (status=..., err_type=myrocks::RDB_IO_ERROR_TX_COMMIT)
    at /git/facebook-mysql-5.6_dbg/storage/rocksdb/ha_rocksdb.cc:11181
#7  0x0000000001283d96 in myrocks::Rdb_transaction_impl::prepare (this=0x7f3f5240e280, 
    name="\000\000\000\000\000\000\000\001\030\000MySQLXid\001\000\000\000\000\000\000\000\r\000\000\000\000\000\000") at /git/facebook-mysql-5.6_dbg/storage/rocksdb/ha_rocksdb.cc:2085
#8  0x000000000125c2f4 in myrocks::rocksdb_prepare (hton=0x7f3f774c1c00, thd=0x7f3f5aba3000, prepare_tx=false, 
    async=false) at /git/facebook-mysql-5.6_dbg/storage/rocksdb/ha_rocksdb.cc:2663
#9  0x0000000000d30d93 in ha_prepare_low (thd=0x7f3f5aba3000, all=false, async=false)
    at /git/facebook-mysql-5.6_dbg/sql/handler.cc:2264
#10 0x000000000119920b in MYSQL_BIN_LOG::prepare (this=0x26acd80 <mysql_bin_log>, thd=0x7f3f5aba3000, 
    all=false, async=false) at /git/facebook-mysql-5.6_dbg/sql/binlog.cc:7054
#11 0x0000000000d2f209 in ha_commit_trans (thd=0x7f3f5aba3000, all=false, async=false, 
    ignore_global_read_lock=false) at /git/facebook-mysql-5.6_dbg/sql/handler.cc:1540
#12 0x0000000000fe8703 in trans_commit_stmt (thd=0x7f3f5aba3000, async=false)
    at /git/facebook-mysql-5.6_dbg/sql/transaction.cc:452
#13 0x0000000000f0ccb9 in mysql_execute_command (thd=0x7f3f5aba3000, statement_start_time=0x7f3f7b64e818, 
    post_parse=0x7f3f7b64f960) at /git/facebook-mysql-5.6_dbg/sql/sql_parse.cc:6254
#14 0x0000000000f103c3 in mysql_parse (thd=0x7f3f5aba3000, rawbuf=0x7f3f52431010 "INSERT INTO t2 VALUES(1)", 
    length=24, parser_state=0x7f3f7b64f230, last_timer=0x7f3f7b64f960, async_commit=0x7f3f7b64f95f "")
    at /git/facebook-mysql-5.6_dbg/sql/sql_parse.cc:7770
#15 0x0000000000f00f4f in dispatch_command (command=COM_QUERY, thd=0x7f3f5aba3000, 
    packet=0x7f3f5aba8001 "INSERT INTO t2 VALUES(1)", packet_length=24, srv_session=0x0)
    at /git/facebook-mysql-5.6_dbg/sql/sql_parse.cc:1862
#16 0x0000000000eff36c in do_command (thd=0x7f3f5aba3000) at /git/facebook-mysql-5.6_dbg/sql/sql_parse.cc:1207
#17 0x0000000000ebee74 in do_handle_one_connection (thd_arg=0x7f3f5aa93000)
    at /git/facebook-mysql-5.6_dbg/sql/sql_connect.cc:1142
#18 0x0000000000ebe907 in handle_one_connection (arg=0x7f3f5aa93000)
    at /git/facebook-mysql-5.6_dbg/sql/sql_connect.cc:963
#19 0x00000000016874af in pfs_spawn_thread (arg=0x7f3f774c2680)
    at /git/facebook-mysql-5.6_dbg/storage/perfschema/pfs.cc:1860
#20 0x00007f3f7b053e25 in start_thread (arg=0x7f3f7b650700) at pthread_create.c:308
#21 0x00007f3f7964934d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

2018-01-01 11:23:40 22185 [Note] Event Scheduler: Loaded 0 events
2018-01-01 11:23:40 22185 [Note] MySQL git hash: 841d41f48b151a8b8bc6ce5a85eeb603301b5c55 - %cI
2018-01-01 11:23:40 22185 [Note] RocksDB git hash: 857e9960be8a5e4ee3aecfd7d27fe7b1ef6bb352 - %cI
2018-01-01 11:23:40 22185 [Note] /sda/FB200717-mysql-5.6.35-linux-x86_64-debug/bin/mysqld: ready for connections.
Version: '5.6.35-debug-log'  socket: '/sda/FB200717-mysql-5.6.35-linux-x86_64-debug/socket.sock'  port: 13144  MySQL Community Server (GPL)
2018-01-01 11:24:02 22185 [ERROR] LibRocksDB:[/git/facebook-mysql-5.6_dbg/rocksdb/db/db_impl_compaction_flush.cc:1257] Waiting after background flush error: IO error: While open a file for random read: ./.rocksdb/000012.sst: Invalid argumentAccumulated background error counts: 1
2018-01-01 11:24:02 22185 [ERROR] RocksDB: BG thread failed to write to RocksDB, Status Code: 5, Status: IO error: While open a file for random read: ./.rocksdb/000012.sst: Invalid argument
2018-01-01 11:24:03 22185 [ERROR] RocksDB: failed to write to WAL, Status Code: 5, Status: IO error: While open a file for random read: ./.rocksdb/000012.sst: Invalid argument
2018-01-01 11:24:03 22185 [ERROR] MyRocks: aborting on WAL write error.

This issue again repeatable on Percona Server 5.7 as well

@hermanlee
Copy link
Contributor

Yes, tmpfs does not support direct I/O, so it is expected the combination of requesting direct_io and running on tmpfs would result in an error. In this case, the server triggers an assertion when it tries to access the files using direct-read. It is unlikely we will try to improve the behavior (i.e. not crash) unless there is a compelling reason to. A user who attempts to set tmpfs as the data storage and requests direct-reads should quickly discover a problem in the setup.

@RoelVdP
Copy link
Author

RoelVdP commented Jan 5, 2018

If there any way to continue similar to what InnoDB does?

log/master.err:2017-12-28T10:30:19.441221Z 0 [Note] InnoDB: Failed to set O_DIRECT on file /dev/shm/939684/90/data/xb_doublewrite;create: Invalid argument, continuing anyway. O_DIRECT is known to result in 'Invalid argument' on Linux on tmpfs, see MySQL Bug#26662.

@hermanlee
Copy link
Contributor

Currently not. MyRocks and RocksDB does not switch from direct I/O to buffered I/O on failures from direct I/O access. MyRocks can test filesystem access with direct I/O use the appropriate settings before passing them down to RocksDB, but there are also downsides when the system is running with a configuration different than what was specified.

@RoelVdP
Copy link
Author

RoelVdP commented Jan 10, 2018

Would the downsides be adequately covered by a warning in the error log?

@rralcala
Copy link

I have submitted a pull request to address this issue.

rralcala pushed a commit to rralcala/mysql-5.6 that referenced this issue Feb 16, 2018
Summary:
This change adds a check to see whether the filesystem backing
rocksdb_datadir allows O_DIRECT when use_direct_reads is true
and prevents rocksdb from coming up.

This fixes facebook#778
@RoelVdP
Copy link
Author

RoelVdP commented Feb 18, 2018

@rralcala that is great, thank you!

@RoelVdP
Copy link
Author

RoelVdP commented Feb 18, 2018

@hermanlee could you review the patch please?

george-lorch pushed a commit to george-lorch/percona-server that referenced this issue May 16, 2018
Upstream commit ID : fb-mysql-5.6.35/bcd7519d0149a9dd946263b80f3a114625897420
PS-4476 : Merge prod--------

Summary:
This change adds a check to see whether the filesystem backing
rocksdb_datadir allows O_DIRECT when use_direct_reads is true
and prevents rocksdb from coming up.

This fixes facebook/mysql-5.6#778
Closes facebook/mysql-5.6#793

Differential Revision: D6995213

Pulled By: rralcala

fbshipit-source-id: f0da21c
george-lorch pushed a commit to george-lorch/percona-server that referenced this issue May 23, 2018
Upstream commit ID : fb-mysql-5.6.35/bcd7519d0149a9dd946263b80f3a114625897420
PS-4476 : Merge prod--------

Summary:
This change adds a check to see whether the filesystem backing
rocksdb_datadir allows O_DIRECT when use_direct_reads is true
and prevents rocksdb from coming up.

This fixes facebook/mysql-5.6#778
Closes facebook/mysql-5.6#793

Differential Revision: D6995213

Pulled By: rralcala

fbshipit-source-id: f0da21c
george-lorch pushed a commit to george-lorch/percona-server that referenced this issue May 23, 2018
Upstream commit ID : fb-mysql-5.6.35/bcd7519d0149a9dd946263b80f3a114625897420
PS-4476 : Merge prod--------

Summary:
This change adds a check to see whether the filesystem backing
rocksdb_datadir allows O_DIRECT when use_direct_reads is true
and prevents rocksdb from coming up.

This fixes facebook/mysql-5.6#778
Closes facebook/mysql-5.6#793

Differential Revision: D6995213

Pulled By: rralcala

fbshipit-source-id: f0da21c
george-lorch pushed a commit to george-lorch/percona-server that referenced this issue May 24, 2018
Upstream commit ID : fb-mysql-5.6.35/bcd7519d0149a9dd946263b80f3a114625897420
PS-4476 : Merge prod--------

Summary:
This change adds a check to see whether the filesystem backing
rocksdb_datadir allows O_DIRECT when use_direct_reads is true
and prevents rocksdb from coming up.

This fixes facebook/mysql-5.6#778
Closes facebook/mysql-5.6#793

Differential Revision: D6995213

Pulled By: rralcala

fbshipit-source-id: f0da21c
george-lorch pushed a commit to george-lorch/percona-server that referenced this issue May 28, 2018
Upstream commit ID : fb-mysql-5.6.35/bcd7519d0149a9dd946263b80f3a114625897420
PS-4476 : Merge prod--------

Summary:
This change adds a check to see whether the filesystem backing
rocksdb_datadir allows O_DIRECT when use_direct_reads is true
and prevents rocksdb from coming up.

This fixes facebook/mysql-5.6#778
Closes facebook/mysql-5.6#793

Differential Revision: D6995213

Pulled By: rralcala

fbshipit-source-id: f0da21c
george-lorch pushed a commit to george-lorch/percona-server that referenced this issue Jun 19, 2018
Upstream commit ID : fb-mysql-5.6.35/bcd7519d0149a9dd946263b80f3a114625897420
PS-4476 : Merge prod201801

Summary:
This change adds a check to see whether the filesystem backing
rocksdb_datadir allows O_DIRECT when use_direct_reads is true
and prevents rocksdb from coming up.

This fixes facebook/mysql-5.6#778
Closes facebook/mysql-5.6#793

Differential Revision: D6995213

Pulled By: rralcala

fbshipit-source-id: f0da21c
george-lorch pushed a commit to george-lorch/percona-server that referenced this issue Jun 21, 2018
Upstream commit ID : fb-mysql-5.6.35/bcd7519d0149a9dd946263b80f3a114625897420
PS-4476 : Merge prod201801

Summary:
This change adds a check to see whether the filesystem backing
rocksdb_datadir allows O_DIRECT when use_direct_reads is true
and prevents rocksdb from coming up.

This fixes facebook/mysql-5.6#778
Closes facebook/mysql-5.6#793

Differential Revision: D6995213

Pulled By: rralcala

fbshipit-source-id: f0da21c
george-lorch pushed a commit to george-lorch/percona-server that referenced this issue Jun 22, 2018
Upstream commit ID : fb-mysql-5.6.35/bcd7519d0149a9dd946263b80f3a114625897420
PS-4476 : Merge prod201801

Summary:
This change adds a check to see whether the filesystem backing
rocksdb_datadir allows O_DIRECT when use_direct_reads is true
and prevents rocksdb from coming up.

This fixes facebook/mysql-5.6#778
Closes facebook/mysql-5.6#793

Differential Revision: D6995213

Pulled By: rralcala

fbshipit-source-id: f0da21c
facebook-github-bot pushed a commit that referenced this issue Dec 23, 2019
Summary:
This change adds a check to see whether the filesystem backing
rocksdb_datadir allows O_DIRECT when use_direct_reads is true
and prevents rocksdb from coming up.

This fixes #778
Closes #793

Differential Revision: D6995213

Pulled By: rralcala

fbshipit-source-id: e8488f5
inikep pushed a commit to inikep/mysql-5.6 that referenced this issue Aug 12, 2020
…ebook#793)

Summary:
This change adds a check to see whether the filesystem backing
rocksdb_datadir allows O_DIRECT when use_direct_reads is true
and prevents rocksdb from coming up.

This fixes facebook#778
Closes facebook#793

Differential Revision: D6995213

Pulled By: rralcala

fbshipit-source-id: e8488f5
inikep pushed a commit to inikep/mysql-5.6 that referenced this issue Sep 9, 2020
…ebook#793)

Summary:
This change adds a check to see whether the filesystem backing
rocksdb_datadir allows O_DIRECT when use_direct_reads is true
and prevents rocksdb from coming up.

This fixes facebook#778
Closes facebook#793

Differential Revision: D6995213

Pulled By: rralcala

fbshipit-source-id: e8488f5
inikep pushed a commit to inikep/mysql-5.6 that referenced this issue Sep 16, 2020
…ebook#793)

Summary:
This change adds a check to see whether the filesystem backing
rocksdb_datadir allows O_DIRECT when use_direct_reads is true
and prevents rocksdb from coming up.

This fixes facebook#778
Closes facebook#793

Differential Revision: D6995213

Pulled By: rralcala

fbshipit-source-id: e8488f5
inikep pushed a commit to inikep/mysql-5.6 that referenced this issue Oct 5, 2020
…ebook#793)

Summary:
This change adds a check to see whether the filesystem backing
rocksdb_datadir allows O_DIRECT when use_direct_reads is true
and prevents rocksdb from coming up.

This fixes facebook#778
Closes facebook#793

Differential Revision: D6995213

Pulled By: rralcala

fbshipit-source-id: e8488f5
inikep pushed a commit to inikep/mysql-5.6 that referenced this issue Nov 11, 2020
…ebook#793)

Summary:
This change adds a check to see whether the filesystem backing
rocksdb_datadir allows O_DIRECT when use_direct_reads is true
and prevents rocksdb from coming up.

This fixes facebook#778
Closes facebook#793

Differential Revision: D6995213

Pulled By: rralcala

fbshipit-source-id: e8488f5
facebook-github-bot pushed a commit that referenced this issue Mar 8, 2021
… (#793)

Summary:
This change adds a check to see whether the filesystem backing
rocksdb_datadir allows O_DIRECT when use_direct_reads is true
and prevents rocksdb from coming up.

This fixes #778
Closes #793

Differential Revision: D6995213 (a62014f)

Pulled By: rralcala

fbshipit-source-id: c61900b39ab
inikep pushed a commit to inikep/mysql-5.6 that referenced this issue Aug 16, 2021
…ebook#793) (facebook#793)

Summary:
This change adds a check to see whether the filesystem backing
rocksdb_datadir allows O_DIRECT when use_direct_reads is true
and prevents rocksdb from coming up.

This fixes facebook#778
Closes facebook#793

Differential Revision: D6995213 (facebook@a62014f)

Pulled By: rralcala

fbshipit-source-id: c61900b39ab
inikep pushed a commit to inikep/mysql-5.6 that referenced this issue Aug 17, 2021
…ebook#793) (facebook#793)

Summary:
This change adds a check to see whether the filesystem backing
rocksdb_datadir allows O_DIRECT when use_direct_reads is true
and prevents rocksdb from coming up.

This fixes facebook#778
Closes facebook#793

Differential Revision: D6995213 (facebook@a62014f)

Pulled By: rralcala

fbshipit-source-id: c61900b39ab
inikep pushed a commit to inikep/mysql-5.6 that referenced this issue Aug 30, 2021
…ebook#793) (facebook#793)

Summary:
This change adds a check to see whether the filesystem backing
rocksdb_datadir allows O_DIRECT when use_direct_reads is true
and prevents rocksdb from coming up.

This fixes facebook#778
Closes facebook#793

Differential Revision: D6995213 (facebook@a62014f)

Pulled By: rralcala

fbshipit-source-id: c61900b39ab
inikep pushed a commit to inikep/mysql-5.6 that referenced this issue Aug 30, 2021
…ebook#793) (facebook#793)

Summary:
This change adds a check to see whether the filesystem backing
rocksdb_datadir allows O_DIRECT when use_direct_reads is true
and prevents rocksdb from coming up.

This fixes facebook#778
Closes facebook#793

Differential Revision: D6995213 (facebook@a62014f)

Pulled By: rralcala

fbshipit-source-id: c61900b39ab
inikep pushed a commit to inikep/mysql-5.6 that referenced this issue Aug 31, 2021
…ebook#793) (facebook#793)

Summary:
This change adds a check to see whether the filesystem backing
rocksdb_datadir allows O_DIRECT when use_direct_reads is true
and prevents rocksdb from coming up.

This fixes facebook#778
Closes facebook#793

Differential Revision: D6995213 (facebook@a62014f)

Pulled By: rralcala

fbshipit-source-id: c61900b39ab
inikep pushed a commit to inikep/mysql-5.6 that referenced this issue Sep 1, 2021
…ebook#793) (facebook#793)

Summary:
This change adds a check to see whether the filesystem backing
rocksdb_datadir allows O_DIRECT when use_direct_reads is true
and prevents rocksdb from coming up.

This fixes facebook#778
Closes facebook#793

Differential Revision: D6995213 (facebook@a62014f)

Pulled By: rralcala

fbshipit-source-id: c61900b39ab
inikep pushed a commit to inikep/mysql-5.6 that referenced this issue Sep 2, 2021
…ebook#793) (facebook#793)

Summary:
This change adds a check to see whether the filesystem backing
rocksdb_datadir allows O_DIRECT when use_direct_reads is true
and prevents rocksdb from coming up.

This fixes facebook#778
Closes facebook#793

Differential Revision: D6995213 (facebook@a62014f)

Pulled By: rralcala

fbshipit-source-id: c61900b39ab
hermanlee pushed a commit that referenced this issue Jan 10, 2022
… (#793)

Summary:
This change adds a check to see whether the filesystem backing
rocksdb_datadir allows O_DIRECT when use_direct_reads is true
and prevents rocksdb from coming up.

This fixes #778
Closes #793

Differential Revision: D6995213

Pulled By: rralcala
inikep pushed a commit to inikep/mysql-5.6 that referenced this issue Jan 17, 2022
…ebook#793) (facebook#793)

Summary:
This change adds a check to see whether the filesystem backing
rocksdb_datadir allows O_DIRECT when use_direct_reads is true
and prevents rocksdb from coming up.

This fixes facebook#778
Closes facebook#793

Differential Revision: D6995213

Pulled By: rralcala
inikep pushed a commit to inikep/mysql-5.6 that referenced this issue Jan 17, 2022
…ebook#793) (facebook#793)

Summary:
This change adds a check to see whether the filesystem backing
rocksdb_datadir allows O_DIRECT when use_direct_reads is true
and prevents rocksdb from coming up.

This fixes facebook#778
Closes facebook#793

Differential Revision: D6995213

Pulled By: rralcala
ldonoso pushed a commit to ldonoso/percona-server that referenced this issue Mar 15, 2022
…cona#793) (percona#793)

Summary:
This change adds a check to see whether the filesystem backing
rocksdb_datadir allows O_DIRECT when use_direct_reads is true
and prevents rocksdb from coming up.

This fixes facebook/mysql-5.6#778
Closes facebook/mysql-5.6#793

Differential Revision: D6995213

Pulled By: rralcala
inikep pushed a commit to inikep/mysql-5.6 that referenced this issue May 20, 2022
…ebook#793) (facebook#793)

Summary:
This change adds a check to see whether the filesystem backing
rocksdb_datadir allows O_DIRECT when use_direct_reads is true
and prevents rocksdb from coming up.

This fixes facebook#778
Closes facebook#793

Differential Revision: D6995213

Pulled By: rralcala
inikep pushed a commit to inikep/mysql-5.6 that referenced this issue May 23, 2022
…ebook#793) (facebook#793)

Summary:
This change adds a check to see whether the filesystem backing
rocksdb_datadir allows O_DIRECT when use_direct_reads is true
and prevents rocksdb from coming up.

This fixes facebook#778
Closes facebook#793

Differential Revision: D6995213

Pulled By: rralcala
laurynas-biveinis pushed a commit to laurynas-biveinis/mysql-5.6 that referenced this issue Aug 11, 2022
…ebook#793) (facebook#793)

Summary:
This change adds a check to see whether the filesystem backing
rocksdb_datadir allows O_DIRECT when use_direct_reads is true
and prevents rocksdb from coming up.

This fixes facebook#778
Closes facebook#793

Differential Revision: D6995213

Pulled By: rralcala
inikep pushed a commit to inikep/mysql-5.6 that referenced this issue Mar 25, 2023
…ebook#793) (facebook#793)

Summary:
This change adds a check to see whether the filesystem backing
rocksdb_datadir allows O_DIRECT when use_direct_reads is true
and prevents rocksdb from coming up.

This fixes facebook#778
Closes facebook#793

Differential Revision: D6995213

Pulled By: rralcala
inikep pushed a commit to inikep/mysql-5.6 that referenced this issue Mar 28, 2023
…ebook#793) (facebook#793)

Summary:
This change adds a check to see whether the filesystem backing
rocksdb_datadir allows O_DIRECT when use_direct_reads is true
and prevents rocksdb from coming up.

This fixes facebook#778
Closes facebook#793

Differential Revision: D6995213

Pulled By: rralcala
inikep pushed a commit to inikep/mysql-5.6 that referenced this issue Jun 1, 2023
…ebook#793) (facebook#793)

Summary:
This change adds a check to see whether the filesystem backing
rocksdb_datadir allows O_DIRECT when use_direct_reads is true
and prevents rocksdb from coming up.

This fixes facebook#778
Closes facebook#793

Differential Revision: D6995213

Pulled By: rralcala
inikep pushed a commit to inikep/mysql-5.6 that referenced this issue Jun 14, 2023
…ebook#793) (facebook#793)

Summary:
This change adds a check to see whether the filesystem backing
rocksdb_datadir allows O_DIRECT when use_direct_reads is true
and prevents rocksdb from coming up.

This fixes facebook#778
Closes facebook#793

Differential Revision: D6995213

Pulled By: rralcala
inikep pushed a commit to inikep/percona-server that referenced this issue Apr 9, 2024
…cona#793) (percona#793)

Summary:
This change adds a check to see whether the filesystem backing
rocksdb_datadir allows O_DIRECT when use_direct_reads is true
and prevents rocksdb from coming up.

This fixes facebook/mysql-5.6#778
Closes facebook/mysql-5.6#793

Differential Revision: D6995213

Pulled By: rralcala
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants