MyRocks crashes when using tmpfs for data directory & --rocksdb-use-direct-reads=1 #778

RoelVdP · 2017-12-31T23:08:20Z

Program terminated with signal 6, Aborted.
#0  0x00007f0db7cbb9b1 in __pthread_kill (threadid=<optimized out>, signo=6)
    at ../nptl/sysdeps/unix/sysv/linux/pthread_kill.c:61
61	  val = INTERNAL_SYSCALL (tgkill, err, 3, THREAD_GETMEM (THREAD_SELF, pid),
(gdb) bt
#0  0x00007f0db7cbb9b1 in __pthread_kill (threadid=<optimized out>, signo=6)
    at ../nptl/sysdeps/unix/sysv/linux/pthread_kill.c:61
#1  0x00000000012473d6 in my_write_core (sig=6) at /git/facebook-mysql-5.6_dbg/mysys/stacktrace.c:424
#2  0x0000000000e4a7e4 in handle_fatal_signal (sig=6) at /git/facebook-mysql-5.6_dbg/sql/signal_handler.cc:239
#3  <signal handler called>
#4  0x00007f0db61e91f7 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#5  0x00007f0db61ea8e8 in __GI_abort () at abort.c:90
#6  0x00007f0db61e2266 in __assert_fail_base (fmt=0x7f0db6334e68 "%s%s%s:%u: %s%sAssertion '%s' failed.\n%n", 
    assertion=assertion@entry=0x1c608a7 "log.getting_synced", 
    file=file@entry=0x1c60050 "/git/facebook-mysql-5.6_dbg/rocksdb/db/db_impl.cc", line=line@entry=710, 
    function=function@entry=0x1c64fc0 <rocksdb::DBImpl::MarkLogsSynced(unsigned long, bool, rocksdb::Status const&)::__PRETTY_FUNCTION__> "void rocksdb::DBImpl::MarkLogsSynced(uint64_t, bool, const rocksdb::Status&)")
    at assert.c:92
#7  0x00007f0db61e2312 in __GI___assert_fail (assertion=0x1c608a7 "log.getting_synced", 
    file=0x1c60050 "/git/facebook-mysql-5.6_dbg/rocksdb/db/db_impl.cc", line=710, 
    function=0x1c64fc0 <rocksdb::DBImpl::MarkLogsSynced(unsigned long, bool, rocksdb::Status const&)::__PRETTY_FUNCTION__> "void rocksdb::DBImpl::MarkLogsSynced(uint64_t, bool, const rocksdb::Status&)") at assert.c:101
#8  0x000000000131a1b2 in rocksdb::DBImpl::MarkLogsSynced (this=0x7f0db407b800, up_to=11, synced_dir=false, 
    status=...) at /git/facebook-mysql-5.6_dbg/rocksdb/db/db_impl.cc:710
#9  0x000000000135ef22 in rocksdb::DBImpl::WriteImpl (this=0x7f0db407b800, write_options=..., my_batch=
    0x7f0d8f03e140, callback=0x0, log_used=0x0, log_ref=0, disable_memtable=false)
    at /git/facebook-mysql-5.6_dbg/rocksdb/db/db_impl_write.cc:297
#10 0x000000000135ddc6 in rocksdb::DBImpl::Write (this=0x7f0db407b800, write_options=..., 
    my_batch=0x7f0d8f03e140) at /git/facebook-mysql-5.6_dbg/rocksdb/db/db_impl_write.cc:51
#11 0x00000000012bd333 in myrocks::Rdb_dict_manager::commit (this=0x26b5840 <myrocks::dict_manager>, 
    batch=0x7f0d8f03e140, sync=@0x7f0db82ad91a: true)
    at /git/facebook-mysql-5.6_dbg/storage/rocksdb/rdb_datadic.cc:4211
#12 0x0000000001272369 in myrocks::ha_rocksdb::delete_table (this=0x7f0d8f147ec0, 
    tablename=0x7f0db82aebf0 "./test/t1") at /git/facebook-mysql-5.6_dbg/storage/rocksdb/ha_rocksdb.cc:9476
#13 0x0000000000d36c4c in handler::ha_delete_table (this=0x7f0d8f147ec0, name=0x7f0db82aebf0 "./test/t1")
    at /git/facebook-mysql-5.6_dbg/sql/handler.cc:4885
#14 0x0000000000d3167d in ha_delete_table (thd=0x7f0d9779f000, table_type=0x7f0db40c1c00, 
    path=0x7f0db82aebf0 "./test/t1", db=0x7f0d8f147928 "test", alias=0x7f0d8f14792d "t1", 
    generate_warning=false) at /git/facebook-mysql-5.6_dbg/sql/handler.cc:2575
#15 0x0000000000f7a747 in mysql_rm_table_no_locks (thd=0x7f0d9779f000, tables=0x7f0d8f1473c0, if_exists=true, 
    drop_temporary=false, drop_view=true, dont_log_query=true)
    at /git/facebook-mysql-5.6_dbg/sql/sql_table.cc:2545
#16 0x0000000000ec5f35 in mysql_rm_db (thd=0x7f0d9779f000, db=0x7f0d8f0310b0 "test", if_exists=false, 
    silent=false) at /git/facebook-mysql-5.6_dbg/sql/sql_db.cc:1062
#17 0x0000000000f08ebe in mysql_execute_command (thd=0x7f0d9779f000, statement_start_time=0x7f0db82b1818, 
    post_parse=0x7f0db82b2960) at /git/facebook-mysql-5.6_dbg/sql/sql_parse.cc:4952
#18 0x0000000000f103c3 in mysql_parse (thd=0x7f0d9779f000, rawbuf=0x7f0d8f031010 "DROP DATABASE test", 
    length=18, parser_state=0x7f0db82b2230, last_timer=0x7f0db82b2960, async_commit=0x7f0db82b295f "")
    at /git/facebook-mysql-5.6_dbg/sql/sql_parse.cc:7770
#19 0x0000000000f00f4f in dispatch_command (command=COM_QUERY, thd=0x7f0d9779f000, 
    packet=0x7f0d977a4001 "DROP DATABASE test", packet_length=18, srv_session=0x0)
    at /git/facebook-mysql-5.6_dbg/sql/sql_parse.cc:1862
#20 0x0000000000eff36c in do_command (thd=0x7f0d9779f000) at /git/facebook-mysql-5.6_dbg/sql/sql_parse.cc:1207
#21 0x0000000000ebee74 in do_handle_one_connection (thd_arg=0x7f0d97690000)
    at /git/facebook-mysql-5.6_dbg/sql/sql_connect.cc:1142
#22 0x0000000000ebe907 in handle_one_connection (arg=0x7f0d97690000)
    at /git/facebook-mysql-5.6_dbg/sql/sql_connect.cc:963
#23 0x00000000016874af in pfs_spawn_thread (arg=0x7f0db40c2680)
    at /git/facebook-mysql-5.6_dbg/storage/perfschema/pfs.cc:1860
#24 0x00007f0db7cb6e25 in start_thread (arg=0x7f0db82b3700) at pthread_create.c:308
#25 0x00007f0db62ac34d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

2018-01-01 09:29:52 16513 [Note] /sda/FB200717-mysql-5.6.35-linux-x86_64-debug/bin/mysqld: ready for connections.
Version: '5.6.35-debug'  socket: '/sda/FB200717-mysql-5.6.35-linux-x86_64-debug/socket.sock'  port: 18240  MySQL Community Server (GPL)
2018-01-01 09:30:04 16513 [ERROR] LibRocksDB:[/git/facebook-mysql-5.6_dbg/rocksdb/db/db_impl_compaction_flush.cc:1257] Waiting after background flush error: IO error: While open a file for random read: ./.rocksdb/000012.sst: Invalid argumentAccumulated background error counts: 1
2018-01-01 09:30:04 16513 [ERROR] RocksDB: BG thread failed to write to RocksDB, Status Code: 5, Status: IO error: While open a file for random read: ./.rocksdb/000012.sst: Invalid argument
mysqld: /git/facebook-mysql-5.6_dbg/rocksdb/db/db_impl.cc:710: void rocksdb::DBImpl::MarkLogsSynced(uint64_t, bool, const rocksdb::Status&): Assertion 'log.getting_synced' failed.
22:30:05 UTC - mysqld got signal 6 ;

1) Make sure to run this with the data directory on /dev/shm (tmpfs) 
2) Testcase:
# mysqld options required for replay:  --no-defaults --default-tmp-storage-engine=MyISAM --rocksdb --skip-innodb --default-storage-engine=RocksDB --rocksdb-use-direct-reads=1
DROP DATABASE test;
CREATE DATABASE test;
USE test;
CREATE TABLE t1(c1 INT) ENGINE=RocksDB;
CREATE TABLE t2(c1 INT) ENGINE=RocksDB;
INSERT INTO t1 VALUES(1);
ALTER TABLE t1 CHANGE c1 c1 FLOAT;
INSERT INTO t2 VALUES(1);

I suspect that the issue is rocksdb-use-direct-reads=1 in combination with /dev/shm? i.e. similar to InnoDB's;

2017-12-31T20:37:30.084316Z 0 [Note] InnoDB: Failed to set O_DIRECT on file /dev/shm/809118/76/data/xb_doublewrite;create: Invalid argument, continuing anyway. O_DIRECT is known to result in 'Invalid argument' on Linux on tmpfs, see MySQL Bug#26662.

Seems like this situation needs to be covered in MyRocks also. This can also be made to crash Percona Server with MyRocks, using the same testcase and these mysqld options;

# mysqld options required for replay:  --plugin-load-add=rocksdb=ha_rocksdb.so --rocksdb-use-direct-reads=1  

Core was generated by '/sda/PS281217-percona-server-5.7.20-18-linux-x86_64-debug/bin/mysqld --no-defau'.
Program terminated with signal 6, Aborted.
#0  0x00007f05f758e9b1 in __pthread_kill (threadid=<optimized out>, signo=6)
    at ../nptl/sysdeps/unix/sysv/linux/pthread_kill.c:61
61	  val = INTERNAL_SYSCALL (tgkill, err, 3, THREAD_GETMEM (THREAD_SELF, pid),
(gdb) bt
#0  0x00007f05f758e9b1 in __pthread_kill (threadid=<optimized out>, signo=6)
    at ../nptl/sysdeps/unix/sysv/linux/pthread_kill.c:61
#1  0x000000000186b89d in my_write_core (sig=6) at /git/PS-5.7-new/PS-5.7-trunk_dbg/mysys/stacktrace.c:249
#2  0x0000000000e8e8b9 in handle_fatal_signal (sig=6)
    at /git/PS-5.7-new/PS-5.7-trunk_dbg/sql/signal_handler.cc:223
#3  <signal handler called>
#4  0x00007f05f58a61f7 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#5  0x00007f05f58a78e8 in __GI_abort () at abort.c:90
#6  0x00007f05f589f266 in __assert_fail_base (fmt=0x7f05f59f1e68 "%s%s%s:%u: %s%sAssertion '%s' failed.\n%n", 
    assertion=assertion@entry=0x7f05b8a0e577 "log.getting_synced", 
    file=file@entry=0x7f05b8a0db80 "/git/PS-5.7-new/PS-5.7-trunk_dbg/storage/rocksdb/rocksdb/db/db_impl.cc", 
    line=line@entry=731, 
    function=function@entry=0x7f05b8a12dc0 <rocksdb::DBImpl::MarkLogsSynced(unsigned long, bool, rocksdb::Status const&)::__PRETTY_FUNCTION__> "void rocksdb::DBImpl::MarkLogsSynced(uint64_t, bool, const rocksdb::Status&)")
    at assert.c:92
#7  0x00007f05f589f312 in __GI___assert_fail (assertion=0x7f05b8a0e577 "log.getting_synced", 
    file=0x7f05b8a0db80 "/git/PS-5.7-new/PS-5.7-trunk_dbg/storage/rocksdb/rocksdb/db/db_impl.cc", line=731, 
    function=0x7f05b8a12dc0 <rocksdb::DBImpl::MarkLogsSynced(unsigned long, bool, rocksdb::Status const&)::__PRETTY_FUNCTION__> "void rocksdb::DBImpl::MarkLogsSynced(uint64_t, bool, const rocksdb::Status&)") at assert.c:101
#8  0x00007f05b85623c0 in rocksdb::DBImpl::MarkLogsSynced (this=0x7f05f389ea00, up_to=11, synced_dir=true, 
    status=...) at /git/PS-5.7-new/PS-5.7-trunk_dbg/storage/rocksdb/rocksdb/db/db_impl.cc:731
#9  0x00007f05b858692c in rocksdb::DBImpl::WriteImpl (this=0x7f05f389ea00, write_options=..., my_batch=
    0x7f057d50f200, callback=0x0, log_used=0x0, log_ref=0, disable_memtable=false, seq_used=0x0)
    at /git/PS-5.7-new/PS-5.7-trunk_dbg/storage/rocksdb/rocksdb/db/db_impl_write.cc:305
#10 0x00007f05b8585641 in rocksdb::DBImpl::Write (this=0x7f05f389ea00, write_options=..., 
    my_batch=0x7f057d50f200) at /git/PS-5.7-new/PS-5.7-trunk_dbg/storage/rocksdb/rocksdb/db/db_impl_write.cc:49
#11 0x00007f05b89537ee in rocksdb::WriteCommittedTxn::CommitWithoutPrepareInternal (this=0x7f057d8b7400)
    at /git/PS-5.7-new/PS-5.7-trunk_dbg/storage/rocksdb/rocksdb/utilities/transactions/pessimistic_transaction.cc:304
#12 0x00007f05b895332f in rocksdb::PessimisticTransaction::Commit (this=0x7f057d8b7400)
    at /git/PS-5.7-new/PS-5.7-trunk_dbg/storage/rocksdb/rocksdb/utilities/transactions/pessimistic_transaction.cc:264
#13 0x00007f05b841f2a6 in myrocks::Rdb_transaction_impl::commit_no_binlog (this=0x7f057d478100)
    at /git/PS-5.7-new/PS-5.7-trunk_dbg/storage/rocksdb/ha_rocksdb.cc:2147
#14 0x00007f05b841e6b7 in myrocks::Rdb_transaction::commit (this=0x7f057d478100)
    at /git/PS-5.7-new/PS-5.7-trunk_dbg/storage/rocksdb/ha_rocksdb.cc:1880
#15 0x00007f05b83f3a2f in myrocks::rocksdb_commit (hton=0x7f05eb7f0f20, thd=0x7f057d412000, commit_tx=false)
    at /git/PS-5.7-new/PS-5.7-trunk_dbg/storage/rocksdb/ha_rocksdb.cc:2879
#16 0x0000000000f1db15 in ha_commit_low (thd=0x7f057d412000, all=false, run_after_commit=true)
    at /git/PS-5.7-new/PS-5.7-trunk_dbg/sql/handler.cc:1933
#17 0x0000000001637819 in TC_LOG_MMAP::commit (this=0x2ce2040 <tc_log_mmap>, thd=0x7f057d412000, all=false)
    at /git/PS-5.7-new/PS-5.7-trunk_dbg/sql/tc_log.cc:306
#18 0x0000000000f1d7f4 in ha_commit_trans (thd=0x7f057d412000, all=false, ignore_global_read_lock=false)
    at /git/PS-5.7-new/PS-5.7-trunk_dbg/sql/handler.cc:1829
#19 0x000000000163a379 in trans_commit_stmt (thd=0x7f057d412000)
    at /git/PS-5.7-new/PS-5.7-trunk_dbg/sql/transaction.cc:458
#20 0x0000000001535661 in mysql_execute_command (thd=0x7f057d412000, first_level=true)
    at /git/PS-5.7-new/PS-5.7-trunk_dbg/sql/sql_parse.cc:5297
#21 0x0000000001536dec in mysql_parse (thd=0x7f057d412000, parser_state=0x7f05f7b7b4b0)
    at /git/PS-5.7-new/PS-5.7-trunk_dbg/sql/sql_parse.cc:5897
#22 0x000000000152b973 in dispatch_command (thd=0x7f057d412000, com_data=0x7f05f7b7bc90, command=COM_QUERY)
    at /git/PS-5.7-new/PS-5.7-trunk_dbg/sql/sql_parse.cc:1493
#23 0x000000000152a7b9 in do_command (thd=0x7f057d412000)
    at /git/PS-5.7-new/PS-5.7-trunk_dbg/sql/sql_parse.cc:1021
#24 0x000000000166a3b0 in handle_connection (arg=0x7f057d411040)
    at /git/PS-5.7-new/PS-5.7-trunk_dbg/sql/conn_handler/connection_handler_per_thread.cc:312
#25 0x000000000189aee7 in pfs_spawn_thread (arg=0x7f05e9bebe20)
    at /git/PS-5.7-new/PS-5.7-trunk_dbg/storage/perfschema/pfs.cc:2190
#26 0x00007f05f7589e25 in start_thread (arg=0x7f05f7b7c700) at pthread_create.c:308
#27 0x00007f05f596934d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

The text was updated successfully, but these errors were encountered:

RoelVdP · 2018-01-01T00:12:25Z

A secondary testcase using --rocksdb-use-direct-io-for-flush-and-compaction=1;

Core was generated by `/sda/FB200717-mysql-5.6.35-linux-x86_64-debug/bin/mysqld --no-defaults --core -'.
Program terminated with signal 6, Aborted.
#0  0x00007f5b189909b1 in __pthread_kill (threadid=<optimized out>, signo=6)
    at ../nptl/sysdeps/unix/sysv/linux/pthread_kill.c:61
61	  val = INTERNAL_SYSCALL (tgkill, err, 3, THREAD_GETMEM (THREAD_SELF, pid),
(gdb) bt
#0  0x00007f5b189909b1 in __pthread_kill (threadid=<optimized out>, signo=6)
    at ../nptl/sysdeps/unix/sysv/linux/pthread_kill.c:61
#1  0x00000000012473d6 in my_write_core (sig=6) at /git/facebook-mysql-5.6_dbg/mysys/stacktrace.c:424
#2  0x0000000000e4a7e4 in handle_fatal_signal (sig=6) at /git/facebook-mysql-5.6_dbg/sql/signal_handler.cc:239
#3  <signal handler called>
#4  0x00007f5b16ebe1f7 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#5  0x00007f5b16ebf8e8 in __GI_abort () at abort.c:90
#6  0x00007f5b16eb7266 in __assert_fail_base (fmt=0x7f5b17009e68 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", 
    assertion=assertion@entry=0x1c608a7 "log.getting_synced", 
    file=file@entry=0x1c60050 "/git/facebook-mysql-5.6_dbg/rocksdb/db/db_impl.cc", line=line@entry=710, 
    function=function@entry=0x1c64fc0 <rocksdb::DBImpl::MarkLogsSynced(unsigned long, bool, rocksdb::Status const&)::__PRETTY_FUNCTION__> "void rocksdb::DBImpl::MarkLogsSynced(uint64_t, bool, const rocksdb::Status&)")
    at assert.c:92
#7  0x00007f5b16eb7312 in __GI___assert_fail (assertion=0x1c608a7 "log.getting_synced", 
    file=0x1c60050 "/git/facebook-mysql-5.6_dbg/rocksdb/db/db_impl.cc", line=710, 
    function=0x1c64fc0 <rocksdb::DBImpl::MarkLogsSynced(unsigned long, bool, rocksdb::Status const&)::__PRETTY_FUNCTION__> "void rocksdb::DBImpl::MarkLogsSynced(uint64_t, bool, const rocksdb::Status&)") at assert.c:101
#8  0x000000000131a1b2 in rocksdb::DBImpl::MarkLogsSynced (this=0x7f5b14c7b800, up_to=11, synced_dir=true, 
    status=...) at /git/facebook-mysql-5.6_dbg/rocksdb/db/db_impl.cc:710
#9  0x000000000135ef22 in rocksdb::DBImpl::WriteImpl (this=0x7f5b14c7b800, write_options=..., my_batch=
    0x7f5afbf3b380, callback=0x0, log_used=0x0, log_ref=0, disable_memtable=false)
    at /git/facebook-mysql-5.6_dbg/rocksdb/db/db_impl_write.cc:297
#10 0x000000000135ddc6 in rocksdb::DBImpl::Write (this=0x7f5b14c7b800, write_options=..., 
    my_batch=0x7f5afbf3b380) at /git/facebook-mysql-5.6_dbg/rocksdb/db/db_impl_write.cc:51
#11 0x00000000012bd333 in myrocks::Rdb_dict_manager::commit (this=0x26b5840 <myrocks::dict_manager>, 
    batch=0x7f5afbf3b380, sync=@0x7f5af27fead7: true)
    at /git/facebook-mysql-5.6_dbg/storage/rocksdb/rdb_datadic.cc:4211
#12 0x00000000012be55f in myrocks::Rdb_dict_manager::finish_indexes_operation (
    this=0x26b5840 <myrocks::dict_manager>, gl_index_ids=std::unordered_set with 2 elements = {...}, 
    dd_type=myrocks::Rdb_key_def::DDL_DROP_INDEX_ONGOING)
    at /git/facebook-mysql-5.6_dbg/storage/rocksdb/rdb_datadic.cc:4622
#13 0x0000000001282298 in myrocks::Rdb_dict_manager::finish_drop_indexes (
    this=0x26b5840 <myrocks::dict_manager>, gl_index_ids=std::unordered_set with 2 elements = {...})
    at /git/facebook-mysql-5.6_dbg/storage/rocksdb/./rdb_datadic.h:1268
#14 0x0000000001272028 in myrocks::Rdb_drop_index_thread::run (this=0x26c6560 <myrocks::rdb_drop_idx_thread>)
    at /git/facebook-mysql-5.6_dbg/storage/rocksdb/ha_rocksdb.cc:9424
#15 0x000000000130c630 in myrocks::Rdb_thread::thread_func (thread_ptr=0x26c6560 <myrocks::rdb_drop_idx_thread>)
    at /git/facebook-mysql-5.6_dbg/storage/rocksdb/rdb_threads.cc:32
#16 0x00000000016874af in pfs_spawn_thread (arg=0x7f5b14cc2200)
    at /git/facebook-mysql-5.6_dbg/storage/perfschema/pfs.cc:1860
#17 0x00007f5b1898be25 in start_thread (arg=0x7f5af27ff700) at pthread_create.c:308
#18 0x00007f5b16f8134d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

# mysqld options required for replay:  --no-defaults --default-tmp-storage-engine=MyISAM --rocksdb --skip-innodb --default-storage-engine=RocksDB --rocksdb-use-direct-io-for-flush-and-compaction=1

CREATE TABLE t1(c1 INT,c2 CHAR)PARTITION BY LINEAR KEY(c1) PARTITIONS 5;
insert into t1 values(0,0);
drop TABLE t1;

RoelVdP · 2018-01-01T00:56:29Z

Slightly altering the original testcase to this (same SQL, different mysqld options);

# mysqld options required for replay:  --default-tmp-storage-engine=MyISAM --rocksdb --skip-innodb --default-storage-engine=RocksDB --rocksdb-use-direct-reads=1 --log-bin --server-id=1 --binlog-format=row
DROP DATABASE test;
CREATE DATABASE test;
USE test;
CREATE TABLE t1(c1 INT) ENGINE=RocksDB;
CREATE TABLE t2(c1 INT) ENGINE=RocksDB;
INSERT INTO t1 VALUES(1);
ALTER TABLE t1 CHANGE c1 c1 FLOAT;
INSERT INTO t2 VALUES(1);

Crashes in a different way;

Core was generated by `/sda/FB200717-mysql-5.6.35-linux-x86_64-debug/bin/mysqld --no-defaults --core -'.
Program terminated with signal 6, Aborted.
#0  0x00007f3f7b0589b1 in __pthread_kill (threadid=<optimized out>, signo=6)
    at ../nptl/sysdeps/unix/sysv/linux/pthread_kill.c:61
61	  val = INTERNAL_SYSCALL (tgkill, err, 3, THREAD_GETMEM (THREAD_SELF, pid),
(gdb) bt
#0  0x00007f3f7b0589b1 in __pthread_kill (threadid=<optimized out>, signo=6)
    at ../nptl/sysdeps/unix/sysv/linux/pthread_kill.c:61
#1  0x00000000012473d6 in my_write_core (sig=6) at /git/facebook-mysql-5.6_dbg/mysys/stacktrace.c:424
#2  0x0000000000e4a7e4 in handle_fatal_signal (sig=6) at /git/facebook-mysql-5.6_dbg/sql/signal_handler.cc:239
#3  <signal handler called>
#4  0x00007f3f795861f7 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#5  0x00007f3f795878e8 in __GI_abort () at abort.c:90
#6  0x0000000001277bcd in myrocks::rdb_handle_io_error (status=..., err_type=myrocks::RDB_IO_ERROR_TX_COMMIT)
    at /git/facebook-mysql-5.6_dbg/storage/rocksdb/ha_rocksdb.cc:11181
#7  0x0000000001283d96 in myrocks::Rdb_transaction_impl::prepare (this=0x7f3f5240e280, 
    name="\000\000\000\000\000\000\000\001\030\000MySQLXid\001\000\000\000\000\000\000\000\r\000\000\000\000\000\000") at /git/facebook-mysql-5.6_dbg/storage/rocksdb/ha_rocksdb.cc:2085
#8  0x000000000125c2f4 in myrocks::rocksdb_prepare (hton=0x7f3f774c1c00, thd=0x7f3f5aba3000, prepare_tx=false, 
    async=false) at /git/facebook-mysql-5.6_dbg/storage/rocksdb/ha_rocksdb.cc:2663
#9  0x0000000000d30d93 in ha_prepare_low (thd=0x7f3f5aba3000, all=false, async=false)
    at /git/facebook-mysql-5.6_dbg/sql/handler.cc:2264
#10 0x000000000119920b in MYSQL_BIN_LOG::prepare (this=0x26acd80 <mysql_bin_log>, thd=0x7f3f5aba3000, 
    all=false, async=false) at /git/facebook-mysql-5.6_dbg/sql/binlog.cc:7054
#11 0x0000000000d2f209 in ha_commit_trans (thd=0x7f3f5aba3000, all=false, async=false, 
    ignore_global_read_lock=false) at /git/facebook-mysql-5.6_dbg/sql/handler.cc:1540
#12 0x0000000000fe8703 in trans_commit_stmt (thd=0x7f3f5aba3000, async=false)
    at /git/facebook-mysql-5.6_dbg/sql/transaction.cc:452
#13 0x0000000000f0ccb9 in mysql_execute_command (thd=0x7f3f5aba3000, statement_start_time=0x7f3f7b64e818, 
    post_parse=0x7f3f7b64f960) at /git/facebook-mysql-5.6_dbg/sql/sql_parse.cc:6254
#14 0x0000000000f103c3 in mysql_parse (thd=0x7f3f5aba3000, rawbuf=0x7f3f52431010 "INSERT INTO t2 VALUES(1)", 
    length=24, parser_state=0x7f3f7b64f230, last_timer=0x7f3f7b64f960, async_commit=0x7f3f7b64f95f "")
    at /git/facebook-mysql-5.6_dbg/sql/sql_parse.cc:7770
#15 0x0000000000f00f4f in dispatch_command (command=COM_QUERY, thd=0x7f3f5aba3000, 
    packet=0x7f3f5aba8001 "INSERT INTO t2 VALUES(1)", packet_length=24, srv_session=0x0)
    at /git/facebook-mysql-5.6_dbg/sql/sql_parse.cc:1862
#16 0x0000000000eff36c in do_command (thd=0x7f3f5aba3000) at /git/facebook-mysql-5.6_dbg/sql/sql_parse.cc:1207
#17 0x0000000000ebee74 in do_handle_one_connection (thd_arg=0x7f3f5aa93000)
    at /git/facebook-mysql-5.6_dbg/sql/sql_connect.cc:1142
#18 0x0000000000ebe907 in handle_one_connection (arg=0x7f3f5aa93000)
    at /git/facebook-mysql-5.6_dbg/sql/sql_connect.cc:963
#19 0x00000000016874af in pfs_spawn_thread (arg=0x7f3f774c2680)
    at /git/facebook-mysql-5.6_dbg/storage/perfschema/pfs.cc:1860
#20 0x00007f3f7b053e25 in start_thread (arg=0x7f3f7b650700) at pthread_create.c:308
#21 0x00007f3f7964934d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

2018-01-01 11:23:40 22185 [Note] Event Scheduler: Loaded 0 events
2018-01-01 11:23:40 22185 [Note] MySQL git hash: 841d41f48b151a8b8bc6ce5a85eeb603301b5c55 - %cI
2018-01-01 11:23:40 22185 [Note] RocksDB git hash: 857e9960be8a5e4ee3aecfd7d27fe7b1ef6bb352 - %cI
2018-01-01 11:23:40 22185 [Note] /sda/FB200717-mysql-5.6.35-linux-x86_64-debug/bin/mysqld: ready for connections.
Version: '5.6.35-debug-log'  socket: '/sda/FB200717-mysql-5.6.35-linux-x86_64-debug/socket.sock'  port: 13144  MySQL Community Server (GPL)
2018-01-01 11:24:02 22185 [ERROR] LibRocksDB:[/git/facebook-mysql-5.6_dbg/rocksdb/db/db_impl_compaction_flush.cc:1257] Waiting after background flush error: IO error: While open a file for random read: ./.rocksdb/000012.sst: Invalid argumentAccumulated background error counts: 1
2018-01-01 11:24:02 22185 [ERROR] RocksDB: BG thread failed to write to RocksDB, Status Code: 5, Status: IO error: While open a file for random read: ./.rocksdb/000012.sst: Invalid argument
2018-01-01 11:24:03 22185 [ERROR] RocksDB: failed to write to WAL, Status Code: 5, Status: IO error: While open a file for random read: ./.rocksdb/000012.sst: Invalid argument
2018-01-01 11:24:03 22185 [ERROR] MyRocks: aborting on WAL write error.

This issue again repeatable on Percona Server 5.7 as well

hermanlee · 2018-01-02T21:23:38Z

Yes, tmpfs does not support direct I/O, so it is expected the combination of requesting direct_io and running on tmpfs would result in an error. In this case, the server triggers an assertion when it tries to access the files using direct-read. It is unlikely we will try to improve the behavior (i.e. not crash) unless there is a compelling reason to. A user who attempts to set tmpfs as the data storage and requests direct-reads should quickly discover a problem in the setup.

RoelVdP · 2018-01-05T02:26:27Z

If there any way to continue similar to what InnoDB does?

log/master.err:2017-12-28T10:30:19.441221Z 0 [Note] InnoDB: Failed to set O_DIRECT on file /dev/shm/939684/90/data/xb_doublewrite;create: Invalid argument, continuing anyway. O_DIRECT is known to result in 'Invalid argument' on Linux on tmpfs, see MySQL Bug#26662.

hermanlee · 2018-01-10T23:34:49Z

Currently not. MyRocks and RocksDB does not switch from direct I/O to buffered I/O on failures from direct I/O access. MyRocks can test filesystem access with direct I/O use the appropriate settings before passing them down to RocksDB, but there are also downsides when the system is running with a configuration different than what was specified.

RoelVdP · 2018-01-10T23:54:20Z

Would the downsides be adequately covered by a warning in the error log?

rralcala · 2018-02-14T21:31:01Z

I have submitted a pull request to address this issue.

Summary: This change adds a check to see whether the filesystem backing rocksdb_datadir allows O_DIRECT when use_direct_reads is true and prevents rocksdb from coming up. This fixes facebook#778

RoelVdP · 2018-02-18T23:21:18Z

@rralcala that is great, thank you!

RoelVdP · 2018-02-18T23:21:57Z

@hermanlee could you review the patch please?

Upstream commit ID : fb-mysql-5.6.35/bcd7519d0149a9dd946263b80f3a114625897420 PS-4476 : Merge prod-------- Summary: This change adds a check to see whether the filesystem backing rocksdb_datadir allows O_DIRECT when use_direct_reads is true and prevents rocksdb from coming up. This fixes facebook/mysql-5.6#778 Closes facebook/mysql-5.6#793 Differential Revision: D6995213 Pulled By: rralcala fbshipit-source-id: f0da21c

Upstream commit ID : fb-mysql-5.6.35/bcd7519d0149a9dd946263b80f3a114625897420 PS-4476 : Merge prod201801 Summary: This change adds a check to see whether the filesystem backing rocksdb_datadir allows O_DIRECT when use_direct_reads is true and prevents rocksdb from coming up. This fixes facebook/mysql-5.6#778 Closes facebook/mysql-5.6#793 Differential Revision: D6995213 Pulled By: rralcala fbshipit-source-id: f0da21c

Summary: This change adds a check to see whether the filesystem backing rocksdb_datadir allows O_DIRECT when use_direct_reads is true and prevents rocksdb from coming up. This fixes #778 Closes #793 Differential Revision: D6995213 Pulled By: rralcala fbshipit-source-id: e8488f5

…ebook#793) Summary: This change adds a check to see whether the filesystem backing rocksdb_datadir allows O_DIRECT when use_direct_reads is true and prevents rocksdb from coming up. This fixes facebook#778 Closes facebook#793 Differential Revision: D6995213 Pulled By: rralcala fbshipit-source-id: e8488f5

… (#793) Summary: This change adds a check to see whether the filesystem backing rocksdb_datadir allows O_DIRECT when use_direct_reads is true and prevents rocksdb from coming up. This fixes #778 Closes #793 Differential Revision: D6995213 (a62014f) Pulled By: rralcala fbshipit-source-id: c61900b39ab

…ebook#793) (facebook#793) Summary: This change adds a check to see whether the filesystem backing rocksdb_datadir allows O_DIRECT when use_direct_reads is true and prevents rocksdb from coming up. This fixes facebook#778 Closes facebook#793 Differential Revision: D6995213 (facebook@a62014f) Pulled By: rralcala fbshipit-source-id: c61900b39ab

… (#793) Summary: This change adds a check to see whether the filesystem backing rocksdb_datadir allows O_DIRECT when use_direct_reads is true and prevents rocksdb from coming up. This fixes #778 Closes #793 Differential Revision: D6995213 Pulled By: rralcala

…ebook#793) (facebook#793) Summary: This change adds a check to see whether the filesystem backing rocksdb_datadir allows O_DIRECT when use_direct_reads is true and prevents rocksdb from coming up. This fixes facebook#778 Closes facebook#793 Differential Revision: D6995213 Pulled By: rralcala

…cona#793) (percona#793) Summary: This change adds a check to see whether the filesystem backing rocksdb_datadir allows O_DIRECT when use_direct_reads is true and prevents rocksdb from coming up. This fixes facebook/mysql-5.6#778 Closes facebook/mysql-5.6#793 Differential Revision: D6995213 Pulled By: rralcala

…ebook#793) (facebook#793) Summary: This change adds a check to see whether the filesystem backing rocksdb_datadir allows O_DIRECT when use_direct_reads is true and prevents rocksdb from coming up. This fixes facebook#778 Closes facebook#793 Differential Revision: D6995213 Pulled By: rralcala

…cona#793) (percona#793) Summary: This change adds a check to see whether the filesystem backing rocksdb_datadir allows O_DIRECT when use_direct_reads is true and prevents rocksdb from coming up. This fixes facebook/mysql-5.6#778 Closes facebook/mysql-5.6#793 Differential Revision: D6995213 Pulled By: rralcala

rralcala mentioned this issue Feb 14, 2018

Abort rocksdb startup if O_DIRECT is requested but not supported #793

Closed

facebook-github-bot closed this as completed in bcd7519 Feb 20, 2018

george-lorch mentioned this issue Nov 1, 2018

rocksdb/env/io_posix.cc:449: rocksdb::PosixMmapReadableFile::PosixMmapReadableFile(int, const string&, void*, size_t, const rocksdb::EnvOptions&): Assertion `!options.use_direct_reads' failed. #895

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

MyRocks crashes when using tmpfs for data directory & --rocksdb-use-direct-reads=1 #778

MyRocks crashes when using tmpfs for data directory & --rocksdb-use-direct-reads=1 #778

RoelVdP commented Dec 31, 2017 •

edited

Loading

RoelVdP commented Jan 1, 2018 •

edited

Loading

RoelVdP commented Jan 1, 2018

hermanlee commented Jan 2, 2018

RoelVdP commented Jan 5, 2018 •

edited

Loading

hermanlee commented Jan 10, 2018

RoelVdP commented Jan 10, 2018

rralcala commented Feb 14, 2018

RoelVdP commented Feb 18, 2018

RoelVdP commented Feb 18, 2018

MyRocks crashes when using tmpfs for data directory & --rocksdb-use-direct-reads=1 #778

MyRocks crashes when using tmpfs for data directory & --rocksdb-use-direct-reads=1 #778

Comments

RoelVdP commented Dec 31, 2017 • edited Loading

RoelVdP commented Jan 1, 2018 • edited Loading

RoelVdP commented Jan 1, 2018

hermanlee commented Jan 2, 2018

RoelVdP commented Jan 5, 2018 • edited Loading

hermanlee commented Jan 10, 2018

RoelVdP commented Jan 10, 2018

rralcala commented Feb 14, 2018

RoelVdP commented Feb 18, 2018

RoelVdP commented Feb 18, 2018

RoelVdP commented Dec 31, 2017 •

edited

Loading

RoelVdP commented Jan 1, 2018 •

edited

Loading

RoelVdP commented Jan 5, 2018 •

edited

Loading