ClickHouse can't start on read-only filesystem under certain conditions

[nvartolomei]

Describe the bug or unexpected behaviour
I'm testing out multiple disk feature and how it works under different failure conditions, though this is probably unrelated to this feature.

If I have a table, write data to it, wait for some merges to happen, re-mount the fs where parts are stored as read-only and restart ClickHouse then it fails to start.

Poco::Exception. Code: 1000, e.code() = 30, e.displayText() = File is read-only: /mnt/disk1/data/default/jbod_table/all_1_1_00. 0xbc3ed9c Poco::FileReadOnlyException::FileReadOnlyException(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int)  in /usr/bin/clickhouse
1. 0x4e4bf03 Poco::FileImpl::handleLastErrorImpl(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)  in /usr/bin/clickhouse
2. 0x4e4c3de ?  in /usr/bin/clickhouse
3. 0x9044c6f DB::MergeTreeDataPart::remove() const  in /usr/bin/clickhouse
4. 0x8ff4efa DB::MergeTreeData::clearPartsFromFilesystem(std::__1::vector<std::__1::shared_ptr<DB::MergeTreeDataPart const>, std::__1::allocator<std::__1::shared_ptr<DB::MergeTreeDataPart const> > > const&)  in /usr/bin/clickhouse
5. 0x9006e22 DB::MergeTreeData::clearOldPartsFromFilesystem(bool)  in /usr/bin/clickhouse
6. 0x8f08738 DB::StorageMergeTree::shutdown()  in /us
2020.02.12 11:10:54.665540 [ 50 ] {} <Fatal> BaseDaemon: ########################################
2020.02.12 11:10:54.666062 [ 50 ] {} <Fatal> BaseDaemon: (version 20.1.4.14 (official build)) (from thread 1) (no query) Received signal Aborted (6).
2020.02.12 11:10:54.666533 [ 50 ] {} <Fatal> BaseDaemon:
2020.02.12 11:10:54.666931 [ 50 ] {} <Fatal> BaseDaemon: Stack trace: 0x7fac215717bb 0x7fac2155c535 0x498fd1b 0xc6c26e6 0xc6b0a81 0xc6b0f49 0xc6b14b6 0xc6cf123 0xc6c22d6 0x4e4bf15 0x4e4c3de 0x9044c6f 0x8ff4efa 0x9006e22 0x8f08738 0x8b6193a 0x9555b18 0x8b9983d 0x4fa994f 0x4eb991e 0x9d3b883 0x4fb1fb1 0x4fa9faf 0x4eb08f9 0x7fac2155e09b 0x4f13eba
2020.02.12 11:10:54.667353 [ 50 ] {} <Fatal> BaseDaemon: 3. 0x377bb gsignal  in /usr/lib/x86_64-linux-gnu/libc-2.28.so
2020.02.12 11:10:54.667634 [ 50 ] {} <Fatal> BaseDaemon: 4. 0x22535 abort  in /usr/lib/x86_64-linux-gnu/libc-2.28.so
2020.02.12 11:10:54.667945 [ 50 ] {} <Fatal> BaseDaemon: 5. 0x498fd1b ?  in /usr/bin/clickhouse
2020.02.12 11:10:54.668225 [ 50 ] {} <Fatal> BaseDaemon: 6. 0xc6c26e6 ?  in /usr/bin/clickhouse
2020.02.12 11:10:54.668504 [ 50 ] {} <Fatal> BaseDaemon: 7. 0xc6b0a81 ?  in /usr/bin/clickhouse
2020.02.12 11:10:54.668779 [ 50 ] {} <Fatal> BaseDaemon: 8. 0xc6b0f49 ?  in /usr/bin/clickhouse
2020.02.12 11:10:54.669058 [ 50 ] {} <Fatal> BaseDaemon: 9. 0xc6b14b6 __gxx_personality_v0  in /usr/bin/clickhouse
2020.02.12 11:10:54.669347 [ 50 ] {} <Fatal> BaseDaemon: 10. 0xc6cf123 _Unwind_RaiseException  in /usr/bin/clickhouse
2020.02.12 11:10:54.669578 [ 50 ] {} <Fatal> BaseDaemon: 11. 0xc6c22d6 __cxa_throw  in /usr/bin/clickhouse
2020.02.12 11:10:54.669740 [ 50 ] {} <Fatal> BaseDaemon: 12. 0x4e4bf15 Poco::FileImpl::handleLastErrorImpl(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)  in /usr/bin/clickhouse
...

• Which ClickHouse server version to use

ClickHouse 20.1.4.14

Expected behavior

I don't see anything wrong with the FS being read-only and it feels like ClickHouse should be able to start without any problems as it does when there are no inactive parts on disk.

[alexey-milovidov]

Should be easy to fix.

[alexey-milovidov]

Should be easy to fix.