Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Compilation errors and tests failed #346

Closed
njanin opened this issue Oct 12, 2014 · 11 comments
Closed

Compilation errors and tests failed #346

njanin opened this issue Oct 12, 2014 · 11 comments

Comments

@njanin
Copy link

njanin commented Oct 12, 2014

Given that -Werror is a compilation flag, the tests in the current tar.gz don't compile flawlessly.

When issueing "make all", you get "unused variable" errors or worse, something like this:

In file included from tools/db_stress.cc:35:0:
tools/db_stress.cc:194:40: error: integer overflow in expression [-Werror=overflow]
DEFINE_int64(cache_size, 2 * KB * KB * KB,
@njanin
Copy link
Author

njanin commented Oct 12, 2014

I should have said that this happened on the Github master. Also several tests crash.
For instance:
==== Test DBTest.Empty
db_test: db/memtable.cc:55: rocksdb::MemTable::MemTable(const rocksdb::InternalKeyComparator&, const rocksdb::Options&): Assertion `!should_flush_' failed.
Received signal 6 (Aborted)
#0 [0x40022424] ?? ??:0
#1 /lib/i386-linux-gnu/libc.so.6(gsignal+0x47) [0x401f6577] ?? ??:0
#2 /lib/i386-linux-gnu/libc.so.6(abort+0x143) [0x401f99a3] ?? ??:0
#3 /lib/i386-linux-gnu/libc.so.6(+0x276c7) [0x401ef6c7] ?? ??:0
#4 /lib/i386-linux-gnu/libc.so.6(+0x27777) [0x401ef777] ?? ??:0
#5 ./db_test() [0x814f929] rocksdb::MemTable::MemTable(rocksdb::InternalKeyComparator const&, rocksdb::Options const&) /home/nicolas/rocksdb-rocksdb-3.5.1/db/memtable.cc:55 (discriminator 1)
#6 ./db_test() [0x8106548] rocksdb::ColumnFamilyData::CreateNewMemtable() /home/nicolas/rocksdb-rocksdb-3.5.1/db/column_family.cc:324 (discriminator 1)
#7 ./db_test() [0x81638c0] rocksdb::ColumnFamilyData::SetLogNumber(unsigned long long) /home/nicolas/rocksdb-rocksdb-3.5.1/./db/column_family.h:165 (discriminator 1)
#8 ./db_test() [0x816945e] rocksdb::VersionSet::Recover(std::vector<rocksdb::ColumnFamilyDescriptor, std::allocatorrocksdb::ColumnFamilyDescriptor > const&, bool) /home/nicolas/rocksdb-rocksdb-3.5.1/db/version_set.cc:2137
#9 ./db_test() [0x8131f2b] rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor, std::allocatorrocksdb::ColumnFamilyDescriptor > const&, bool, bool) /home/nicolas/rocksdb-rocksdb-3.5.1/db/db_impl.cc:1180
#10 ./db_test() [0x8133503] rocksdb::DB::Open(rocksdb::DBOptions const&, std::string const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocatorrocksdb::ColumnFamilyDescriptor > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocatorrocksdb::ColumnFamilyHandle* >, rocksdb::DB__) /home/nicolas/rocksdb-rocksdb-3.5.1/db/db_impl.cc:4874
#11 ./db_test() [0x8134524] rocksdb::DB::Open(rocksdb::Options const&, std::string const&, rocksdb::DB__) /home/nicolas/rocksdb-rocksdb-3.5.1/db/db_impl.cc:4811
#12 ./db_test() [0x80f0b33] rocksdb::DBTest::TryReopen(rocksdb::Options) /home/nicolas/rocksdb-rocksdb-3.5.1/db/db_test.cc:664
#13 ./db_test() [0x80f0deb] Tester /home/nicolas/rocksdb-rocksdb-3.5.1/./util/testharness.h:56
#14 ./db_test() [0x8093260] rocksdb::DBTest::ChangeOptions(int) /home/nicolas/rocksdb-rocksdb-3.5.1/db/db_test.cc:429
#15 ./db_test() [0x80f8581] ~_Test_Empty /home/nicolas/rocksdb-rocksdb-3.5.1/db/db_test.cc:1120
#16 ./db_test() [0x82133c8] rocksdb::test::RunAllTests() /home/nicolas/rocksdb-rocksdb-3.5.1/util/testharness.cc:60
#17 ./db_test() [0x804f73b] main /home/nicolas/rocksdb-rocksdb-3.5.1/db/db_test.cc:7750

@njanin
Copy link
Author

njanin commented Oct 12, 2014

Dammit, same errors on release 3.51

@njanin njanin changed the title Compilation errors Compilation errors and tests failed Oct 12, 2014
@igorcanadi
Copy link
Collaborator

@njanin what is your environment?

@igorcanadi
Copy link
Collaborator

Can you also post the full compilation output?

@njanin
Copy link
Author

njanin commented Oct 14, 2014

Ubuntu 14.04 64 bits, gcc 4.8 I believe. I don't have the output here but I'm surprised you don't get these errors.

@igorcanadi
Copy link
Collaborator

We don't get any errors on Mac 64bit or CentOS 6.4 64bit with gcc 4.8 and clang 6.0. Interesting.

@igorcanadi
Copy link
Collaborator

Can you try RocksDB 3.8?

@pihomeserver
Copy link

Same here on a Raspberry Pi

@siying
Copy link
Contributor

siying commented Apr 6, 2015

@njanin @pihomeserver Is it still an issue?

@pihomeserver
Copy link

@siying : no i now use influxdb 0.9 wich is pure Go and without rocksdb ....

@igorcanadi
Copy link
Collaborator

Closing this since it's no longer an issue.

suuyaoo pushed a commit to suuyaoo/rocksdb that referenced this issue Feb 29, 2024
@tabokie @cbi42
Cherry-pick compaction enhancements and add range check function (fac…
fe76937 
…ebook#346)

* Always allow L0->L1 trivial move during manual compaction (facebook#11375)

Summary:
during manual compaction (CompactRange()), L0->L1 trivial move is disabled when only L0 overlaps with compacting key range (introduced in facebook#7368 to enforce kForce* contract). This can cause large memory usage due to compaction readahead when number of L0 files is large. This PR allows L0->L1 trivial move in this case, and will do a L1 -> L1 intra-level compaction when needed (`bottommost_level_compaction` is kForce*). In brief, consider a DB with only L0 file, and user calls CompactRange(kForce, nullptr, nullptr),
- before this PR, RocksDB does a L0 -> L1 compaction (disallow trivial move),
- after this PR, RocksDB does a L0 -> L1 compaction (allow trivial move), and a L1 -> L1 compaction.
Users can use kForceOptimized to avoid this extra L1->L1 compaction overhead when L0s are overlapping and cannot be trivial moved.

This PR also fixed a bug (see previous discussion in facebook#11041) where `final_output_level` of a manual compaction can be miscalculated when `level_compaction_dynamic_level_bytes=true`. This bug could cause incorrect level being moved when CompactRangeOptions::change_level is specified.

Pull Request resolved: facebook#11375

Test Plan: - Added new unit tests to test that L0 -> L1 compaction allows trivial move and L1 -> L1 compaction is done when needed.

Reviewed By: ajkr

Differential Revision: D44943518

Pulled By: cbi42

fbshipit-source-id: e9fb770d17b163c18a623e1d1bd6b81159192708
Signed-off-by: tabokie <xy.tao@outlook.com>

* `CompactRange()` always compacts to bottommost level for leveled compaction (facebook#11468)

Summary:
currently for leveled compaction, the max output level of a call to `CompactRange()` is pre-computed before compacting each level. This max output level is the max level whose key range overlaps with the manual compaction key range. However, during manual compaction, files in the max output level may be compacted down further by some background compaction. When this background compaction is a trivial move, there is a race condition and the manual compaction may not be able to compact all keys in the specified key range. This PR updates `CompactRange()` to always compact to the bottommost level to make this race condition more unlikely (it can still happen, see more in comment here: https://github.com/cbi42/rocksdb/blob/796f58f42ad1bdbf49e5fcf480763f11583b790e/db/db_impl/db_impl_compaction_flush.cc#L1180C29-L1184).

This PR also changes the behavior of CompactRange() when `bottommost_level_compaction=kIfHaveCompactionFilter` (the default option). The old behavior is that, if a compaction filter is provided, CompactRange() always does an intra-level compaction at the final output level for all files in the manual compaction key range. The only exception when `first_overlapped_level = 0` and `max_overlapped_level = 0`. It’s awkward to maintain the same behavior after this PR since we do not compute max_overlapped_level anymore. So the new behavior is similar to kForceOptimized: always does intra-level compaction at the bottommost level, but not including new files generated during this manual compaction.

Several unit tests are updated to work with this new manual compaction behavior.

Pull Request resolved: facebook#11468

Test Plan: Add new unit tests `DBCompactionTest.ManualCompactionCompactAllKeysInRange*`

Reviewed By: ajkr

Differential Revision: D46079619

Pulled By: cbi42

fbshipit-source-id: 19d844ba4ec8dc1a0b8af5d2f36ff15820c6e76f
Signed-off-by: tabokie <xy.tao@outlook.com>

* add check in range interface

Signed-off-by: tabokie <xy.tao@outlook.com>

---------

Signed-off-by: tabokie <xy.tao@outlook.com>
Co-authored-by: Changyu Bi <changyubi@meta.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@igorcanadi @pihomeserver @siying @njanin