CompressionType #152

zengqingfu1442 · 2017-11-05T15:11:01Z

How add a new compression type such as gzip or openssl for rust-rocksdb?And then how to build it for the TiKV storage engine?Thanks.

siddontang · 2017-11-06T00:37:30Z

Hi @zengqingfu1442

The compression type must be already supported in RocksDB, so you should first support this in RocksDB.

zengqingfu1442 · 2017-11-06T02:43:58Z

Yes, I know. It is supported in RocksDB now and installed it on my machine. But should I rewrite it with rust language?

siddontang · 2017-11-06T03:13:55Z

You just need to add them in https://github.com/pingcap/rust-rocksdb/blob/master/librocksdb_sys/crocksdb/crocksdb/c.h#L949 and https://github.com/pingcap/rust-rocksdb/blob/master/librocksdb_sys/src/lib.rs#L87 if RocksDB supports it.

zengqingfu1442 · 2017-11-06T05:06:32Z

Thanks, but how can I build the TiKV with the new storage engine RocksDB supporting new compression type ?

siddontang · 2017-11-06T05:23:26Z

you can build rocksdb dynamically, use your own rust-rocksdb in https://github.com/pingcap/tikv/blob/master/Cargo.toml#L78 then run make release.

zengqingfu1442 · 2017-11-06T06:14:08Z

Thanks, I see. Should I also put the source code of rocksdb which I have modified in my own rust-rocksdb repository? And put where?

siddontang · 2017-11-06T06:19:39Z

TiKV only needs linking your RocksDB lib dynamically, it doesn't care the source.

BTW, if you want to link your own RocksDB statically, you can update https://github.com/pingcap/rust-rocksdb/blob/master/librocksdb_sys/build.sh#L127

zengqingfu1442 · 2017-11-06T06:38:58Z

Thanks, I see. Now, i have used tidb-ansible to deploy a TiDB cluster(including TiDB server, PD server and TiKV server), should I install my own RocksDB on each node?

siddontang · 2017-11-06T08:34:47Z

If you let TiKV link RocksDB dynamically, you should install on each node.
If you build RocksDB statically in TiKV, you don't need to do it.

zengqingfu1442 · 2017-11-10T05:47:10Z

Thanks, but why don't rocksdb for TiKV use JEMALLOC when compiling?
https://github.com/pingcap/rust-rocksdb/blob/master/librocksdb_sys/build.sh#L138

siddontang · 2017-11-10T06:27:48Z

/cc @BusyJay

BusyJay · 2017-11-10T07:03:17Z

Allocation will be taken over by rustc when building a binary, so link to jemalloc is unnecessary.

zengqingfu1442 · 2017-11-10T09:04:39Z

Thanks, Jay Lee. Can use the librocksdb_sys/build.sh other than build.rs written in librocksdb_sys/Cargo.toml when compiling rust-rocksdb? @BusyJay

BusyJay · 2017-11-10T09:25:11Z

I don't get it. Do you mean you want to use librocksdb_sys/build.sh to build a native rocksdb library? I think it should work but it's recommended to follow the instruction in rocksdb repository instead.

zengqingfu1442 · 2017-11-14T06:54:45Z

Thanks, I now have built the tikv link rocksdb dynamically supporting a new compression type. I used the rocksdb5.7.0 though the code in https://github.com/pingcap/rust-rocksdb/blob/master/librocksdb_sys/crocksdb/c.cc#L40 only support 5.7.3+, but I changed the 3 to 0, and tikv built successfully. What problem will it happen or come across using rocksdb5.7.0?thanks

siddontang · 2017-11-14T07:49:33Z

I remember that 5.7.3+ fixes some bugs, but I think it doesn't matter in your test.

But If you want to use in production, I suggest using a newer one.

zengqingfu1442 · 2017-11-14T08:57:29Z

Thanks, I have integrated a new compression which is supported by rocksdb5.7.0 into tikv-v1.0.1, but now I use tikv-v1.0.0, it come across this error when compilation, it's the file "tikv/src/config.rs"

Compiling tikv v1.0.0 (file:///home/dzeng/tidb/build-tidb/deps/tikv)
error[E0061]: this function takes 4 parameters but 1 parameter was supplied
--> src/config.rs:92:39
|
92 | block_base_opts.set_lru_cache($opt.block_cache_size.0 as usize);
| ^
93 | | block_base_opts.set_cache_index_and_filter_blocks($opt.cache_index_and_filter_blocks);
94 | | block_base_opts.set_pin_l0_filter_and_index_blocks_in_cache(
95 | | $opt.pin_l0_filter_and_index_blocks);
... |
157 | | pub fn build_opt(&self) -> ColumnFamilyOptions {
158 | | let mut cf_opts = build_cf_opt!(self);
| | -----------------^-
| ||____|
| | expected 4 parameters
| in this macro invocation
....

siddontang · 2017-11-15T01:14:10Z

@zengqingfu1442

TiKV doesn't use the latest rust RocksDB, we will update it later. Now you can complement the missing args by yourself.

zengqingfu1442 · 2017-11-15T04:22:56Z

The default compression of tidb is snappy,right? So, if I wanna use another compression type, should I set the parameter related to compression in tidb-ansible/conf/tikv.yml before deploying tidb?

siddontang · 2017-11-15T06:08:59Z

No, the default is lz4 and zstd for different levels.

You must update with your own type before deploying.

zengqingfu1442 · 2017-11-15T06:42:46Z

Thanks, it's in tidb-ansible/conf/tikv.yml to update?

siddontang · 2017-11-15T08:08:10Z

yes, you must also put your tikv-server in the resource/bin

zengqingfu1442 · 2017-11-15T11:18:32Z

If I put the my built tikv-server in tidb-ansible/downloads, will it work? Must I put it under tidb-ansible/resource/bin? Thanks.

siddontang · 2017-11-15T11:20:42Z

tidb-ansible/resource/bin then run rolling update.

zengqingfu1442 · 2017-11-15T11:30:47Z

I tried to put my built tikv-server,tikv-ctl,tikv-fail into tidb-v1.0.1.tar.gz which replaced the original tikv*,and then deploy, at last, I can also get the same tikv-* binary under the deploy directory (just need to copy the share_lib *.so to preper directory).

zengqingfu1442 · 2017-11-15T11:35:44Z

I wiil also put the built tikv binaries to tidb-ansible/resources/bin to replace the original binary. Thanks.

zengqingfu1442 · 2017-11-16T05:12:14Z

What's the meaning of the compression-per-level?How can I use the specific compression type in rocksdb defaultcf?

siddontang · 2017-11-16T08:40:47Z

you must first know rocksdb konws your compression, and then you can set the compression name for the level you want to compress.

zengqingfu1442 · 2017-11-16T09:58:12Z

Yes, I have let rocksdb know it, and tikv-server, tikv-ctl have also linked to the dynamical linking library of the compression.

zengqingfu1442 · 2017-11-20T03:35:38Z

Now I have let rocksdb know the new compression, but tikv doesn't know it.
I have put the new tikv binary under resources/bin and then rolling_update it.When I use tidb-ansible to start the tidb cluster, tikv always can't start up. The tikv_stderr.log says:
invalid configuration file "conf/tikv.toml": invalid value: string "csszlib", expected invalid compression type for key rocksdb.defaultcf.compression-per-level

The new tikv binary has been able to linked to the dynamical linking library of the new compression.
[dzeng@dzeng bin]$ ldd tikv-server
linux-vdso.so.1 => (0x00007ffc89bf0000)
librocksdb.so.5.7 => /usr/local/lib/librocksdb.so.5.7 (0x00007f8960065000)
libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x00007f895fd5c000)
libdl.so.2 => /usr/lib64/libdl.so.2 (0x00007f895fb58000)
librt.so.1 => /usr/lib64/librt.so.1 (0x00007f895f950000)
libpthread.so.0 => /usr/lib64/libpthread.so.0 (0x00007f895f733000)
libgcc_s.so.1 => /usr/lib64/libgcc_s.so.1 (0x00007f895f51d000)
libc.so.6 => /usr/lib64/libc.so.6 (0x00007f895f15a000)
/lib64/ld-linux-x86-64.so.2 (0x000055a81029f000)
libm.so.6 => /usr/lib64/libm.so.6 (0x00007f895ee57000)
libsnappy.so.1 => /usr/lib64/libsnappy.so.1 (0x00007f895ec51000)
libgflags.so.2.1 => /usr/lib64/libgflags.so.2.1 (0x00007f895ea30000)
libz.so.1 => /usr/lib64/libz.so.1 (0x00007f895e819000)
libcssz.so => /usr/lib64/libcssz.so (0x00007f895e571000)
libbz2.so.1 => /usr/lib64/libbz2.so.1 (0x00007f895e361000)
libnuma.so.1 => /usr/lib64/libnuma.so.1 (0x00007f895e154000)
libtbb.so.2 => /usr/lib64/libtbb.so.2 (0x00007f895df1f000)

How can I let tikv know it when useing tidb-ansible to depoly the cluster? Should I also modify some code in tidb-ansible?Thanks.

siddontang · 2017-12-01T00:51:52Z

@huachaohuang

Maybe this is a bug for us. If the user doesn't want to use compression, we can't use it for the snapshot SST too.

zengqingfu1442 · 2017-12-01T05:45:06Z

Thanks,

I change the COMPRESSION_PRIORITY:
tikv/src/util/rocksdb/mod.rs
// Zlib and bzip2 are too slow.
const COMPRESSION_PRIORITY: [DBCompressionType; 3] = [
// DBCompressionType::Lz4,
DBCompressionType::Snappy,
DBCompressionType::Zstd,
DBCompressionType::CSSZlib,
];
Changed the default compression-per-level for rocksdb-defaultcf, rocksdb-writecf and raftrocksdb-defaultcf from [no,no,lz4,lz4,lz4,zstd,zstd] to [no,no,csszlib,csszlib,csszlib,zstd,zstd].
tikv/src/config.rs
compression_per_level: [
DBCompressionType::No,
DBCompressionType::No,
DBCompressionType::CSSZlib,
DBCompressionType::CSSZlib,
DBCompressionType::CSSZlib,
DBCompressionType::Zstd,
DBCompressionType::Zstd,
],

Then I re-built tikv.
Now, I have setup the tidb cluster with my built tikv binary supported csszlib.
The tikv instances don't crash this time.
And now I'm running prepare.sh in tidb-bench/sysbench/ to generate data for test and it appears to work now.

zengqingfu1442 · 2017-12-01T06:22:30Z

The default parameter of rocksdb: max_bytes_for_level_multiplier = 10, then how about the value of max_bytes_for_level_multiplier in rocksdb for tikv?

huachaohuang · 2017-12-01T08:01:04Z

We use the same default as rocksdb for tikv.

zengqingfu1442 · 2017-12-01T11:54:17Z

Can you please tell me how you run the sysbench benchmark?

./prepare.sh Generate data fisrt
./oltp.sh
./select.sh
./insert.sh
Is this the order that you ran sysbench?
Thanks, now I'm running sysbench to evaluate the performance of CSSZlib as well as our CSS card.

huachaohuang · 2017-12-01T13:27:52Z

Sorry, I don't know exactly how that benchmark run, feel free to open an issue there, I think you can get an answer there.
By the way, since the compression problem is fixed now, can we close this issue now?

zengqingfu1442 · 2017-12-20T07:35:14Z

Can you please tell me how to check that the data is really compressed by lz4 or zstd or any other compression library in tidb?

zhangjinpeng87 · 2017-12-20T07:42:08Z

You can check it in rocksdb's LOG file.

zengqingfu1442 · 2017-12-20T08:01:41Z

In the log of rocksdb used in tikv, there are some following messages,
2017/12/13-11:21:38.935983 7f4f220c0e40 RocksDB version: 5.7.0
2017/12/13-11:21:38.936317 7f4f220c0e40 Git sha rocksdb_build_git_sha:9f6165a2eb71fc32f74518a2884fb5811fbea693
2017/12/13-11:21:38.936324 7f4f220c0e40 Compile date Dec 1 2017
2017/12/13-11:21:38.936331 7f4f220c0e40 DB SUMMARY
2017/12/13-11:21:38.936391 7f4f220c0e40 SST files in /mnt/sfx-card-root/tikv3_1/data/db dir, Total Num: 0, files:
...
...
2017/12/13-11:21:38.936573 7f4f220c0e40 Compression algorithms supported:
2017/12/13-11:21:38.936578 7f4f220c0e40 Snappy supported: 1
2017/12/13-11:21:38.936581 7f4f220c0e40 Zlib supported: 1
2017/12/13-11:21:38.936583 7f4f220c0e40 Bzip supported: 1
2017/12/13-11:21:38.936585 7f4f220c0e40 LZ4 supported: 1
2017/12/13-11:21:38.936595 7f4f220c0e40 ZSTD supported: 1
2017/12/13-11:21:38.936599 7f4f220c0e40 Fast CRC32 supported: 0
2017/12/13-11:21:38.936683 7f4f220c0e40 [db/db_impl_open.cc:216] Creating manifest 1
...
...
2017/12/13-11:21:38.938269 7f4f220c0e40 Options.write_buffer_size: 134217728
2017/12/13-11:21:38.938271 7f4f220c0e40 Options.max_write_buffer_number: 5
2017/12/13-11:21:38.938273 7f4f220c0e40 Options.compression[0]: NoCompression
2017/12/13-11:21:38.938275 7f4f220c0e40 Options.compression[1]: NoCompression
2017/12/13-11:21:38.938277 7f4f220c0e40 Options.compression[2]: CSSZlib
2017/12/13-11:21:38.938279 7f4f220c0e40 Options.compression[3]: CSSZlib
2017/12/13-11:21:38.938280 7f4f220c0e40 Options.compression[4]: CSSZlib
2017/12/13-11:21:38.938282 7f4f220c0e40 Options.compression[5]: CSSZlib
2017/12/13-11:21:38.938283 7f4f220c0e40 Options.compression[6]: CSSZlib
2017/12/13-11:21:38.938285 7f4f220c0e40 Options.bottommost_compression: Disabled

So can I make sure that my compression has been integrated into tikv?

zengqingfu1442 · 2017-12-20T08:44:45Z

How to understand the sentence "the tidb server is stateless"?

zengqingfu1442 · 2017-12-21T05:57:37Z

Besides, when the tikv cluster is running, we can see that the tikv-server process is using sfx0 and css0 character device for csszlib compression, as well as libcssz.so, librocksdb.so and so on.
[tcn@sfx-003 ~]$ sudo lsof -c tikv-server | grep sfx0
tikv-serv 8158 tcn 77u CHR 10,58 0t0 12097 /dev/sfx0
tikv-serv 8166 tcn 261u CHR 10,58 0t0 12097 /dev/sfx0
tikv-serv 8169 tcn 137u CHR 10,58 0t0 12097 /dev/sfx0
[tcn@sfx-003 ~]$ sudo lsof -c tikv-server | grep css0
tikv-serv 8158 tcn 76u CHR 245,0 0t0 12203 /dev/css0
tikv-serv 8166 tcn 260u CHR 245,0 0t0 12203 /dev/css0
tikv-serv 8169 tcn 101u CHR 245,0 0t0 12203 /dev/css0
[tcn@sfx-003 ~]$ sudo lsof -c tikv-server | grep rocksdb
tikv-serv 8158 tcn mem REG 8,3 114224722 3687805 /usr/local/lib/librocksdb.so.5.7.0
tikv-serv 8166 tcn mem REG 8,3 114224722 3687805 /usr/local/lib/librocksdb.so.5.7.0
tikv-serv 8169 tcn mem REG 8,3 114224722 3687805 /usr/local/lib/librocksdb.so.5.7.0
[tcn@sfx-003 ~]$ sudo lsof -c tikv-server | grep libcss
tikv-serv 8158 tcn mem REG 8,3 966712 3672421 /usr/lib64/libcssz.so
tikv-serv 8166 tcn mem REG 8,3 966712 3672421 /usr/lib64/libcssz.so
tikv-serv 8169 tcn mem REG 8,3 966712 3672421 /usr/lib64/libcssz.so

siddontang · 2017-12-22T00:14:41Z

@zengqingfu1442

"stateless" means tidb-server keep no persistent data and you can add as many as you want.

using sfx0 and css0 character device for csszlib compression, as well as libcssz.so, librocksdb.so and so on

Sorry, what do you want to say? Can you explain more clearly?

zengqingfu1442 · 2017-12-22T02:27:16Z

I mean that the csszlib compression really works for compressing data, which is handled by the hardware, the character device sfx0 and css0.

siddontang · 2017-12-24T08:05:15Z

Can we close this issue now? @zengqingfu1442

zengqingfu1442 · 2018-01-04T05:16:49Z

Now I can integrate my compression into tikv-v1.0.2, but when I use the same method to integrate my compression into tikv-v1.0.5, it failed. Is there any suggestions about the errors? Thanks.
[dzeng@dzeng tikv-1.0.5]$ make release
Compiling librocksdb_sys v0.1.0 (file:///home/dzeng/tikv-v1.0.5/tikv-1.0.5/rust-rocksdb/librocksdb_sys)
Compiling tikv v1.0.5 (file:///home/dzeng/tikv-v1.0.5/tikv-1.0.5)
Compiling rocksdb v0.3.0 (file:///home/dzeng/tikv-v1.0.5/tikv-1.0.5/rust-rocksdb)
warning: unused import: str
--> src/coprocessor/dag/expr/compare.rs:14:11
|
14 | use std::{str, i64};
| ^^^
|
= note: #[warn(unused_imports)] on by default

error[E0061]: this function takes 0 parameters but 1 parameter was supplied
--> src/util/rocksdb/mod.rs:96:31
|
96 | db_opts.enable_statistics(true);
| ^^^^ expected 0 parameters

error[E0599]: no associated item named CompactionOptimizedDelDropObsolete found for type rocksdb::DBStatisticsTickerType in the current scope
--> src/util/rocksdb/engine_metrics.rs:299:9
|
299 | TickerType::CompactionOptimizedDelDropObsolete => {
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

error[E0599]: no associated item named NumberKeysUpdated found for type rocksdb::DBStatisticsTickerType in the current scope
--> src/util/rocksdb/engine_metrics.rs:317:9
|
317 | TickerType::NumberKeysUpdated => {
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

error[E0061]: this function takes 0 parameters but 1 parameter was supplied
--> src/config.rs:405:36
|
405 | opts.enable_statistics(true);
| ^^^^ expected 0 parameters

error[E0061]: this function takes 0 parameters but 1 parameter was supplied
--> src/config.rs:578:36
|
578 | opts.enable_statistics(true);
| ^^^^ expected 0 parameters

error[E0599]: no method named set_iterate_lower_bound found for type rocksdb::ReadOptions in the current scope
--> src/raftstore/store/engine.rs:235:18
|
235 | opts.set_iterate_lower_bound(key);
| ^^^^^^^^^^^^^^^^^^^^^^^
|
= help: did you mean set_iterate_upper_bound?

error: aborting due to 6 previous errors

error: Could not compile tikv.

To learn more, run the command again with --verbose.
make: *** [release] Error 101

siddontang · 2018-01-04T05:22:09Z

hi @zengqingfu1442

We upgrade RocksDB version now.
Btw, rust-rocksdb uses and statically link the specified RocksDB, you must update your change at https://github.com/pingcap/rust-rocksdb/tree/master/librocksdb_sys/rocksdb directly.

zengqingfu1442 · 2018-01-04T05:36:04Z

I built RocksDB dynamically in TiKV before. How can I build RocksDB statically in TiKV?
So if I wanna integrate the compression into tikv-v1.0.5, I need to upgrade my rocksdb version to 5.8.7+ and use the updated rust-rocksdb?

siddontang · 2018-01-04T06:05:28Z

@zengqingfu1442

We don't support dynamic linking RocksDB now, rust-rocksdb will link RocksDB statically itself.

As I said above, you can add your changes at https://github.com/pingcap/rust-rocksdb/tree/master/librocksdb_sys/rocksdb directly, then rust-rocksdb will compile and link it automatically.

/cc @BusyJay

zengqingfu1442 · 2018-01-04T12:30:35Z

Now I can integrate my compression into your used rocksdb-5.8.7 at https://github.com/pingcap/rust-rocksdb/tree/master/librocksdb_sys/rocksdb, but when I put the rocksdb under rust-rocksdb/librocksdb_sys and then make test, the compilation failed and show the following error message:

= note: "cc" "-Wl,--as-needed" "-Wl,-z,noexecstack" "-m64" "-L" "/opt/rust/lib/rustlib/x86_64-unknown-linux-gnu/lib" "/home/dzeng/tikv-v1.0.5/rust-rocksdb/target/debug/deps/test-0ff9cffe471a7518.test0.rust-cgu.o" "/home/dzeng/tikv-v1.0.5/rust-rocksdb/target/debug/deps/test-0ff9cffe471a7518.test1.rust-cgu.o" "/home/dzeng/tikv-v1.0.5/rust-rocksdb/target/debug/deps/test-0ff9cffe471a7518.test10.rust-cgu.o" "/home/dzeng/tikv-v1.0.5/rust-rocksdb/target/debug/deps/test-0ff9cffe471a7518.test11.rust-cgu.o" "/home/dzeng/tikv-v1.0.5/rust-rocksdb/target/debug/deps/test-0ff9cffe471a7518.test12.rust-cgu.o" "/home/dzeng/tikv-v1.0.5/rust-rocksdb/target/debug/deps/test-0ff9cffe471a7518.test13.rust-cgu.o" "/home/dzeng/tikv-v1.0.5/rust-rocksdb/target/debug/deps/test-0ff9cffe471a7518.test14.rust-cgu.o" "/home/dzeng/tikv-v1.0.5/rust-rocksdb/target/debug/deps/test-0ff9cffe471a7518.test15.rust-cgu.o" "/home/dzeng/tikv-v1.0.5/rust-rocksdb/target/debug/deps/test-0ff9cffe471a7518.test2.rust-cgu.o" "/home/dzeng/tikv-v1.0.5/rust-rocksdb/target/debug/deps/test-0ff9cffe471a7518.test3.rust-cgu.o" "/home/dzeng/tikv-v1.0.5/rust-rocksdb/target/debug/deps/test-0ff9cffe471a7518.test4.rust-cgu.o" "/home/dzeng/tikv-v1.0.5/rust-rocksdb/target/debug/deps/test-0ff9cffe471a7518.test5.rust-cgu.o" "/home/dzeng/tikv-v1.0.5/rust-rocksdb/target/debug/deps/test-0ff9cffe471a7518.test6.rust-cgu.o" "/home/dzeng/tikv-v1.0.5/rust-rocksdb/target/debug/deps/test-0ff9cffe471a7518.test7.rust-cgu.o" "/home/dzeng/tikv-v1.0.5/rust-rocksdb/target/debug/deps/test-0ff9cffe471a7518.test8.rust-cgu.o" "/home/dzeng/tikv-v1.0.5/rust-rocksdb/target/debug/deps/test-0ff9cffe471a7518.test9.rust-cgu.o" "-o" "/home/dzeng/tikv-v1.0.5/rust-rocksdb/target/debug/deps/test-0ff9cffe471a7518" "/home/dzeng/tikv-v1.0.5/rust-rocksdb/target/debug/deps/test-0ff9cffe471a7518.crate.allocator.rust-cgu.o" "-Wl,--gc-sections" "-pie" "-Wl,-z,relro,-z,now" "-nodefaultlibs" "-L" "/home/dzeng/tikv-v1.0.5/rust-rocksdb/target/debug/deps" "-L" "/home/dzeng/tikv-v1.0.5/rust-rocksdb/target/debug/build/librocksdb_sys-4017653e8b1568c4/out/build" "-L" "/home/dzeng/tikv-v1.0.5/rust-rocksdb/target/debug/build/librocksdb_sys-4017653e8b1568c4/out" "-L" "/home/dzeng/tikv-v1.0.5/rust-rocksdb/target/debug/build/bzip2-sys-185c4b847ab6e9c3/out" "-L" "/home/dzeng/tikv-v1.0.5/rust-rocksdb/target/debug/build/libz-sys-3229c174bc0fdbf9/out/lib" "-L" "/home/dzeng/tikv-v1.0.5/rust-rocksdb/target/debug/build/lz4-sys-88d6d70c663268b4/out" "-L" "/home/dzeng/tikv-v1.0.5/rust-rocksdb/target/debug/build/snappy-sys-73a1604664d92f84/out/build" "-L" "/home/dzeng/tikv-v1.0.5/rust-rocksdb/target/debug/build/zstd-sys-bce74bb6931f7809/out" "-L" "/opt/rust/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-Wl,-Bstatic" "/home/dzeng/tikv-v1.0.5/rust-rocksdb/target/debug/deps/libbyteorder-d8e24fb2567c46da.rlib" "/home/dzeng/tikv-v1.0.5/rust-rocksdb/target/debug/deps/libtempdir-1b62bc638e04fbe4.rlib" "/home/dzeng/tikv-v1.0.5/rust-rocksdb/target/debug/deps/libcrc-ec9ec9c4acc8ebc7.rlib" "/home/dzeng/tikv-v1.0.5/rust-rocksdb/target/debug/deps/librand-01e396faece4661e.rlib" "/home/dzeng/tikv-v1.0.5/rust-rocksdb/target/debug/deps/librocksdb-15853619758bf25c.rlib" "/home/dzeng/tikv-v1.0.5/rust-rocksdb/target/debug/deps/liblibrocksdb_sys-f27c48a190b17db2.rlib" "/home/dzeng/tikv-v1.0.5/rust-rocksdb/target/debug/deps/libbzip2_sys-261b0a7fafd34b51.rlib" "/home/dzeng/tikv-v1.0.5/rust-rocksdb/target/debug/deps/liblibc-358a6a97c1fa421b.rlib" "/opt/rust/lib/rustlib/x86_64-unknown-linux-gnu/lib/libtest-6245c8679edb0632.rlib" "/opt/rust/lib/rustlib/x86_64-unknown-linux-gnu/lib/libterm-13a38f6ff5c555a2.rlib" "/opt/rust/lib/rustlib/x86_64-unknown-linux-gnu/lib/libgetopts-07d97ec288432093.rlib" "/opt/rust/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd-fa858cff1c179ad5.rlib" "/opt/rust/lib/rustlib/x86_64-unknown-linux-gnu/lib/libpanic_unwind-f1d6ab1572dda01f.rlib" "/opt/rust/lib/rustlib/x86_64-unknown-linux-gnu/lib/liballoc_jemalloc-5f25e79d42fb3337.rlib" "/opt/rust/lib/rustlib/x86_64-unknown-linux-gnu/lib/libunwind-908ca35c4b5eb851.rlib" "/opt/rust/lib/rustlib/x86_64-unknown-linux-gnu/lib/liballoc_system-46041e6493fca4e6.rlib" "/opt/rust/lib/rustlib/x86_64-unknown-linux-gnu/lib/liblibc-5083eb9d4145f818.rlib" "/opt/rust/lib/rustlib/x86_64-unknown-linux-gnu/lib/liballoc-84783d1968d0025f.rlib" "/opt/rust/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd_unicode-7b76b705776632dd.rlib" "/opt/rust/lib/rustlib/x86_64-unknown-linux-gnu/lib/librand-27ae42d059b7591a.rlib" "/opt/rust/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcore-31df71e4c5d464da.rlib" "/opt/rust/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcompiler_builtins-598fb63ce2fee417.rlib" "-Wl,-Bdynamic" "-l" "stdc++" "-l" "util" "-l" "util" "-l" "dl" "-l" "rt" "-l" "pthread" "-l" "pthread" "-l" "gcc_s" "-l" "c" "-l" "m" "-l" "rt" "-l" "pthread" "-l" "util" "-l" "util"
= note: /home/dzeng/tikv-v1.0.5/rust-rocksdb/target/debug/deps/liblibrocksdb_sys-f27c48a190b17db2.rlib(format.cc.o): In function rocksdb::UncompressBlockContentsForCompressionType(char const*, unsigned long, rocksdb::BlockContents*, unsigned int, rocksdb::Slice const&, rocksdb::CompressionType, rocksdb::ImmutableCFOptions const&)': /home/dzeng/tikv-v1.0.5/rust-rocksdb/librocksdb_sys/rocksdb/table/format.cc:569: undefined reference to rocksdb::CSSZlib_Uncompress(char const, unsigned long, int, unsigned int, rocksdb::Slice const&, int)'
/home/dzeng/tikv-v1.0.5/rust-rocksdb/target/debug/deps/liblibrocksdb_sys-f27c48a190b17db2.rlib(block_based_table_builder.cc.o): In function rocksdb::CompressBlock(rocksdb::Slice const&, rocksdb::CompressionOptions const&, rocksdb::CompressionType*, unsigned int, rocksdb::Slice const&, std::string*)': /home/dzeng/tikv-v1.0.5/rust-rocksdb/librocksdb_sys/rocksdb/table/block_based_table_builder.cc:179: undefined reference to rocksdb::CSSZlib_Compress(rocksdb::CompressionOptions const&, unsigned int, char const*, unsigned long, std::string*, rocksdb::Slice const&)'**
collect2: error: ld returned 1 exit status

error: aborting due to previous error

error: Could not compile rocksdb.

To learn more, run the command again with --verbose.
make: *** [test] Error 101

huachaohuang · 2018-01-04T13:49:43Z

undefined reference to rocksdb::CSSZlib_Compress(rocksdb::CompressionOptions const&, unsigned int, char const*, unsigned long, std::string*, rocksdb::Slice const&)'

Seems like it can't find your custom compression implementation, where do you put the implementation of the above function?

zengqingfu1442 · 2018-01-05T04:55:41Z

I have defined the CSSZlib_Compress and CSSZlib_Uncompress function in util/css.h and implemetation in util/css.cc. And there isn't any compilation error when only compiling the rocksdb-5.8.7.

huachaohuang · 2018-01-05T05:11:19Z

Can you write a test using the CSSZlib_Compress/CSSZlib_Uncompress functions and link with the custom RocksDB library statically to see if everything works fine?

zengqingfu1442 · 2018-01-10T04:33:37Z

Yes, it works fine.

zengqingfu1442 · 2018-01-11T07:52:08Z

I learned that the rust-rocksdb integrate rocksdb into tikv by calling C API in rocksdb. When compiling rust-rocksdb, what's the detailed process of compiling rocksdb in rust-rocksdb? Can you help me explain more clearly?Thanks.

siddontang · 2018-01-12T00:33:15Z

what's the detailed process of compiling rocksdb in rust-rocksdb

/cc @BusyJay

siddontang mentioned this issue Dec 1, 2017

is it suitable to use specified compression for snapshot SST implicitly? tikv/tikv#2536

Open

zengqingfu1442 closed this as completed Dec 4, 2017

zengqingfu1442 reopened this Dec 20, 2017

zengqingfu1442 closed this as completed Dec 24, 2017

zengqingfu1442 reopened this Jan 4, 2018

CompressionType #152

CompressionType #152

Comments

zengqingfu1442 commented Nov 5, 2017

siddontang commented Nov 6, 2017

zengqingfu1442 commented Nov 6, 2017

siddontang commented Nov 6, 2017

zengqingfu1442 commented Nov 6, 2017

siddontang commented Nov 6, 2017

zengqingfu1442 commented Nov 6, 2017

siddontang commented Nov 6, 2017

zengqingfu1442 commented Nov 6, 2017

siddontang commented Nov 6, 2017

zengqingfu1442 commented Nov 10, 2017

siddontang commented Nov 10, 2017

BusyJay commented Nov 10, 2017

zengqingfu1442 commented Nov 10, 2017

BusyJay commented Nov 10, 2017

zengqingfu1442 commented Nov 14, 2017

siddontang commented Nov 14, 2017

zengqingfu1442 commented Nov 14, 2017

siddontang commented Nov 15, 2017

zengqingfu1442 commented Nov 15, 2017

siddontang commented Nov 15, 2017

zengqingfu1442 commented Nov 15, 2017

siddontang commented Nov 15, 2017

zengqingfu1442 commented Nov 15, 2017

siddontang commented Nov 15, 2017

zengqingfu1442 commented Nov 15, 2017

zengqingfu1442 commented Nov 15, 2017

zengqingfu1442 commented Nov 16, 2017

siddontang commented Nov 16, 2017

zengqingfu1442 commented Nov 16, 2017

zengqingfu1442 commented Nov 20, 2017

siddontang commented Dec 1, 2017

zengqingfu1442 commented Dec 1, 2017

zengqingfu1442 commented Dec 1, 2017

huachaohuang commented Dec 1, 2017

zengqingfu1442 commented Dec 1, 2017

huachaohuang commented Dec 1, 2017

zengqingfu1442 commented Dec 20, 2017

zhangjinpeng87 commented Dec 20, 2017

zengqingfu1442 commented Dec 20, 2017

zengqingfu1442 commented Dec 20, 2017

zengqingfu1442 commented Dec 21, 2017

siddontang commented Dec 22, 2017

zengqingfu1442 commented Dec 22, 2017

siddontang commented Dec 24, 2017

zengqingfu1442 commented Jan 4, 2018

siddontang commented Jan 4, 2018

zengqingfu1442 commented Jan 4, 2018 • edited Loading

siddontang commented Jan 4, 2018

zengqingfu1442 commented Jan 4, 2018 • edited Loading

huachaohuang commented Jan 4, 2018

zengqingfu1442 commented Jan 5, 2018 • edited Loading

huachaohuang commented Jan 5, 2018

zengqingfu1442 commented Jan 10, 2018

zengqingfu1442 commented Jan 11, 2018

siddontang commented Jan 12, 2018

zengqingfu1442 commented Jan 4, 2018 •

edited

Loading

zengqingfu1442 commented Jan 4, 2018 •

edited

Loading

zengqingfu1442 commented Jan 5, 2018 •

edited

Loading