Export Import sst files #5495

vpallipadi · 2019-06-21T09:24:08Z

Refresh of the earlier change here - #5135

This is a review request for code change needed for - #3469
"Add support for taking snapshot of a column family and creating column family from a given CF snapshot"

We have an implementation for this that we have been testing internally. We have two new APIs that together provide this functionality.

(1) ExportColumnFamily() - This API is modelled after CreateCheckpoint() as below.
// Exports all live SST files of a specified Column Family onto export_dir,
// returning SST files information in metadata.
// - SST files will be created as hard links when the directory specified
// is in the same partition as the db directory, copied otherwise.
// - export_dir should not already exist and will be created by this API.
// - Always triggers a flush.
virtual Status ExportColumnFamily(ColumnFamilyHandle* handle,
const std::string& export_dir,
ExportImportFilesMetaData** metadata);

Internally, the API will DisableFileDeletions(), GetColumnFamilyMetaData(), Parse through
metadata, creating links/copies of all the sst files, EnableFileDeletions() and complete the call by
returning the list of file metadata.

(2) CreateColumnFamilyWithImport() - This API is modeled after IngestExternalFile(), but invoked only during a CF creation as below.
// CreateColumnFamilyWithImport() will create a new column family with
// column_family_name and import external SST files specified in metadata into
// this column family.
// (1) External SST files can be created using SstFileWriter.
// (2) External SST files can be exported from a particular column family in
// an existing DB.
// Option in import_options specifies whether the external files are copied or
// moved (default is copy). When option specifies copy, managing files at
// external_file_path is caller's responsibility. When option specifies a
// move, the call ensures that the specified files at external_file_path are
// deleted on successful return and files are not modified on any error
// return.
// On error return, column family handle returned will be nullptr.
// ColumnFamily will be present on successful return and will not be present
// on error return. ColumnFamily may be present on any crash during this call.
virtual Status CreateColumnFamilyWithImport(
const ColumnFamilyOptions& options, const std::string& column_family_name,
const ImportColumnFamilyOptions& import_options,
const ExportImportFilesMetaData& metadata,
ColumnFamilyHandle** handle);

Internally, this API creates a new CF, parses all the sst files and adds it to the specified column family, at the same level and with same sequence number as in the metadata. Also performs safety checks with respect to overlaps between the sst files being imported.

If incoming sequence number is higher than current local sequence number, local sequence
number is updated to reflect this.

Note, as the sst files is are being moved across Column Families, Column Family name in sst file
will no longer match the actual column family on destination DB. The API does not modify Column
Family name or id in the sst files being imported.

Introduce a new API ExportColumnFamily to export all the live sst files of a particular column family and metadata information to be used to import these sst files. This is part 1 of the change for #3469 " Add support for taking snapshot of a CF and creating a CF from a given CF snapshot" Partially Solves #3469

This change adds a new API to create a column family and importing sst files into the column family. The files imported can come from ExportColumnFamily API or can be created with SST file writer. This is part 2 of the change for #3469 " Add support for taking snapshot of a CF and creating a CF from a given CF snapshot" Partially Solves #3469

Add new unit tests for ExportColumnFamily() and CreateColumnFamilyWithImport() APIs. This is part 3 of the change for #3469 " Add support for taking snapshot of a CF and creating a CF from a given CF snapshot" Partially Solves #3469

facebook-github-bot

@siying has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

facebook-github-bot

@siying has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

siying · 2019-07-02T21:08:37Z

@vpallipadi sanitizer fails:

==2786007==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 56 byte(s) in 1 object(s) allocated from:
     #0 checkpoint_test_bin+0x337480             operator new(unsigned long)
     #1 internal_repo_rocksdb/repo/utilities/checkpoint/checkpoint_impl.cc:416 rocksdb::CheckpointImpl::ExportColumnFamily(rocksdb::ColumnFamilyHandle*, std::__cxx11::basic_string<...> const&, rocksdb::ExportImportFilesMetaData**)
    #2 0x2af1c5 in rocksdb::CheckpointTest_ExportColumnFamilyWithLinks_Test::TestBody() internal_repo_rocksdb/repo/utilities/checkpoint/checkpoint_test.cc:374:27:52:50:6:14:6:31:6:46:27:52:55
    #16 internal_repo_rocksdb/repo/utilities/checkpoint/checkpoint_test.cc:811 main
    #17 libc.so.6+0x211a5                        __libc_start_main

Direct leak of 56 byte(s) in 1 object(s) allocated from:
     #0 checkpoint_test_bin+0x337480             operator new(unsigned long)
     #1 internal_repo_rocksdb/repo/utilities/checkpoint/checkpoint_impl.cc:416 rocksdb::CheckpointImpl::ExportColumnFamily(rocksdb::ColumnFamilyHandle*, std::__cxx11::basic_string<...> const&, rocksdb::ExportImportFilesMetaData**)
    #2 0x2adcc9 in rocksdb::CheckpointTest_ExportColumnFamilyWithLinks_Test::TestBody() internal_repo_rocksdb/repo/utilities/checkpoint/checkpoint_test.cc:352:27:52:50:6:14:6:31:6:46:27:52:55
    #16 internal_repo_rocksdb/repo/utilities/checkpoint/checkpoint_test.cc:811 main
    #17 libc.so.6+0x211a5                        __libc_start_main

Direct leak of 16 byte(s) in 1 object(s) allocated from:
     #0 checkpoint_test_bin+0x337480             operator new(unsigned long)
     #1 internal_repo_rocksdb/repo/utilities/checkpoint/checkpoint_impl.cc:33 rocksdb::Checkpoint::Create(rocksdb::DB*, rocksdb::Checkpoint**)
    #2 0x2acbad in rocksdb::CheckpointTest_ExportColumnFamilyWithLinks_Test::TestBody() internal_repo_rocksdb/repo/utilities/checkpoint/checkpoint_test.cc:337:27:52:50:6:14:6:31:6:46:27:52:55
    #16 internal_repo_rocksdb/repo/utilities/checkpoint/checkpoint_test.cc:811 main
    #17 libc.so.6+0x211a5                        __libc_start_main

Direct leak of 16 byte(s) in 1 object(s) allocated from:
     #0 checkpoint_test_bin+0x337480             operator new(unsigned long)
     #1 internal_repo_rocksdb/repo/utilities/checkpoint/checkpoint_impl.cc:33 rocksdb::Checkpoint::Create(rocksdb::DB*, rocksdb::Checkpoint**)
    #2 0x2aee02 in rocksdb::CheckpointTest_ExportColumnFamilyWithLinks_Test::TestBody() internal_repo_rocksdb/repo/utilities/checkpoint/checkpoint_test.cc:370:27:52:50:6:14:6:31:6:46:27:52:55
    #16 internal_repo_rocksdb/repo/utilities/checkpoint/checkpoint_test.cc:811 main
    #17 libc.so.6+0x211a5                        __libc_start_main

Direct leak of 16 byte(s) in 1 object(s) allocated from:
     #0 checkpoint_test_bin+0x337480             operator new(unsigned long)
     #1 internal_repo_rocksdb/repo/utilities/checkpoint/checkpoint_impl.cc:33 rocksdb::Checkpoint::Create(rocksdb::DB*, rocksdb::Checkpoint**)
    #2 0x2b1ff7 in rocksdb::CheckpointTest_ExportColumnFamilyNegativeTest_Test::TestBody() internal_repo_rocksdb/repo/utilities/checkpoint/checkpoint_test.cc:393:27:52:50:6:14:6:31:6:46:27:52:55
    #16 internal_repo_rocksdb/repo/utilities/checkpoint/checkpoint_test.cc:811 main
    #17 libc.so.6+0x211a5                        __libc_start_main

Indirect leak of 448 byte(s) in 1 object(s) allocated from:
     #0 checkpoint_test_bin+0x337480             operator new(unsigned long)
     #6 internal_repo_rocksdb/repo/utilities/checkpoint/checkpoint_impl.cc:429 rocksdb::CheckpointImpl::ExportColumnFamily(rocksdb::ColumnFamilyHandle*, std::__cxx11::basic_string<...> const&, rocksdb::ExportImportFilesMetaData**)
    #7 0x2adcc9 in rocksdb::CheckpointTest_ExportColumnFamilyWithLinks_Test::TestBody() internal_repo_rocksdb/repo/utilities/checkpoint/checkpoint_test.cc:352:27:52:50:6:14:6:31:6:46:27:52:55
    #21 internal_repo_rocksdb/repo/utilities/checkpoint/checkpoint_test.cc:811 main
    #22 libc.so.6+0x211a5                        __libc_start_main

Indirect leak of 224 byte(s) in 1 object(s) allocated from:
     #0 checkpoint_test_bin+0x337480             operator new(unsigned long)
     #6 internal_repo_rocksdb/repo/utilities/checkpoint/checkpoint_impl.cc:429 rocksdb::CheckpointImpl::ExportColumnFamily(rocksdb::ColumnFamilyHandle*, std::__cxx11::basic_string<...> const&, rocksdb::ExportImportFilesMetaData**)
    #7 0x2af1c5 in rocksdb::CheckpointTest_ExportColumnFamilyWithLinks_Test::TestBody() internal_repo_rocksdb/repo/utilities/checkpoint/checkpoint_test.cc:374:27:52:50:6:14:6:31:6:46:27:52:55
    #21 internal_repo_rocksdb/repo/utilities/checkpoint/checkpoint_test.cc:811 main
    #22 libc.so.6+0x211a5                        __libc_start_main

Indirect leak of 78 byte(s) in 2 object(s) allocated from:
     #0 checkpoint_test_bin+0x337480             operator new(unsigned long)
     #8 internal_repo_rocksdb/repo/include/rocksdb/metadata.h:55 rocksdb::SstFileMetaData::SstFileMetaData(rocksdb::SstFileMetaData const&)
     #9 internal_repo_rocksdb/repo/include/rocksdb/metadata.h:106 rocksdb::LiveFileMetaData::LiveFileMetaData(rocksdb::LiveFileMetaData const&)
    #14 internal_repo_rocksdb/repo/utilities/checkpoint/checkpoint_impl.cc:429 rocksdb::CheckpointImpl::ExportColumnFamily(rocksdb::ColumnFamilyHandle*, std::__cxx11::basic_string<...> const&, rocksdb::ExportImportFilesMetaData**)
    #15 0x2adcc9 in rocksdb::CheckpointTest_ExportColumnFamilyWithLinks_Test::TestBody() internal_repo_rocksdb/repo/utilities/checkpoint/checkpoint_test.cc:352:27:52:50:6:14:6:31:6:46:27:52:55
    #29 internal_repo_rocksdb/repo/utilities/checkpoint/checkpoint_test.cc:811 main
    #30 libc.so.6+0x211a5                        __libc_start_main

Indirect leak of 39 byte(s) in 1 object(s) allocated from:
     #0 checkpoint_test_bin+0x337480             operator new(unsigned long)
     #8 internal_repo_rocksdb/repo/include/rocksdb/metadata.h:55 rocksdb::SstFileMetaData::SstFileMetaData(rocksdb::SstFileMetaData const&)
     #9 internal_repo_rocksdb/repo/include/rocksdb/metadata.h:106 rocksdb::LiveFileMetaData::LiveFileMetaData(rocksdb::LiveFileMetaData const&)
    #14 internal_repo_rocksdb/repo/utilities/checkpoint/checkpoint_impl.cc:429 rocksdb::CheckpointImpl::ExportColumnFamily(rocksdb::ColumnFamilyHandle*, std::__cxx11::basic_string<...> const&, rocksdb::ExportImportFilesMetaData**)
    #15 0x2af1c5 in rocksdb::CheckpointTest_ExportColumnFamilyWithLinks_Test::TestBody() internal_repo_rocksdb/repo/utilities/checkpoint/checkpoint_test.cc:374:27:52:50:6:14:6:31:6:46:27:52:55
    #29 internal_repo_rocksdb/repo/utilities/checkpoint/checkpoint_test.cc:811 main
    #30 libc.so.6+0x211a5                        __libc_start_main

Indirect leak of 34 byte(s) in 1 object(s) allocated from:
     #0 checkpoint_test_bin+0x337480             operator new(unsigned long)
     #8 internal_repo_rocksdb/repo/utilities/checkpoint/checkpoint_impl.cc:417 rocksdb::CheckpointImpl::ExportColumnFamily(rocksdb::ColumnFamilyHandle*, std::__cxx11::basic_string<...> const&, rocksdb::ExportImportFilesMetaData**)
    #9 0x2af1c5 in rocksdb::CheckpointTest_ExportColumnFamilyWithLinks_Test::TestBody() internal_repo_rocksdb/repo/utilities/checkpoint/checkpoint_test.cc:374:27:52:50:6:14:6:31:6:46:27:52:55
    #23 internal_repo_rocksdb/repo/utilities/checkpoint/checkpoint_test.cc:811 main
    #24 libc.so.6+0x211a5                        __libc_start_main

Indirect leak of 31 byte(s) in 1 object(s) allocated from:
     #0 checkpoint_test_bin+0x337480             operator new(unsigned long)
     #8 internal_repo_rocksdb/repo/utilities/checkpoint/checkpoint_impl.cc:417 rocksdb::CheckpointImpl::ExportColumnFamily(rocksdb::ColumnFamilyHandle*, std::__cxx11::basic_string<...> const&, rocksdb::ExportImportFilesMetaData**)
    #9 0x2adcc9 in rocksdb::CheckpointTest_ExportColumnFamilyWithLinks_Test::TestBody() internal_repo_rocksdb/repo/utilities/checkpoint/checkpoint_test.cc:352:27:52:50:6:14:6:31:6:46:27:52:55
    #23 internal_repo_rocksdb/repo/utilities/checkpoint/checkpoint_test.cc:811 main
    #24 libc.so.6+0x211a5                        __libc_start_main

Can you take a look?

vpallipadi · 2019-07-02T21:13:22Z

Will check and get back.

…

On Tue, Jul 2, 2019 at 2:09 PM Siying Dong ***@***.***> wrote: @vpallipadi <https://github.com/vpallipadi> sanitizer fails: ==2786007==ERROR: LeakSanitizer: detected memory leaks Direct leak of 56 byte(s) in 1 object(s) allocated from: #0 checkpoint_test_bin+0x337480 operator new(unsigned long) #1 internal_repo_rocksdb/repo/utilities/checkpoint/checkpoint_impl.cc:416 rocksdb::CheckpointImpl::ExportColumnFamily(rocksdb::ColumnFamilyHandle*, std::__cxx11::basic_string<...> const&, rocksdb::ExportImportFilesMetaData**) #2 0x2af1c5 in rocksdb::CheckpointTest_ExportColumnFamilyWithLinks_Test::TestBody() internal_repo_rocksdb/repo/utilities/checkpoint/checkpoint_test.cc:374:27:52:50:6:14:6:31:6:46:27:52:55 #16 internal_repo_rocksdb/repo/utilities/checkpoint/checkpoint_test.cc:811 main #17 libc.so.6+0x211a5 __libc_start_main Direct leak of 56 byte(s) in 1 object(s) allocated from: #0 checkpoint_test_bin+0x337480 operator new(unsigned long) #1 internal_repo_rocksdb/repo/utilities/checkpoint/checkpoint_impl.cc:416 rocksdb::CheckpointImpl::ExportColumnFamily(rocksdb::ColumnFamilyHandle*, std::__cxx11::basic_string<...> const&, rocksdb::ExportImportFilesMetaData**) #2 0x2adcc9 in rocksdb::CheckpointTest_ExportColumnFamilyWithLinks_Test::TestBody() internal_repo_rocksdb/repo/utilities/checkpoint/checkpoint_test.cc:352:27:52:50:6:14:6:31:6:46:27:52:55 #16 internal_repo_rocksdb/repo/utilities/checkpoint/checkpoint_test.cc:811 main #17 libc.so.6+0x211a5 __libc_start_main Direct leak of 16 byte(s) in 1 object(s) allocated from: #0 checkpoint_test_bin+0x337480 operator new(unsigned long) #1 internal_repo_rocksdb/repo/utilities/checkpoint/checkpoint_impl.cc:33 rocksdb::Checkpoint::Create(rocksdb::DB*, rocksdb::Checkpoint**) #2 0x2acbad in rocksdb::CheckpointTest_ExportColumnFamilyWithLinks_Test::TestBody() internal_repo_rocksdb/repo/utilities/checkpoint/checkpoint_test.cc:337:27:52:50:6:14:6:31:6:46:27:52:55 #16 internal_repo_rocksdb/repo/utilities/checkpoint/checkpoint_test.cc:811 main #17 libc.so.6+0x211a5 __libc_start_main Direct leak of 16 byte(s) in 1 object(s) allocated from: #0 checkpoint_test_bin+0x337480 operator new(unsigned long) #1 internal_repo_rocksdb/repo/utilities/checkpoint/checkpoint_impl.cc:33 rocksdb::Checkpoint::Create(rocksdb::DB*, rocksdb::Checkpoint**) #2 0x2aee02 in rocksdb::CheckpointTest_ExportColumnFamilyWithLinks_Test::TestBody() internal_repo_rocksdb/repo/utilities/checkpoint/checkpoint_test.cc:370:27:52:50:6:14:6:31:6:46:27:52:55 #16 internal_repo_rocksdb/repo/utilities/checkpoint/checkpoint_test.cc:811 main #17 libc.so.6+0x211a5 __libc_start_main Direct leak of 16 byte(s) in 1 object(s) allocated from: #0 checkpoint_test_bin+0x337480 operator new(unsigned long) #1 internal_repo_rocksdb/repo/utilities/checkpoint/checkpoint_impl.cc:33 rocksdb::Checkpoint::Create(rocksdb::DB*, rocksdb::Checkpoint**) #2 0x2b1ff7 in rocksdb::CheckpointTest_ExportColumnFamilyNegativeTest_Test::TestBody() internal_repo_rocksdb/repo/utilities/checkpoint/checkpoint_test.cc:393:27:52:50:6:14:6:31:6:46:27:52:55 #16 internal_repo_rocksdb/repo/utilities/checkpoint/checkpoint_test.cc:811 main #17 libc.so.6+0x211a5 __libc_start_main Indirect leak of 448 byte(s) in 1 object(s) allocated from: #0 checkpoint_test_bin+0x337480 operator new(unsigned long) #6 internal_repo_rocksdb/repo/utilities/checkpoint/checkpoint_impl.cc:429 rocksdb::CheckpointImpl::ExportColumnFamily(rocksdb::ColumnFamilyHandle*, std::__cxx11::basic_string<...> const&, rocksdb::ExportImportFilesMetaData**) #7 0x2adcc9 in rocksdb::CheckpointTest_ExportColumnFamilyWithLinks_Test::TestBody() internal_repo_rocksdb/repo/utilities/checkpoint/checkpoint_test.cc:352:27:52:50:6:14:6:31:6:46:27:52:55 #21 internal_repo_rocksdb/repo/utilities/checkpoint/checkpoint_test.cc:811 main #22 libc.so.6+0x211a5 __libc_start_main Indirect leak of 224 byte(s) in 1 object(s) allocated from: #0 checkpoint_test_bin+0x337480 operator new(unsigned long) #6 internal_repo_rocksdb/repo/utilities/checkpoint/checkpoint_impl.cc:429 rocksdb::CheckpointImpl::ExportColumnFamily(rocksdb::ColumnFamilyHandle*, std::__cxx11::basic_string<...> const&, rocksdb::ExportImportFilesMetaData**) #7 0x2af1c5 in rocksdb::CheckpointTest_ExportColumnFamilyWithLinks_Test::TestBody() internal_repo_rocksdb/repo/utilities/checkpoint/checkpoint_test.cc:374:27:52:50:6:14:6:31:6:46:27:52:55 #21 internal_repo_rocksdb/repo/utilities/checkpoint/checkpoint_test.cc:811 main #22 libc.so.6+0x211a5 __libc_start_main Indirect leak of 78 byte(s) in 2 object(s) allocated from: #0 checkpoint_test_bin+0x337480 operator new(unsigned long) #8 internal_repo_rocksdb/repo/include/rocksdb/metadata.h:55 rocksdb::SstFileMetaData::SstFileMetaData(rocksdb::SstFileMetaData const&) #9 internal_repo_rocksdb/repo/include/rocksdb/metadata.h:106 rocksdb::LiveFileMetaData::LiveFileMetaData(rocksdb::LiveFileMetaData const&) #14 internal_repo_rocksdb/repo/utilities/checkpoint/checkpoint_impl.cc:429 rocksdb::CheckpointImpl::ExportColumnFamily(rocksdb::ColumnFamilyHandle*, std::__cxx11::basic_string<...> const&, rocksdb::ExportImportFilesMetaData**) #15 0x2adcc9 in rocksdb::CheckpointTest_ExportColumnFamilyWithLinks_Test::TestBody() internal_repo_rocksdb/repo/utilities/checkpoint/checkpoint_test.cc:352:27:52:50:6:14:6:31:6:46:27:52:55 #29 internal_repo_rocksdb/repo/utilities/checkpoint/checkpoint_test.cc:811 main #30 libc.so.6+0x211a5 __libc_start_main Indirect leak of 39 byte(s) in 1 object(s) allocated from: #0 checkpoint_test_bin+0x337480 operator new(unsigned long) #8 internal_repo_rocksdb/repo/include/rocksdb/metadata.h:55 rocksdb::SstFileMetaData::SstFileMetaData(rocksdb::SstFileMetaData const&) #9 internal_repo_rocksdb/repo/include/rocksdb/metadata.h:106 rocksdb::LiveFileMetaData::LiveFileMetaData(rocksdb::LiveFileMetaData const&) #14 internal_repo_rocksdb/repo/utilities/checkpoint/checkpoint_impl.cc:429 rocksdb::CheckpointImpl::ExportColumnFamily(rocksdb::ColumnFamilyHandle*, std::__cxx11::basic_string<...> const&, rocksdb::ExportImportFilesMetaData**) #15 0x2af1c5 in rocksdb::CheckpointTest_ExportColumnFamilyWithLinks_Test::TestBody() internal_repo_rocksdb/repo/utilities/checkpoint/checkpoint_test.cc:374:27:52:50:6:14:6:31:6:46:27:52:55 #29 internal_repo_rocksdb/repo/utilities/checkpoint/checkpoint_test.cc:811 main #30 libc.so.6+0x211a5 __libc_start_main Indirect leak of 34 byte(s) in 1 object(s) allocated from: #0 checkpoint_test_bin+0x337480 operator new(unsigned long) #8 internal_repo_rocksdb/repo/utilities/checkpoint/checkpoint_impl.cc:417 rocksdb::CheckpointImpl::ExportColumnFamily(rocksdb::ColumnFamilyHandle*, std::__cxx11::basic_string<...> const&, rocksdb::ExportImportFilesMetaData**) #9 0x2af1c5 in rocksdb::CheckpointTest_ExportColumnFamilyWithLinks_Test::TestBody() internal_repo_rocksdb/repo/utilities/checkpoint/checkpoint_test.cc:374:27:52:50:6:14:6:31:6:46:27:52:55 #23 internal_repo_rocksdb/repo/utilities/checkpoint/checkpoint_test.cc:811 main #24 libc.so.6+0x211a5 __libc_start_main Indirect leak of 31 byte(s) in 1 object(s) allocated from: #0 checkpoint_test_bin+0x337480 operator new(unsigned long) #8 internal_repo_rocksdb/repo/utilities/checkpoint/checkpoint_impl.cc:417 rocksdb::CheckpointImpl::ExportColumnFamily(rocksdb::ColumnFamilyHandle*, std::__cxx11::basic_string<...> const&, rocksdb::ExportImportFilesMetaData**) #9 0x2adcc9 in rocksdb::CheckpointTest_ExportColumnFamilyWithLinks_Test::TestBody() internal_repo_rocksdb/repo/utilities/checkpoint/checkpoint_test.cc:352:27:52:50:6:14:6:31:6:46:27:52:55 #23 internal_repo_rocksdb/repo/utilities/checkpoint/checkpoint_test.cc:811 main #24 libc.so.6+0x211a5 __libc_start_main Can you take a look? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#5495?email_source=notifications&email_token=AIYNMRNTL5QKZJNPKVNZIL3P5O7Y5A5CNFSM4H2PLAHKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZCRU6I#issuecomment-507845241>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AIYNMRKDNRXDPGHI6MPNUNLP5O7Y5ANCNFSM4H2PLAHA> .

vpallipadi · 2019-07-04T06:21:36Z

This looks to be a bug in the unit test I added. How do I verify the fix? valgrind_check and asan_check did not detect the original error seen below.

…

On Tue, Jul 2, 2019 at 2:09 PM Siying Dong ***@***.***> wrote: @vpallipadi <https://github.com/vpallipadi> sanitizer fails: ==2786007==ERROR: LeakSanitizer: detected memory leaks Direct leak of 56 byte(s) in 1 object(s) allocated from: #0 checkpoint_test_bin+0x337480 operator new(unsigned long) #1 internal_repo_rocksdb/repo/utilities/checkpoint/checkpoint_impl.cc:416 rocksdb::CheckpointImpl::ExportColumnFamily(rocksdb::ColumnFamilyHandle*, std::__cxx11::basic_string<...> const&, rocksdb::ExportImportFilesMetaData**) #2 0x2af1c5 in rocksdb::CheckpointTest_ExportColumnFamilyWithLinks_Test::TestBody() internal_repo_rocksdb/repo/utilities/checkpoint/checkpoint_test.cc:374:27:52:50:6:14:6:31:6:46:27:52:55 #16 internal_repo_rocksdb/repo/utilities/checkpoint/checkpoint_test.cc:811 main #17 libc.so.6+0x211a5 __libc_start_main Direct leak of 56 byte(s) in 1 object(s) allocated from: #0 checkpoint_test_bin+0x337480 operator new(unsigned long) #1 internal_repo_rocksdb/repo/utilities/checkpoint/checkpoint_impl.cc:416 rocksdb::CheckpointImpl::ExportColumnFamily(rocksdb::ColumnFamilyHandle*, std::__cxx11::basic_string<...> const&, rocksdb::ExportImportFilesMetaData**) #2 0x2adcc9 in rocksdb::CheckpointTest_ExportColumnFamilyWithLinks_Test::TestBody() internal_repo_rocksdb/repo/utilities/checkpoint/checkpoint_test.cc:352:27:52:50:6:14:6:31:6:46:27:52:55 #16 internal_repo_rocksdb/repo/utilities/checkpoint/checkpoint_test.cc:811 main #17 libc.so.6+0x211a5 __libc_start_main Direct leak of 16 byte(s) in 1 object(s) allocated from: #0 checkpoint_test_bin+0x337480 operator new(unsigned long) #1 internal_repo_rocksdb/repo/utilities/checkpoint/checkpoint_impl.cc:33 rocksdb::Checkpoint::Create(rocksdb::DB*, rocksdb::Checkpoint**) #2 0x2acbad in rocksdb::CheckpointTest_ExportColumnFamilyWithLinks_Test::TestBody() internal_repo_rocksdb/repo/utilities/checkpoint/checkpoint_test.cc:337:27:52:50:6:14:6:31:6:46:27:52:55 #16 internal_repo_rocksdb/repo/utilities/checkpoint/checkpoint_test.cc:811 main #17 libc.so.6+0x211a5 __libc_start_main Direct leak of 16 byte(s) in 1 object(s) allocated from: #0 checkpoint_test_bin+0x337480 operator new(unsigned long) #1 internal_repo_rocksdb/repo/utilities/checkpoint/checkpoint_impl.cc:33 rocksdb::Checkpoint::Create(rocksdb::DB*, rocksdb::Checkpoint**) #2 0x2aee02 in rocksdb::CheckpointTest_ExportColumnFamilyWithLinks_Test::TestBody() internal_repo_rocksdb/repo/utilities/checkpoint/checkpoint_test.cc:370:27:52:50:6:14:6:31:6:46:27:52:55 #16 internal_repo_rocksdb/repo/utilities/checkpoint/checkpoint_test.cc:811 main #17 libc.so.6+0x211a5 __libc_start_main Direct leak of 16 byte(s) in 1 object(s) allocated from: #0 checkpoint_test_bin+0x337480 operator new(unsigned long) #1 internal_repo_rocksdb/repo/utilities/checkpoint/checkpoint_impl.cc:33 rocksdb::Checkpoint::Create(rocksdb::DB*, rocksdb::Checkpoint**) #2 0x2b1ff7 in rocksdb::CheckpointTest_ExportColumnFamilyNegativeTest_Test::TestBody() internal_repo_rocksdb/repo/utilities/checkpoint/checkpoint_test.cc:393:27:52:50:6:14:6:31:6:46:27:52:55 #16 internal_repo_rocksdb/repo/utilities/checkpoint/checkpoint_test.cc:811 main #17 libc.so.6+0x211a5 __libc_start_main Indirect leak of 448 byte(s) in 1 object(s) allocated from: #0 checkpoint_test_bin+0x337480 operator new(unsigned long) #6 internal_repo_rocksdb/repo/utilities/checkpoint/checkpoint_impl.cc:429 rocksdb::CheckpointImpl::ExportColumnFamily(rocksdb::ColumnFamilyHandle*, std::__cxx11::basic_string<...> const&, rocksdb::ExportImportFilesMetaData**) #7 0x2adcc9 in rocksdb::CheckpointTest_ExportColumnFamilyWithLinks_Test::TestBody() internal_repo_rocksdb/repo/utilities/checkpoint/checkpoint_test.cc:352:27:52:50:6:14:6:31:6:46:27:52:55 #21 internal_repo_rocksdb/repo/utilities/checkpoint/checkpoint_test.cc:811 main #22 libc.so.6+0x211a5 __libc_start_main Indirect leak of 224 byte(s) in 1 object(s) allocated from: #0 checkpoint_test_bin+0x337480 operator new(unsigned long) #6 internal_repo_rocksdb/repo/utilities/checkpoint/checkpoint_impl.cc:429 rocksdb::CheckpointImpl::ExportColumnFamily(rocksdb::ColumnFamilyHandle*, std::__cxx11::basic_string<...> const&, rocksdb::ExportImportFilesMetaData**) #7 0x2af1c5 in rocksdb::CheckpointTest_ExportColumnFamilyWithLinks_Test::TestBody() internal_repo_rocksdb/repo/utilities/checkpoint/checkpoint_test.cc:374:27:52:50:6:14:6:31:6:46:27:52:55 #21 internal_repo_rocksdb/repo/utilities/checkpoint/checkpoint_test.cc:811 main #22 libc.so.6+0x211a5 __libc_start_main Indirect leak of 78 byte(s) in 2 object(s) allocated from: #0 checkpoint_test_bin+0x337480 operator new(unsigned long) #8 internal_repo_rocksdb/repo/include/rocksdb/metadata.h:55 rocksdb::SstFileMetaData::SstFileMetaData(rocksdb::SstFileMetaData const&) #9 internal_repo_rocksdb/repo/include/rocksdb/metadata.h:106 rocksdb::LiveFileMetaData::LiveFileMetaData(rocksdb::LiveFileMetaData const&) #14 internal_repo_rocksdb/repo/utilities/checkpoint/checkpoint_impl.cc:429 rocksdb::CheckpointImpl::ExportColumnFamily(rocksdb::ColumnFamilyHandle*, std::__cxx11::basic_string<...> const&, rocksdb::ExportImportFilesMetaData**) #15 0x2adcc9 in rocksdb::CheckpointTest_ExportColumnFamilyWithLinks_Test::TestBody() internal_repo_rocksdb/repo/utilities/checkpoint/checkpoint_test.cc:352:27:52:50:6:14:6:31:6:46:27:52:55 #29 internal_repo_rocksdb/repo/utilities/checkpoint/checkpoint_test.cc:811 main #30 libc.so.6+0x211a5 __libc_start_main Indirect leak of 39 byte(s) in 1 object(s) allocated from: #0 checkpoint_test_bin+0x337480 operator new(unsigned long) #8 internal_repo_rocksdb/repo/include/rocksdb/metadata.h:55 rocksdb::SstFileMetaData::SstFileMetaData(rocksdb::SstFileMetaData const&) #9 internal_repo_rocksdb/repo/include/rocksdb/metadata.h:106 rocksdb::LiveFileMetaData::LiveFileMetaData(rocksdb::LiveFileMetaData const&) #14 internal_repo_rocksdb/repo/utilities/checkpoint/checkpoint_impl.cc:429 rocksdb::CheckpointImpl::ExportColumnFamily(rocksdb::ColumnFamilyHandle*, std::__cxx11::basic_string<...> const&, rocksdb::ExportImportFilesMetaData**) #15 0x2af1c5 in rocksdb::CheckpointTest_ExportColumnFamilyWithLinks_Test::TestBody() internal_repo_rocksdb/repo/utilities/checkpoint/checkpoint_test.cc:374:27:52:50:6:14:6:31:6:46:27:52:55 #29 internal_repo_rocksdb/repo/utilities/checkpoint/checkpoint_test.cc:811 main #30 libc.so.6+0x211a5 __libc_start_main Indirect leak of 34 byte(s) in 1 object(s) allocated from: #0 checkpoint_test_bin+0x337480 operator new(unsigned long) #8 internal_repo_rocksdb/repo/utilities/checkpoint/checkpoint_impl.cc:417 rocksdb::CheckpointImpl::ExportColumnFamily(rocksdb::ColumnFamilyHandle*, std::__cxx11::basic_string<...> const&, rocksdb::ExportImportFilesMetaData**) #9 0x2af1c5 in rocksdb::CheckpointTest_ExportColumnFamilyWithLinks_Test::TestBody() internal_repo_rocksdb/repo/utilities/checkpoint/checkpoint_test.cc:374:27:52:50:6:14:6:31:6:46:27:52:55 #23 internal_repo_rocksdb/repo/utilities/checkpoint/checkpoint_test.cc:811 main #24 libc.so.6+0x211a5 __libc_start_main Indirect leak of 31 byte(s) in 1 object(s) allocated from: #0 checkpoint_test_bin+0x337480 operator new(unsigned long) #8 internal_repo_rocksdb/repo/utilities/checkpoint/checkpoint_impl.cc:417 rocksdb::CheckpointImpl::ExportColumnFamily(rocksdb::ColumnFamilyHandle*, std::__cxx11::basic_string<...> const&, rocksdb::ExportImportFilesMetaData**) #9 0x2adcc9 in rocksdb::CheckpointTest_ExportColumnFamilyWithLinks_Test::TestBody() internal_repo_rocksdb/repo/utilities/checkpoint/checkpoint_test.cc:352:27:52:50:6:14:6:31:6:46:27:52:55 #23 internal_repo_rocksdb/repo/utilities/checkpoint/checkpoint_test.cc:811 main #24 libc.so.6+0x211a5 __libc_start_main Can you take a look? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#5495?email_source=notifications&email_token=AIYNMRNTL5QKZJNPKVNZIL3P5O7Y5A5CNFSM4H2PLAHKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZCRU6I#issuecomment-507845241>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AIYNMRKDNRXDPGHI6MPNUNLP5O7Y5ANCNFSM4H2PLAHA> .

siying · 2019-07-05T18:00:18Z

@vpallipadi how about running asan with clang? Does it catch the issue?

facebook-github-bot

@siying has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

facebook-github-bot · 2019-07-08T06:13:05Z

@vpallipadi has updated the pull request. Re-import the pull request

vpallipadi · 2019-07-08T06:18:01Z

@siying I was not able to run with clang due to some library dependency issues on my build machine. I have sent in an incremental change that fixes leaks in unit test based on review. Should help the failure seen here. Can you check please.

…

On Fri, Jul 5, 2019 at 11:01 AM Siying Dong ***@***.***> wrote: @vpallipadi <https://github.com/vpallipadi> how about running asan with clang? Does it catch the issue? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#5495?email_source=notifications&email_token=AIYNMRIUDQOVJ4SKAW3YDELP56D6FA5CNFSM4H2PLAHKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZKBXWQ#issuecomment-508828634>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AIYNMROE2ADS754HNDM5QY3P56D6FANCNFSM4H2PLAHA> .

facebook-github-bot

@siying has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

siying · 2019-07-09T00:13:40Z

@vpallipadi still some failures:

=================================================================
==2787156==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 16 byte(s) in 1 object(s) allocated from:
     #0 checkpoint_test_bin+0x337620             operator new(unsigned long)
     #1 internal_repo_rocksdb/repo/utilities/checkpoint/checkpoint_impl.cc:33 rocksdb::Checkpoint::Create(rocksdb::DB*, rocksdb::Checkpoint**)
    #2 0x2aeeaf in rocksdb::CheckpointTest_ExportColumnFamilyWithLinks_Test::TestBody() internal_repo_rocksdb/repo/utilities/checkpoint/checkpoint_test.cc:377:27:52:50:6:14:6:31:6:46:27:52:55
    #16 internal_repo_rocksdb/repo/utilities/checkpoint/checkpoint_test.cc:816 main
    #17 libc.so.6+0x211a5                        __libc_start_main

Direct leak of 16 byte(s) in 1 object(s) allocated from:
     #0 checkpoint_test_bin+0x337620             operator new(unsigned long)
     #1 internal_repo_rocksdb/repo/utilities/checkpoint/checkpoint_impl.cc:33 rocksdb::Checkpoint::Create(rocksdb::DB*, rocksdb::Checkpoint**)
    #2 0x2b20cc in rocksdb::CheckpointTest_ExportColumnFamilyNegativeTest_Test::TestBody() internal_repo_rocksdb/repo/utilities/checkpoint/checkpoint_test.cc:399:27:52:50:6:14:6:31:6:46:27:52:55
    #16 internal_repo_rocksdb/repo/utilities/checkpoint/checkpoint_test.cc:816 main
    #17 libc.so.6+0x211a5                        __libc_start_main

Direct leak of 16 byte(s) in 1 object(s) allocated from:
     #0 checkpoint_test_bin+0x337620             operator new(unsigned long)
     #1 internal_repo_rocksdb/repo/utilities/checkpoint/checkpoint_impl.cc:33 rocksdb::Checkpoint::Create(rocksdb::DB*, rocksdb::Checkpoint**)
    #2 0x2acb7f in rocksdb::CheckpointTest_ExportColumnFamilyWithLinks_Test::TestBody() internal_repo_rocksdb/repo/utilities/checkpoint/checkpoint_test.cc:343:27:52:50:6:14:6:31:6:46:27:52:55
    #16 internal_repo_rocksdb/repo/utilities/checkpoint/checkpoint_test.cc:816 main
    #17 libc.so.6+0x211a5                        __libc_start_main

SUMMARY: AddressSanitizer: 48 byte(s) leaked in 3 allocation(s).

facebook-github-bot · 2019-07-11T18:52:20Z

@vpallipadi has updated the pull request. Re-import the pull request

vpallipadi · 2019-07-11T21:39:25Z

New pull request with additional fix that should handle this.

…

On Mon, Jul 8, 2019 at 5:14 PM Siying Dong ***@***.***> wrote: @vpallipadi <https://github.com/vpallipadi> still some failures: ================================================================= ==2787156==ERROR: LeakSanitizer: detected memory leaks Direct leak of 16 byte(s) in 1 object(s) allocated from: #0 checkpoint_test_bin+0x337620 operator new(unsigned long) #1 internal_repo_rocksdb/repo/utilities/checkpoint/checkpoint_impl.cc:33 rocksdb::Checkpoint::Create(rocksdb::DB*, rocksdb::Checkpoint**) #2 0x2aeeaf in rocksdb::CheckpointTest_ExportColumnFamilyWithLinks_Test::TestBody() internal_repo_rocksdb/repo/utilities/checkpoint/checkpoint_test.cc:377:27:52:50:6:14:6:31:6:46:27:52:55 #16 internal_repo_rocksdb/repo/utilities/checkpoint/checkpoint_test.cc:816 main #17 libc.so.6+0x211a5 __libc_start_main Direct leak of 16 byte(s) in 1 object(s) allocated from: #0 checkpoint_test_bin+0x337620 operator new(unsigned long) #1 internal_repo_rocksdb/repo/utilities/checkpoint/checkpoint_impl.cc:33 rocksdb::Checkpoint::Create(rocksdb::DB*, rocksdb::Checkpoint**) #2 0x2b20cc in rocksdb::CheckpointTest_ExportColumnFamilyNegativeTest_Test::TestBody() internal_repo_rocksdb/repo/utilities/checkpoint/checkpoint_test.cc:399:27:52:50:6:14:6:31:6:46:27:52:55 #16 internal_repo_rocksdb/repo/utilities/checkpoint/checkpoint_test.cc:816 main #17 libc.so.6+0x211a5 __libc_start_main Direct leak of 16 byte(s) in 1 object(s) allocated from: #0 checkpoint_test_bin+0x337620 operator new(unsigned long) #1 internal_repo_rocksdb/repo/utilities/checkpoint/checkpoint_impl.cc:33 rocksdb::Checkpoint::Create(rocksdb::DB*, rocksdb::Checkpoint**) #2 0x2acb7f in rocksdb::CheckpointTest_ExportColumnFamilyWithLinks_Test::TestBody() internal_repo_rocksdb/repo/utilities/checkpoint/checkpoint_test.cc:343:27:52:50:6:14:6:31:6:46:27:52:55 #16 internal_repo_rocksdb/repo/utilities/checkpoint/checkpoint_test.cc:816 main #17 libc.so.6+0x211a5 __libc_start_main SUMMARY: AddressSanitizer: 48 byte(s) leaked in 3 allocation(s). — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#5495?email_source=notifications&email_token=AIYNMRMXU3K4SFYLIO5LOELP6PJ6LA5CNFSM4H2PLAHKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZOWSLY#issuecomment-509438255>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AIYNMRLETU3HSUHOQDDPDWDP6PJ6LANCNFSM4H2PLAHA> .

facebook-github-bot

@siying has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

siying · 2019-07-17T19:05:24Z

I already started the merging process, but I realized that HISTORY.md is not updated for this new feature. Can you open a new pull request to update HISTORY.md?

facebook-github-bot · 2019-07-17T20:41:22Z

This pull request has been merged in 22ce462.

vpallipadi · 2019-07-18T04:44:20Z

Sorry. Did not know about HISTORY. Pull request here - #5587 Thanks, Venki

…

On Wed, Jul 17, 2019 at 12:06 PM Siying Dong ***@***.***> wrote: I already started the merging process, but I realized that HISTORY.md is not updated for this new feature. Can you open a new pull request to update HISTORY.md? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#5495?email_source=notifications&email_token=AIYNMRMKT3HM3UVTF3VGYWDP75USTA5CNFSM4H2PLAHKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD2GIZ6Q#issuecomment-512527610>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AIYNMRPNZZ4AL5WJCBFTCSTP75USTANCNFSM4H2PLAHA> .

Summary: Refresh of the earlier change here - facebook#5135 This is a review request for code change needed for - facebook#3469 "Add support for taking snapshot of a column family and creating column family from a given CF snapshot" We have an implementation for this that we have been testing internally. We have two new APIs that together provide this functionality. (1) ExportColumnFamily() - This API is modelled after CreateCheckpoint() as below. // Exports all live SST files of a specified Column Family onto export_dir, // returning SST files information in metadata. // - SST files will be created as hard links when the directory specified // is in the same partition as the db directory, copied otherwise. // - export_dir should not already exist and will be created by this API. // - Always triggers a flush. virtual Status ExportColumnFamily(ColumnFamilyHandle* handle, const std::string& export_dir, ExportImportFilesMetaData** metadata); Internally, the API will DisableFileDeletions(), GetColumnFamilyMetaData(), Parse through metadata, creating links/copies of all the sst files, EnableFileDeletions() and complete the call by returning the list of file metadata. (2) CreateColumnFamilyWithImport() - This API is modeled after IngestExternalFile(), but invoked only during a CF creation as below. // CreateColumnFamilyWithImport() will create a new column family with // column_family_name and import external SST files specified in metadata into // this column family. // (1) External SST files can be created using SstFileWriter. // (2) External SST files can be exported from a particular column family in // an existing DB. // Option in import_options specifies whether the external files are copied or // moved (default is copy). When option specifies copy, managing files at // external_file_path is caller's responsibility. When option specifies a // move, the call ensures that the specified files at external_file_path are // deleted on successful return and files are not modified on any error // return. // On error return, column family handle returned will be nullptr. // ColumnFamily will be present on successful return and will not be present // on error return. ColumnFamily may be present on any crash during this call. virtual Status CreateColumnFamilyWithImport( const ColumnFamilyOptions& options, const std::string& column_family_name, const ImportColumnFamilyOptions& import_options, const ExportImportFilesMetaData& metadata, ColumnFamilyHandle** handle); Internally, this API creates a new CF, parses all the sst files and adds it to the specified column family, at the same level and with same sequence number as in the metadata. Also performs safety checks with respect to overlaps between the sst files being imported. If incoming sequence number is higher than current local sequence number, local sequence number is updated to reflect this. Note, as the sst files is are being moved across Column Families, Column Family name in sst file will no longer match the actual column family on destination DB. The API does not modify Column Family name or id in the sst files being imported. Pull Request resolved: facebook#5495 Differential Revision: D16018881 fbshipit-source-id: 9ae2251025d5916d35a9fc4ea4d6707f6be16ff9

cscetbon · 2023-03-15T03:07:58Z

@siying any news on this ? I'm looking for a similar feature

LIBA-S · 2024-03-25T08:30:00Z

@vpallipadi @siying hi, for cross-machine import, does the ExportImportFilesMetaData data structure need to define its own serialization/deserialization functions?

Venki Pallipadi added 3 commits June 21, 2019 01:56

facebook-github-bot added the CLA Signed label Jun 21, 2019

facebook-github-bot reviewed Jun 26, 2019

View reviewed changes

facebook-github-bot reviewed Jul 2, 2019

View reviewed changes

facebook-github-bot reviewed Jul 5, 2019

View reviewed changes

mem leak fix in ExportColumnFamily() unit tests

d18a645

facebook-github-bot reviewed Jul 8, 2019

View reviewed changes

mem leak fix for ExportColumnFamily tests in checkpoint_test

e74c0e9

facebook-github-bot reviewed Jul 15, 2019

View reviewed changes

facebook-github-bot closed this in 22ce462 Jul 17, 2019

facebook-github-bot added the Merged label Jul 17, 2019

This was referenced Sep 6, 2019

Export Import sst files #5135

Closed

Export Import sst files #3822

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Export Import sst files #5495

Export Import sst files #5495

vpallipadi commented Jun 21, 2019

facebook-github-bot left a comment

facebook-github-bot left a comment

siying commented Jul 2, 2019

vpallipadi commented Jul 2, 2019 via email

vpallipadi commented Jul 4, 2019 via email

siying commented Jul 5, 2019

facebook-github-bot left a comment

facebook-github-bot commented Jul 8, 2019

vpallipadi commented Jul 8, 2019 via email

facebook-github-bot left a comment

siying commented Jul 9, 2019

facebook-github-bot commented Jul 11, 2019

vpallipadi commented Jul 11, 2019 via email

facebook-github-bot left a comment

siying commented Jul 17, 2019

facebook-github-bot commented Jul 17, 2019

vpallipadi commented Jul 18, 2019 via email

cscetbon commented Mar 15, 2023

LIBA-S commented Mar 25, 2024

Export Import sst files #5495

Export Import sst files #5495

Conversation

vpallipadi commented Jun 21, 2019

facebook-github-bot left a comment

Choose a reason for hiding this comment

facebook-github-bot left a comment

Choose a reason for hiding this comment

siying commented Jul 2, 2019

vpallipadi commented Jul 2, 2019 via email

vpallipadi commented Jul 4, 2019 via email

siying commented Jul 5, 2019

facebook-github-bot left a comment

Choose a reason for hiding this comment

facebook-github-bot commented Jul 8, 2019

vpallipadi commented Jul 8, 2019 via email

facebook-github-bot left a comment

Choose a reason for hiding this comment

siying commented Jul 9, 2019

facebook-github-bot commented Jul 11, 2019

vpallipadi commented Jul 11, 2019 via email

facebook-github-bot left a comment

Choose a reason for hiding this comment

siying commented Jul 17, 2019

facebook-github-bot commented Jul 17, 2019

vpallipadi commented Jul 18, 2019 via email

cscetbon commented Mar 15, 2023

LIBA-S commented Mar 25, 2024