test #339
Closed
test #339
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
disable unittest of zookeeper use gtest for testing add code coverage report add test cases for DeltaMerge::(StorageDeltaMerge / DeltaMergeStore / Segment / DiskValueSpace / Chunk)
* add test cases for PageStorage * split PageStorage gc stage into small helper functions * add test cases for PageStorage gc concurrency * add stress test and dump utils of PageStorage * Fix bug: 1. ensure PageFile with meta && data, in case gc drop file but be killed before drop dir 2. turn PageStorage::Config::sync_on_write = true by default 3. avoid PageStorage::gc run by multi-threads 4. print PageFile's path if checksum is not correct * throw exception for must_exist==false and errno != ENOENT
…97) * Page storage bug fix (#87) * add test cases for PageStorage * split PageStorage gc stage into small helper functions * add test cases for PageStorage gc concurrency * add stress test and dump utils of PageStorage * Fix bug: 1. ensure PageFile with meta && data, in case gc drop file but be killed before drop dir 2. turn PageStorage::Config::sync_on_write = true by default 3. avoid PageStorage::gc run by multi-threads 4. print PageFile's path if checksum is not correct * throw exception for must_exist==false and errno != ENOENT * add RefPage && ref counting * 1. move PageCacheMap into isolated file 2. rename PageCacheMap -> PageEntryMap * accept non-exist Page reference/del while being used by WriteBatch or Gc * adjust gc on RefPages * add more test on gc RefPages * add memory usage comparing between vec/hashmap/treemap * fix tests code format * Bug fix: Add RefPage to non exist Page * Add unit test for PageStorage * fix bug * stress test for benchmark * fix bug of PageEntryMap updating RefPage * Keep RefPage entry ref-count if it has been gc and move to another place * MVCC on PageMap, All previse tests passed. TODO: refactor on read_mutex on PageStorage * PageStorage gc: only delete file that are not used by any version * fix bug under multi-threads; remove unused code && format codes * 1. support snapshot read 2. standalone VersionSet/MultiVersionCountable template * 1. Apply merge delta if no read ref 2. Apply generate new delta if has read ref 3. Delta merged when snapshot released 4. Add simple restore test 5. Use template and constexpr instead of copying codes in PageEntryMap * add test for PageStorage snapshot read * use typed test * Bug fix: Add RefPage to non exist Page * fix bug * Delta merged when view is finished * 1. VersionDeltaSet: add snapshots for snapshot linked-list, so that gc can collect valid PageFiles 2. fix some bugs * do compact on delta -> base * rebase after do compact on delta * fix bugs * avoid duplicted codes * 1. new find API 2. all tests passed * 1. remove legacy API 2. remove duplicated codes * reformat * apply inplace if there are no readers * fix bug of applying edits * fix bug of gc apply * fix bug of merging PageEntryMap * fix bug of invalid ref page * fix bug: 1. listAllLiveFiles 2. double compaction on same version * tmp * fixbug * split PageEntryMapView to single cpp file * minor fix * 1. Avoid visit same version multiple time in PageEntryMapDeltaVersionSet::listAllLiveFiles 2. use template to reduce duplicated code for gcApply * todo mark * fix bug: RefPage -> RefPage record may become invalid record when PageStorage GC * remove unused function * simple fix on unittest * fix bug: PageEntryMapView::isRefId * fix bug: PageEntryMapDeltaBuilder::applyPut * fix * fix bug: PageEntryView::validNormalPageIds filter out tombstone of PageEntry * iterator over PageEntryMap * remove unused tests * Refactor on PageEntryMapDeltaVersionSet * Reduce lock range when PageStorage Snapshot is release * Add metrics for PageStorage MVCC * rename some classes * reformat. * use exception instead * adress comment * adress comment * fix compiler error under gcc * fix compiler error under gcc
* Segment support range scan * Bug fix: ColumnVector insertRangeFrom out of range * Only read tag column when necessary * print more info on sidLowerBound exception * Refactor DeltaMergeBlockInputStream * Optimize DMVersionFilterBlockInputStream, try unroll loops. * Support multi range read * bug fix * Optimize synchronization mechanism between read/write thread, decrease latency. * Fix testes * Address comments * fix typo
* ci run unittests defined using gtest * fix bug in CMakeLists * 1. disable zookeeper test cases 2. use template struct SchemaBuilder instead of template class 3. more portable run-gtest.sh * use EXCLUDE_FROM_ALL in add_subdirectory instead of add_executable * new container of tics just for gtest * disable tics-gtest in unused command * address comment * address comment * add continue_on_error option in run-gtest.sh * ci run unittests defined using gtest * fix bug in CMakeLists * 1. disable zookeeper test cases 2. use template struct SchemaBuilder instead of template class 3. more portable run-gtest.sh * use EXCLUDE_FROM_ALL in add_subdirectory instead of add_executable * new container of tics just for gtest * disable tics-gtest in unused command * address comment * address comment * add continue_on_error option in run-gtest.sh
* Atomic read/write on DeltaMergeStore level * Bug fix * bugfix 2 * bug fix * fix compile error * remove useless code * address comments * fix compile error
* cast DataType while reading from PageStorage * add isLossyCast function * alter for StroageDeltaMerge, WIP * add Alter for StorageDeltaMerge * add Alter for StorageDeltaMerge * add TableInfo in StroageDeltaMerge * rename table for StorageDeltaMerge * fix bug: the TableInfo from TiDB * add comments for DeltaMerge flush && cache * more faster(?) way to cast mismatch datatype * support cast for numeric type null/not null * support for other data type just change null / not null * isLossyCast -> isSupportedDataTypeCast * isSupportedDataTypeCast add decimal detect * rename function * fix compile errors in gtests * small fix * fix broken tests * support new column with non-zero default value * remove unused code * fix compile error in CI * fix bug in table rename * small fix * minor fix * refine cast function in chunk * update DeltaMergeStore's segments within lock * [WIP]Add test cases for default value ddl. Still has bugs * fix bugs after rebasing to latest master * add some TODO marks * fix compile error in gtests && remove unused comments * fix broken gtests * flush cached chunks in delta instead of doing DeltaMerge when ddl-changes apply * use TypeIndex instead of typeid_cast * clean up data after tests * address comment
* Read ranges indicated by mvcc_query_info * Optimization: Don't do handle range filter inside Segment
* Idempotent del for PageStorage * Fix the won't decrease of Normal Page in PageStorage * Add more test case for checking the ref-count of NormalPage * add metrics: PSMVCCNumSnapshots
…#266) * Support merge delta in background. * Avoid slow down query after delete range. * Add a new sql feature manage table tb delete range 100, to test delete range.
…rge delta and split.
* fix typo Signed-off-by: leiysky <leiysky@outlook.com> * Add test cases * resolve conflict * disable WriteLargeBlock test * make temp vector a stack variable
* Throw exception with region status; Use reverseGetColumnInfo * Use IManageableStorage instead of StorageMergeTree * Fix args of mock_tidb_table && ensure engine is DeltaMerge * For learner read from TiDB/TiSpark, we set num_streams by mvcc_query_info.concurrent * fix compile error * address comment; add log for level trace
* refactor default_value * Apply suggestions from code review * enable alter default value test * add default_value to make sure read can fill value with it * fix issue with reading Float default value * add test for read Float default value * resolve suggestions * add abs * fix issue with DateTime default value * add test for DateTime default value * modify test case * handle Decimal64 * modify expect values * format * fix issue with Decimal default_value * modify test case * resolve suggestions * fix bug: column in StorageDeltaMerge but not in TiDB::TableInfo::columns
* do handle range filter in * add status support for DM * compact continue insert * bug fix, and temporary split big insert block * enable split logical * add store restore log * fix lock in PageEntriesVersionSetWithDelta::listAllLiveFiles * disable posix_fadvise in PageStorage * Update default DM settings * optimize DMVersionFilterBlockInputStream * optimize DMVersionFilterBlockInputStream v2 * added some metrics * add dm_insert_max_rows setting * Fix compilation error of TiDB.h * address comment * fix broken unit tests * fix unit tests of DeltaTree * Bug fix: DiskValueSpace::createAppendTask remove data pages by mistake * remove debug code
* Add test case for migrate ref-page to ref-page * check if ref-page is valid * only remove PageFile's data * Add MOVE_NORMAL_PAGE for only move normal page * enhance utils_get_valid_pages
…315) * Add concurrency for wait index of regions while dong learner read * Add some status about StoragePool
* Remove obsoleted data in DeltaMerge when region is removed * Add test case for region removed * Update GC safe point from PD to remove obsolated data * applySnapshot: TMT don't required to lock data * only update gc safe point in background merge delta * Add dm_safe_point_update_interval_seconds * fix bug: acquire lock && ensure not null pointer * fix uninitialized variables * Add cache in PDClientHelper for gc safe point
…316) * add configuration to disable background flush * Add log to trace disable_bg_flush * Add log for debug * enable background KVStore persist * judge whether if a region is dirty with result * add time count for snapshot * enable background flush task in tests * Apply suggestions from code review Co-Authored-By: JaySon <jayson.hjs@gmail.com> * add disable_bg_flush to example configs
* Add callback for gc * Add helper for list capacity of PageFile * Add external page scanner for get external pages when gc begin * fix compile error of tests
* Check region version while doing learner read * remove unused code * use global_context for getting PDClient in doing DM bg tasks
* Ensure external_pages_remover will be called each time PageStorage gc run * If callback of external_pages is empty, return empty live_normal_pages
* make cast when applying DDL * handle nullable * enhance handleing nullable
* simple parser * multiple conditions with operator and by default * add more debugging info * [FLASH-477] Make chunks in stable not overlap * Add option dm_enable_rough_set_filter * A quick impl for rough set filter ast parser * Fix bug: compare Date/DateTime to String in rough set filter * Turn off verbose logging info in restoreSegment * Add event tracking * Add more debugging info for ast * address comments * fix compile error under Mac OSX * use dynamic_cast instead of static_cast
* Import DMFile * Integration of DMFile * optimize read: if delta is empty, then we could try clean read * fix clean read optimize * fix compile error in tests Signed-off-by: JaySon-Huang <jayson.hjs@gmail.com> * Make pk in different chunks not overlap Signed-off-by: JaySon-Huang <jayson.hjs@gmail.com> * check max version before do clean read * bugfix: DMVersionFilterBlockInputStream remove pk columns by mistake
…the previous row (#334) * Don't do range filter for delta when split or merge delta * limit the rows read from delta, and read from cache if possible * Fix Unexpected end of stable stream * bugfix: miss some rows after insert
12 tasks
12 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.