support CDC #221
Merged
support CDC #221
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ondrajodas
force-pushed
the
CM-692-ondra
branch
from
a87515d
to
1e62341
Compare
ondrajodas
commented
Aug 22, 2023
| { | ||
| $this->cdcMode = $cdcMode; | ||
| if ($cdcMode === true) { | ||
| $this->disableBcp = true; |
There was a problem hiding this comment.
při CDC se spouští export jen přes PDO
| SET @from_lsn = sys.fn_cdc_map_time_to_lsn('smallest greater than or equal', @begin_time); | ||
| SET @from_lsn = ISNULL(sys.fn_cdc_map_time_to_lsn('smallest greater than or equal', @begin_time), [sys].[fn_cdc_get_min_lsn]('$cdcName')); | ||
| SET @to_lsn = sys.fn_cdc_map_time_to_lsn('largest less than or equal', @end_time); | ||
| IF @to_lsn < @from_lsn |
There was a problem hiding this comment.
když když konec časového okna bude před začátkem (to znamená že jsme nějaké okno přeskočili) tak to skončí errorem
| try { | ||
| $result = parent::export($cdcExportConfig); | ||
| } catch (Throwable $e) { | ||
| if (strpos($e->getMessage(), 'The end LSN is less than the start LSN') && |
There was a problem hiding this comment.
při chybě časových oken je volitelný fallback na full load
| throw $e; | ||
| } | ||
| } | ||
| $result['state']['lastFetchedTime'] = $lsnTime; |
There was a problem hiding this comment.
ukládám konec časového okna
Comment on lines
+120
to
+128
| $sqlToLsnTime = <<<SQL | ||
| DECLARE @to_lsn binary(10); | ||
| SET @to_lsn = [sys].[fn_cdc_get_max_lsn](); | ||
| SELECT sys.fn_cdc_map_lsn_to_time(@to_lsn) as last_fetched_time; | ||
| SQL; | ||
| $sqlToLsnTime = $this->connection->query($sqlToLsnTime); | ||
| $lsnTimeResponse = $sqlToLsnTime->fetchAll(); | ||
| assert(count($lsnTimeResponse) === 1, 'Expected one row'); | ||
| $lsnTime = $lsnTimeResponse[0]['last_fetched_time']; |
There was a problem hiding this comment.
zjištění času konce časového okna
Comment on lines
+104
to
+109
| DECLARE @begin_time datetime, @end_time datetime, @from_lsn binary(10), @to_lsn binary(10); | ||
| SET @begin_time = CONVERT(DATETIME, '{$this->state['lastFetchedTime']}'); | ||
| SET @end_time = GETDATE(); | ||
| SET @from_lsn = sys.fn_cdc_map_time_to_lsn('smallest greater than or equal', @begin_time); | ||
| SET @from_lsn = ISNULL(sys.fn_cdc_map_time_to_lsn('smallest greater than or equal', @begin_time), [sys].[fn_cdc_get_min_lsn]('$cdcName')); | ||
| SET @to_lsn = sys.fn_cdc_map_time_to_lsn('largest less than or equal', @end_time); |
There was a problem hiding this comment.
| $columns = $this->getQueryFactory()->getColumnsForSelect($exportConfig, $this->connection); | ||
|
|
||
| $cdcExportConfig = clone $exportConfig; | ||
| if (!empty($this->state['lastFetchedTime'])) { |
There was a problem hiding this comment.
CDC mode se může použít až při druhém a dalším běhu konfigurace... při prvním běhu se tohle přeskočí a pustí fullLoad
ondrajodas
requested review from
AdamVyborny
and removed request for
AdamVyborny
|
tak jsem ještě doplnil 2 testy |
|
@AdamVyborny ještě jednou pls.... fixoval jsem testy |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Jira - https://keboola.atlassian.net/browse/CM-692