-
Notifications
You must be signed in to change notification settings - Fork 4.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Stop sync on a null value in a cursor column #19889
Merged
rodireich
merged 19 commits into
master
from
17150-database-sources-handle-when-a-cursor-value-is-null
Dec 6, 2022
Merged
Changes from 5 commits
Commits
Show all changes
19 commits
Select commit
Hold shift + click to select a range
43e6bf5
Stop sync on a null value in a cursor column
rodireich 3d85e67
Fix quoting in query to solve a failing test
rodireich e6150e3
Fix another escaping issue in query
rodireich d567f6b
Fix failing test
rodireich f47a713
Fix failing tests
rodireich 963d331
Test view with null value cursor
rodireich 4d197ad
Improve error message
rodireich 94b30a7
bump dockerfile version and update note
rodireich 3281a4c
Merge branch 'master' into 17150-database-sources-handle-when-a-curso…
rodireich b3241d0
Merge branch 'master' into 17150-database-sources-handle-when-a-curso…
rodireich 1d16cf9
Merge branch 'master' into 17150-database-sources-handle-when-a-curso…
rodireich d52fea2
bump version to 1.0.31
rodireich ebac1b9
Merge branch 'master' into 17150-database-sources-handle-when-a-curso…
rodireich 5be47f3
Merge branch 'master' into 17150-database-sources-handle-when-a-curso…
rodireich 09b3d2e
Merge branch 'master' into 17150-database-sources-handle-when-a-curso…
rodireich 7dd7b78
Merge branch 'master' into 17150-database-sources-handle-when-a-curso…
rodireich 0641b56
Fix failing test
rodireich 00bbe1c
Merge branch 'master' into 17150-database-sources-handle-when-a-curso…
rodireich 46c341e
auto-bump connector version
octavia-squidington-iii File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -500,7 +500,7 @@ public void tableWithInvalidCursorShouldThrowException() throws Exception { | |
final Throwable throwable = catchThrowable(() -> MoreIterators.toSet(new PostgresSource().read(config, configuredAirbyteCatalog, null))); | ||
assertThat(throwable).isInstanceOf(ConfigErrorException.class) | ||
.hasMessageContaining( | ||
"The following tables have invalid columns selected as cursor, please select a column with a well-defined ordering as a cursor. {tableName='public.test_table', cursorColumnName='id', cursorSqlType=OTHER}"); | ||
"The following tables have invalid columns selected as cursor, please select a column with a well-defined ordering as a cursor. {tableName='public.test_table', cursorColumnName='id', cursorSqlType=OTHER, cause=Unsupported cursor type}"); | ||
} finally { | ||
db.stop(); | ||
} | ||
|
@@ -544,4 +544,44 @@ private JsonNode buildConfigEscapingNeeded() { | |
JdbcUtils.SSL_KEY, "false")); | ||
} | ||
|
||
@Test | ||
public void tableWithNullValueCursorShouldThrowException() throws SQLException { | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Lets add a similar test where we have a VIEW with null value for cursor column There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Added |
||
try (final PostgreSQLContainer<?> db = new PostgreSQLContainer<>("postgres:13-alpine")) { | ||
db.start(); | ||
final JsonNode config = getConfig(db); | ||
try (final DSLContext dslContext = getDslContext(config)) { | ||
final Database database = new Database(dslContext); | ||
final ConfiguredAirbyteStream table = createTableWithNullValueCursor(database); | ||
final ConfiguredAirbyteCatalog catalog = new ConfiguredAirbyteCatalog().withStreams(Collections.singletonList(table)); | ||
|
||
final Throwable throwable = catchThrowable(() -> MoreIterators.toSet(new PostgresSource().read(config, catalog, null))); | ||
assertThat(throwable).isInstanceOf(ConfigErrorException.class) | ||
.hasMessageContaining( | ||
"The following tables have invalid columns selected as cursor, please select a column with a well-defined ordering as a cursor. {tableName='public.test_table_null_cursor', cursorColumnName='id', cursorSqlType=INTEGER, cause=Cursor column contains NULL value}"); | ||
|
||
} finally { | ||
db.stop(); | ||
} | ||
} | ||
} | ||
|
||
private ConfiguredAirbyteStream createTableWithNullValueCursor(final Database database) throws SQLException { | ||
database.query(ctx -> { | ||
ctx.fetch("CREATE TABLE IF NOT EXISTS public.test_table_null_cursor(id INTEGER NULL);"); | ||
ctx.fetch("INSERT INTO public.test_table_null_cursor(id) VALUES (1), (2), (NULL);"); | ||
return null; | ||
}); | ||
|
||
return new ConfiguredAirbyteStream().withSyncMode(SyncMode.INCREMENTAL) | ||
.withCursorField(Lists.newArrayList("id")) | ||
.withDestinationSyncMode(DestinationSyncMode.APPEND) | ||
.withSyncMode(SyncMode.INCREMENTAL) | ||
.withStream(CatalogHelpers.createAirbyteStream( | ||
"test_table_null_cursor", | ||
"public", | ||
Field.of("id", JsonSchemaType.STRING)) | ||
.withSupportedSyncModes(Lists.newArrayList(SyncMode.FULL_REFRESH, SyncMode.INCREMENTAL)) | ||
.withSourceDefinedPrimaryKey(List.of(List.of("id")))); | ||
|
||
} | ||
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Will this query also cover VIEWS with null values?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes. I verified this with views as well.
Good idea to add a test of view. working on it