Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Catalog: Attach namespace to stream #13923

Merged
merged 24 commits into from
Jul 4, 2022
Merged
Show file tree
Hide file tree
Changes from 14 commits
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
ccffa11
allow nullable container_expiration_policy
jordan-glitch Apr 12, 2022
097297c
Update Dockerfile
jordan-glitch Apr 12, 2022
d826cda
Update source_definitions.yaml
jordan-glitch Apr 12, 2022
8af6e79
Update source_specs.yaml
jordan-glitch Apr 12, 2022
50ae513
fix unspecified projects
jordan-glitch Apr 12, 2022
bff9044
Merge branch 'master' of github.com:jordan-glitch/airbyte into marcos…
marcosmarxm May 2, 2022
5b9e863
add doc update
marcosmarxm May 2, 2022
70daccd
Merge branch 'master' into marcos/test-pr-11907
marcosmarxm May 2, 2022
2cfe2d2
Merge branch 'airbytehq:master' into master
jordan-glitch Jun 20, 2022
f54698b
attach namespace property to stream if exists
jordan-glitch Jun 20, 2022
086c7e3
rework namespace location in stream
jordan-glitch Jun 23, 2022
7ca925e
add namespace property to stream
jordan-glitch Jun 23, 2022
03c3457
add tests for namespace
jordan-glitch Jun 23, 2022
edb6ed5
make consistent spacing
jordan-glitch Jun 23, 2022
68f7433
test as_airbyte_stream() method
jordan-glitch Jun 23, 2022
2db5885
Update CHANGELOG.md
jordan-glitch Jun 23, 2022
02c0632
Merge branch 'master' into jordan-glitch-patch-1
alafanechere Jun 24, 2022
9e13429
bump version in setup.py
alafanechere Jun 24, 2022
0bc2972
Update airbyte-cdk/python/airbyte_cdk/sources/streams/core.py
jordan-glitch Jun 24, 2022
8cdfd17
format
alafanechere Jun 24, 2022
907a626
update namespace tests
jordan-glitch Jun 27, 2022
9ed38f6
Merge branch 'master' into jordan-glitch-patch-1
alafanechere Jul 4, 2022
c869fbd
fix test_namespace_set_to_empty_string
alafanechere Jul 4, 2022
71bc9f8
fix test_namespace_set_to_empty_string
alafanechere Jul 4, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 12 additions & 1 deletion airbyte-cdk/python/airbyte_cdk/sources/streams/core.py
Original file line number Diff line number Diff line change
Expand Up @@ -111,7 +111,10 @@ def get_json_schema(self) -> Mapping[str, Any]:

def as_airbyte_stream(self) -> AirbyteStream:
stream = AirbyteStream(name=self.name, json_schema=dict(self.get_json_schema()), supported_sync_modes=[SyncMode.full_refresh])
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
stream = AirbyteStream(name=self.name, json_schema=dict(self.get_json_schema()), supported_sync_modes=[SyncMode.full_refresh])
stream = AirbyteStream(name=self.name, json_schema=dict(self.get_json_schema()), supported_sync_modes=[SyncMode.full_refresh], namespace=self.namespace)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you, I have made this requested change.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reverted based on discussion below.



if self.namespace and self.namespace != "":
jordan-glitch marked this conversation as resolved.
Show resolved Hide resolved
stream.namespace = self.namespace

if self.supports_incremental:
stream.source_defined_cursor = self.source_defined_cursor
stream.supported_sync_modes.append(SyncMode.incremental) # type: ignore
Expand Down Expand Up @@ -141,6 +144,14 @@ def cursor_field(self) -> Union[str, List[str]]:
"""
return []

@property
def namespace(self) -> Optional[str]:
"""
Override to return the namespace of this stream, e.g. the Postgres schema which this stream will emit records for.
:return: A string containing the name of the namespace.
"""
return None

@property
def source_defined_cursor(self) -> bool:
"""
Expand Down
29 changes: 29 additions & 0 deletions airbyte-cdk/python/unit_tests/sources/streams/test_streams_core.py
Original file line number Diff line number Diff line change
Expand Up @@ -99,6 +99,35 @@ def test_supports_incremental_cursor_not_set():
assert not test_stream.supports_incremental


def test_namespace_set():
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we should also add tests to verify that as_airbyte_stream works as expected. See the tests above for an example

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i added the namespace property to the existing test, i hope that is okay? Otherwise, happy to repeat underneath.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great!

"""
Should allow namespace property to be set.
"""
test_stream = StreamStubFullRefresh()
test_stream.namespace = "test_namespace"

assert test_stream.namespace == "test_namespace"


def test_namespace_set_to_empty_string():
"""
Should not set namespace property if equal to empty string.
"""
test_stream = StreamStubFullRefresh()
test_stream.namespace = ""

assert test_stream.namespace == None


def test_namespace_not_set():
"""
Should be equal to unset value of None.
"""
test_stream = StreamStubFullRefresh()

assert test_stream.namespace == None


@pytest.mark.parametrize(
"test_input, expected",
[("key", [["key"]]), (["key1", "key2"], [["key1"], ["key2"]]), ([["key1", "key2"], ["key3"]], [["key1", "key2"], ["key3"]])],
Expand Down