Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Update source-gnews manifest to use inline stream schemas #20405

Merged
merged 12 commits into from
Dec 19, 2022
Merged

Update source-gnews manifest to use inline stream schemas #20405

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
5d5e5a5
Update source-gnews manifest to use inline stream schemas
clnoll Dec 13, 2022
1678bb2
Fix gnews.md table formatting
clnoll Dec 14, 2022
febe06f
Use `airbyte-cdk~=0.14`
clnoll Dec 14, 2022
10d7558
Merge branch 'master' into source-gnews-use-inline-schema
clnoll Dec 14, 2022
e39c2d9
Merge branch 'master' into source-gnews-use-inline-schema
clnoll Dec 14, 2022
c7620fe
Update gnews changelog
clnoll Dec 16, 2022
a8b6946
Merge branch 'master' into source-gnews-use-inline-schema
clnoll Dec 16, 2022
4a351c1
Update Dockerfile io.airbyte.version
clnoll Dec 16, 2022
56b0895
auto-bump connector version
octavia-squidington-iii Dec 16, 2022
a69fbb6
Merge branch 'master' into source-gnews-use-inline-schema
clnoll Dec 16, 2022
8221d2e
Merge branch 'master' into source-gnews-use-inline-schema
clnoll Dec 16, 2022
f9f1700
Merge branch 'master' into source-gnews-use-inline-schema
clnoll Dec 19, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion airbyte-config/init/src/main/resources/seed/source_definitions.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -578,7 +578,7 @@
- name: GNews
sourceDefinitionId: ce38aec4-5a77-439a-be29-9ca44fd4e811
dockerRepository: airbyte/source-gnews
dockerImageTag: 0.1.0
dockerImageTag: 0.1.2
documentationUrl: https://docs.airbyte.com/integrations/sources/gnews
sourceType: api
releaseStage: alpha
Expand Down
2 changes: 1 addition & 1 deletion airbyte-config/init/src/main/resources/seed/source_specs.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4746,7 +4746,7 @@
supportsNormalization: false
supportsDBT: false
supported_destination_sync_modes: []
- dockerImage: "airbyte/source-gnews:0.1.0"
- dockerImage: "airbyte/source-gnews:0.1.2"
spec:
documentationUrl: "https://docs.airbyte.com/integrations/sources/gnews"
connectionSpecification:
Expand Down
2 changes: 1 addition & 1 deletion airbyte-integrations/connectors/source-gnews/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,5 +34,5 @@ COPY source_gnews ./source_gnews
ENV AIRBYTE_ENTRYPOINT "python /airbyte/integration_code/main.py"
ENTRYPOINT ["python", "/airbyte/integration_code/main.py"]

LABEL io.airbyte.version=0.1.0
LABEL io.airbyte.version=0.1.2
LABEL io.airbyte.name=airbyte/source-gnews
2 changes: 1 addition & 1 deletion airbyte-integrations/connectors/source-gnews/setup.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
from setuptools import find_packages, setup

MAIN_REQUIREMENTS = [
"airbyte-cdk~=0.1",
"airbyte-cdk~=0.14",
]

TEST_REQUIREMENTS = [
Expand Down
110 changes: 110 additions & 0 deletions airbyte-integrations/connectors/source-gnews/source_gnews/gnews.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,107 @@
version: "0.1.0"

schemas:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit, but can we move the schemas to the bottom of the file? i think the definitions and streams are likely to be the parts of the manifest that change most, so it's more intuitive to have them at the top.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ah sorry you are correct and my mistake. I thought we did some preparsing to allow for forward references even if defined after, but doesn't seem like we do, so it's fine to have this at the top.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No worries (sorry, deleted my comment because I thought maybe I'd spoken prematurely and was looking more closely at the issue). Going to write up an issue for updating the parser to handle forward references, since I do think that will make the yaml more readable.

search_stream_schema:
type: object
properties:
title:
type:
- "null"
- string
description: The main title of the article.
description:
type:
- "null"
- string
description: The small paragraph under the title.
content:
type:
- "null"
- string
description: All the content of the article.
url:
type:
- "null"
- string
description: The URL of the article.
image:
type:
- "null"
- string
description: The main image of the article.
publishedAt:
type:
- "null"
- string
description:
The date of publication of the article. The date is always in the
UTC time zone.
source:
type:
- "null"
- object
properties:
name:
type:
- "null"
- string
description: The name of the source.
url:
type:
- "null"
- string
description: The home page of the source.
top_headlines_stream_schema:
type: object
properties:
title:
type:
- "null"
- string
description: The main title of the article.
description:
type:
- "null"
- string
description: The small paragraph under the title.
content:
type:
- "null"
- string
description: All the content of the article.
url:
type:
- "null"
- string
description: The URL of the article.
image:
type:
- "null"
- string
description: The main image of the article.
publishedAt:
type:
- "null"
- string
description:
The date of publication of the article. The date is always in the
UTC time zone.
source:
type:
- "null"
- object
properties:
name:
type:
- "null"
- string
description: The name of the source.
url:
type:
- "null"
- string
description: The home page of the source.

definitions:
selector:
extractor:
Expand Down Expand Up @@ -44,7 +146,12 @@ definitions:
nullable: "{{ ','.join(config['nullable']) }}"
from: "{{ stream_slice['start_time'] }}"
to: "{{ stream_slice['end_time'] }}"
schema_loader:
type: InlineSchemaLoader
search_stream:
schema_loader:
$ref: "*ref(definitions.schema_loader)"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fwiw I would find it more understandable if this said

  schema_loader:
    type: InlineSchemaLoader

instead of the ref

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated.

schema: "*ref(schemas.search_stream_schema)"
$options:
name: "search"
primary_key: "url"
Expand All @@ -61,6 +168,9 @@ definitions:
in: "{{ ','.join(config['in']) }}"
sortby: "{{ config['sortby'] }}"
top_headlines_stream:
schema_loader:
$ref: "*ref(definitions.schema_loader)"
schema: "*ref(schemas.top_headlines_stream_schema)"
$options:
name: "top_headlines"
primary_key: "url"
Expand Down
43 changes: 0 additions & 43 deletions airbyte-integrations/connectors/source-gnews/source_gnews/schemas/search.json
View file
Open in desktop

This file was deleted.

43 changes: 0 additions & 43 deletions airbyte-integrations/connectors/source-gnews/source_gnews/schemas/top_headlines.json
View file
Open in desktop

This file was deleted.

9 changes: 5 additions & 4 deletions docs/integrations/sources/gnews.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,8 @@ Rate Limiting is based on the API Key tier subscription, get more info [here](ht

## Changelog

| Version | Date | Pull Request | Subject |
|:--------|:-----------|:---------------------------------------------------------|:-------------------------------------|
| 0.1.1 | 2022-12-13 | [20460](https://github.com/airbytehq/airbyte/pull/20460) | Update source acceptance test config |
| 0.1.0 | 2022-11-01 | [18808](https://github.com/airbytehq/airbyte/pull/18808) | 🎉 New Source: GNews |
| Version | Date | Pull Request | Subject |
|:--------|:-----------|:---------------------------------------------------------|:-------------------------------------------------|
| 0.1.2 | 2022-12-16 | [20405](https://github.com/airbytehq/airbyte/pull/20405) | Update the manifest to use inline stream schemas |
| 0.1.1 | 2022-12-13 | [20460](https://github.com/airbytehq/airbyte/pull/20460) | Update source acceptance test config |
| 0.1.0 | 2022-11-01 | [18808](https://github.com/airbytehq/airbyte/pull/18808) | 🎉 New Source: GNews |