Skip to content

Commit

Permalink
Merge branch 'master' into grubberr/8916-source-amazon-ads
Browse files Browse the repository at this point in the history
  • Loading branch information
@grubberr
grubberr committed Apr 28, 2022
2 parents 1c05a04 + 9ee1f79 commit 699efae
Show file tree
Hide file tree
Showing 108 changed files with 2,789 additions and 733 deletions.
2 changes: 1 addition & 1 deletion .bumpversion.cfg
View file
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
[bumpversion]
current_version = 0.36.3-alpha
current_version = 0.36.4-alpha
commit = False
tag = False
parse = (?P<major>\d+)\.(?P<minor>\d+)\.(?P<patch>\d+)(\-[a-z]+)?
Expand Down
4 changes: 2 additions & 2 deletions .env
View file
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@


### SHARED ###
VERSION=0.36.3-alpha
VERSION=0.36.4-alpha

# When using the airbyte-db via default docker image
CONFIG_ROOT=/data
Expand Down Expand Up @@ -40,7 +40,7 @@ DATABASE_PASSWORD=docker
DATABASE_HOST=db
DATABASE_PORT=5432
DATABASE_DB=airbyte
# translate manually DATABASE_URL=jdbc:postgresql://${DATABASE_HOST}:${DATABASE_PORT/${DATABASE_DB} (do not include the username or password here)
# translate manually DATABASE_URL=jdbc:postgresql://${DATABASE_HOST}:${DATABASE_PORT}/${DATABASE_DB} (do not include the username or password here)
DATABASE_URL=jdbc:postgresql://db:5432/airbyte
JOBS_DATABASE_MINIMUM_FLYWAY_MIGRATION_VERSION=0.29.15.001

Expand Down
37 changes: 0 additions & 37 deletions .github/workflows/documentation.yml
View file

This file was deleted.

1 change: 1 addition & 0 deletions .github/workflows/publish-command.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -228,6 +228,7 @@ jobs:
run: |
git add -u
git commit -m "auto-bump connector version"
git pull origin ${{ github.event.inputs.gitref }}
git push origin ${{ github.event.inputs.gitref }}
- name: Add Version Bump Success Comment
if: github.event.inputs.comment-id && github.event.inputs.auto-bump-version == 'true' && success()
Expand Down
2 changes: 1 addition & 1 deletion airbyte-bootloader/Dockerfile
View file
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
ARG JDK_VERSION=17.0.1
FROM openjdk:${JDK_VERSION}-slim

ARG VERSION=0.36.3-alpha
ARG VERSION=0.36.4-alpha

ENV APPLICATION airbyte-bootloader
ENV VERSION ${VERSION}
Expand Down
2 changes: 1 addition & 1 deletion airbyte-config/init/src/main/resources/seed/destination_definitions.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -167,7 +167,7 @@
- name: Postgres
destinationDefinitionId: 25c5221d-dce2-4163-ade9-739ef790f503
dockerRepository: airbyte/destination-postgres
dockerImageTag: 0.3.18
dockerImageTag: 0.3.19
documentationUrl: https://docs.airbyte.io/integrations/destinations/postgres
icon: postgresql.svg
- name: Pulsar
Expand Down
2 changes: 1 addition & 1 deletion airbyte-config/init/src/main/resources/seed/destination_specs.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3047,7 +3047,7 @@
supported_destination_sync_modes:
- "overwrite"
- "append"
- dockerImage: "airbyte/destination-postgres:0.3.18"
- dockerImage: "airbyte/destination-postgres:0.3.19"
spec:
documentationUrl: "https://docs.airbyte.io/integrations/destinations/postgres"
connectionSpecification:
Expand Down
10 changes: 5 additions & 5 deletions airbyte-config/init/src/main/resources/seed/source_definitions.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@
- name: Apify Dataset
sourceDefinitionId: 47f17145-fe20-4ef5-a548-e29b048adf84
dockerRepository: airbyte/source-apify-dataset
dockerImageTag: 0.1.9
dockerImageTag: 0.1.11
documentationUrl: https://docs.airbyte.io/integrations/sources/apify-dataset
icon: apify.svg
sourceType: api
Expand Down Expand Up @@ -529,7 +529,7 @@
- name: OpenWeather
sourceDefinitionId: d8540a80-6120-485d-b7d6-272bca477d9b
dockerRepository: airbyte/source-openweather
dockerImageTag: 0.1.1
dockerImageTag: 0.1.4
documentationUrl: https://docs.airbyte.io/integrations/sources/openweather
sourceType: api
- name: Oracle DB
Expand Down Expand Up @@ -737,7 +737,7 @@
- name: Smartsheets
sourceDefinitionId: 374ebc65-6636-4ea0-925c-7d35999a8ffc
dockerRepository: airbyte/source-smartsheets
dockerImageTag: 0.1.9
dockerImageTag: 0.1.10
documentationUrl: https://docs.airbyte.io/integrations/sources/smartsheets
icon: smartsheet.svg
sourceType: api
Expand All @@ -751,7 +751,7 @@
- name: Snowflake
sourceDefinitionId: e2d65910-8c8b-40a1-ae7d-ee2416b2bfa2
dockerRepository: airbyte/source-snowflake
dockerImageTag: 0.1.10
dockerImageTag: 0.1.11
documentationUrl: https://docs.airbyte.io/integrations/sources/snowflake
icon: snowflake.svg
sourceType: database
Expand Down Expand Up @@ -799,7 +799,7 @@
- name: TikTok Marketing
sourceDefinitionId: 4bfac00d-ce15-44ff-95b9-9e3c3e8fbd35
dockerRepository: airbyte/source-tiktok-marketing
dockerImageTag: 0.1.6
dockerImageTag: 0.1.7
documentationUrl: https://docs.airbyte.io/integrations/sources/tiktok-marketing
icon: tiktok.svg
sourceType: api
Expand Down
Loading

1 comment on commit 699efae

@github-actions
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SonarQube Report

SonarQube report for Airbyte Connectors Source Amazon Ads(#11730)

Measures

Name Value Name Value Name Value
Lines to Cover 34 Bugs 0 Code Smells 6
Security Rating A Quality Gate Status OK Lines of Code 1514
Duplicated Lines (%) 0.0 Reliability Rating A Vulnerabilities 0
Coverage 97.1 Duplicated Blocks 0 Blocker Issues 0
Critical Issues 0 Major Issues 13 Minor Issues 57

Detected Issues

Rule File Description Message
python:black_need_format (MINOR) report_streams/products_report.py Please run one of the commands: "black --config ./pyproject.toml <path_to_updated_folder>" or "./gradlew format" 1 code part(s) should be updated.
python:S112 (MAJOR) streams/common.py:153 "Exception" and "BaseException" should not be raised Replace this generic exception class with a more specific one.
python:mypy_assignment (MINOR) report_streams/products_report.py:251 Check that assigned value is compatible with target Incompatible types in assignment (expression has type "List[str]", base class "ReportStream" defined the type as "None") . Code line: primary_key = ["profileId", "recordType", "reportDate"]
python:mypy_attr_defined (MINOR) report_streams/report_streams.py:22 Check that attribute exists Module "pendulum" does not explicitly export attribute "DateTime"; implicit reexport disabled . Code line: from pendulum import DateTime
python:mypy_assignment (MINOR) report_streams/report_streams.py:380 Check that assigned value is compatible with target Incompatible types in assignment (expression has type "DateTime", variable has type "str") . Code line: report_date = pendulum.from_format(report_date, ReportStream.R...
python:mypy_no_any_return (MINOR) report_streams/report_streams.py:383 Reject returning value with "Any" type if return type is not "Any" Returning Any from function declared to return "str" . Code line: return profile_time.format(ReportStream.REPORT_DATE_FORMAT)
python:mypy_arg_type (MINOR) report_streams/report_streams.py:169 Check argument types in calls Argument 1 to "backoff_max_tries" has incompatible type "Callable[[ReportStream, Any], Any]"; expected "ReportStream" . Code line: @backoff_max_tries
python:mypy_arg_type (MINOR) report_streams/report_streams.py:176 Check argument types in calls Argument 1 to "backoff_max_time" has incompatible type "Callable[[ReportStream, Any], Any]"; expected "ReportStream" . Code line: @backoff_max_time
python:mypy_no_any_return (MINOR) report_streams/report_streams.py Reject returning value with "Any" type if return type is not "Any" Returning Any from function declared to return "str" . Code line: return profile_time.strftime(ReportStream.REPORT_DATE_FORMAT)
python:mypy_arg_type (MINOR) source_amazon_ads/source.py:72 Check argument types in calls Argument 1 to "_choose_profiles" of "SourceAmazonAds" has incompatible type "Mapping[str, Any]"; expected "AmazonAdsConfig" . Code line: ... stream_args["profiles"] = self._choose_profiles(config, profiles_list...
python:S1066 (MAJOR) schemas/common.py:32 Collapsible "if" statements should be merged Merge this if statement with the enclosing one.
python:mypy_call_overload (MINOR) schemas/common.py:45 Check that an overload variant matches arguments No overload variant of "create_model" matches argument types "str", "Dict[str, Tuple[Type[str], None]]", "Type[CatalogModel]" . Code line: metrics_obj_model = create_model("MetricObjModel", **{f: (str,...
python:mypy_return_value (MINOR) schemas/common.py:46 Check that return value is compatible with signature Incompatible return value type (got "Type[MetricsReport]", expected "CatalogModel") . Code line: return create_model("MetricsModel", metric=(metrics_obj_model,...
python:S5890 (MAJOR) schemas/profile.py:14 Values assigned to variables should match their type annotations Assign to "name" a value of type "str" instead of "NoneType" or update its type hint.
python:mypy_assignment (MINOR) schemas/profile.py:14 Check that assigned value is compatible with target Incompatible types in assignment (expression has type "None", variable has type "str") . Code line: name: str = None
python:S5890 (MAJOR) schemas/profile.py:15 Values assigned to variables should match their type annotations Assign to "subType" a value of type "str" instead of "NoneType" or update its type hint.
python:mypy_assignment (MINOR) schemas/profile.py:15 Check that assigned value is compatible with target Incompatible types in assignment (expression has type "None", variable has type "str") . Code line: subType: str = None
python:S5890 (MAJOR) schemas/profile.py:16 Values assigned to variables should match their type annotations Assign to "validPaymentMethod" a value of type "bool" instead of "NoneType" or update its type hint.
python:mypy_assignment (MINOR) schemas/profile.py:16 Check that assigned value is compatible with target Incompatible types in assignment (expression has type "None", variable has type "bool") . Code line: validPaymentMethod: bool = None
python:S5890 (MAJOR) schemas/profile.py:21 Values assigned to variables should match their type annotations Assign to "countryCode" a value of type "str" instead of "NoneType" or update its type hint.
python:mypy_assignment (MINOR) schemas/profile.py:21 Check that assigned value is compatible with target Incompatible types in assignment (expression has type "None", variable has type "str") . Code line: countryCode: str = None
python:S5890 (MAJOR) schemas/profile.py:22 Values assigned to variables should match their type annotations Assign to "currencyCode" a value of type "str" instead of "NoneType" or update its type hint.
python:mypy_assignment (MINOR) schemas/profile.py:22 Check that assigned value is compatible with target Incompatible types in assignment (expression has type "None", variable has type "str") . Code line: currencyCode: str = None
python:S5890 (MAJOR) schemas/profile.py:23 Values assigned to variables should match their type annotations Assign to "dailyBudget" a value of type "Decimal" instead of "NoneType" or update its type hint.
python:mypy_assignment (MINOR) schemas/profile.py:23 Check that assigned value is compatible with target Incompatible types in assignment (expression has type "None", variable has type "Decimal") . Code line: dailyBudget: Decimal = None
python:S5890 (MAJOR) schemas/sponsored_brands.py:23 Values assigned to variables should match their type annotations Assign to "bidOptimization" a value of type "bool" instead of "NoneType" or update its type hint.
python:mypy_assignment (MINOR) schemas/sponsored_brands.py:23 Check that assigned value is compatible with target Incompatible types in assignment (expression has type "None", variable has type "bool") . Code line: bidOptimization: bool = None
python:S5890 (MAJOR) schemas/sponsored_brands.py:24 Values assigned to variables should match their type annotations Assign to "bidMultiplier" a value of type "Decimal" instead of "NoneType" or update its type hint.
python:mypy_assignment (MINOR) schemas/sponsored_brands.py:24 Check that assigned value is compatible with target Incompatible types in assignment (expression has type "None", variable has type "Decimal") . Code line: bidMultiplier: Decimal = None
python:S5890 (MAJOR) schemas/sponsored_display.py:16 Values assigned to variables should match their type annotations Assign to "endDate" a value of type "str" instead of "NoneType" or update its type hint.
python:mypy_assignment (MINOR) schemas/sponsored_display.py:16 Check that assigned value is compatible with target Incompatible types in assignment (expression has type "None", variable has type "str") . Code line: endDate: str = None
python:S5890 (MAJOR) schemas/sponsored_display.py:19 Values assigned to variables should match their type annotations Assign to "portfolioId" a value of type "str" instead of "NoneType" or update its type hint.
python:mypy_assignment (MINOR) schemas/sponsored_display.py:19 Check that assigned value is compatible with target Incompatible types in assignment (expression has type "None", variable has type "str") . Code line: portfolioId: str = None
python:S5890 (MAJOR) schemas/sponsored_products.py:31 Values assigned to variables should match their type annotations Assign to "endDate" a value of type "str" instead of "NoneType" or update its type hint.
python:mypy_assignment (MINOR) schemas/sponsored_products.py:31 Check that assigned value is compatible with target Incompatible types in assignment (expression has type "None", variable has type "str") . Code line: endDate: str = None
python:mypy_valid_type (MINOR) source_amazon_ads/source.py:42 Check that type (annotation) is valid Function "builtins.any" is not valid as a type . Code line: ...logger: AirbyteLogger, config: Mapping[str, Any]) -> Tuple[bool, any]:
python:mypy_assignment (MINOR) source_amazon_ads/source.py:48 Check that assigned value is compatible with target Incompatible types in assignment (expression has type "AmazonAdsConfig", variable has type "Mapping[str, Any]") . Code line: config = AmazonAdsConfig(**config)
python:mypy_arg_type (MINOR) source_amazon_ads/source.py:54 Check argument types in calls Argument 1 to "Profiles" has incompatible type "Mapping[str, Any]"; expected "AmazonAdsConfig" . Code line: Profiles(config, authenticator=self._make_authenticator(config...
python:mypy_arg_type (MINOR) source_amazon_ads/source.py:54 Check argument types in calls Argument 1 to "_make_authenticator" of "SourceAmazonAds" has incompatible type "Mapping[str, Any]"; expected "AmazonAdsConfig" . Code line: ...s(config, authenticator=self._make_authenticator(config)).get_all_prof...
python:mypy_assignment (MINOR) source_amazon_ads/source.py:62 Check that assigned value is compatible with target Incompatible types in assignment (expression has type "AmazonAdsConfig", variable has type "Mapping[str, Any]") . Code line: config = AmazonAdsConfig(**config)
python:mypy_arg_type (MINOR) source_amazon_ads/source.py:63 Check argument types in calls Argument 1 to "_make_authenticator" of "SourceAmazonAds" has incompatible type "Mapping[str, Any]"; expected "AmazonAdsConfig" . Code line: auth = self._make_authenticator(config)
python:mypy_import (MINOR) streams/common.py:9 Require that imported module can be found or has stubs Library stubs not installed for "requests" (or incompatible with Python 3.9) . Code line: import requests
python:mypy_override (MINOR) streams/common.py:172 Check that method override is compatible with base class Return type "Optional[int]" of "next_page_token" incompatible with return type "Optional[Mapping[str, Any]]" in supertype "AmazonAdsStream" . Code line: def next_page_token(self, response: requests.Response) -> Optional...
python:mypy_no_any_return (MINOR) streams/common.py:183 Reject returning value with "Any" type if return type is not "Any" Returning Any from function declared to return "Optional[int]" . Code line: return next_offset
python:mypy_import (MINOR) streams/profiles.py:7 Require that imported module can be found or has stubs Library stubs not installed for "requests" (or incompatible with Python 3.9) . Code line: import requests
python:mypy_assignment (MINOR) streams/profiles.py:20 Check that assigned value is compatible with target Incompatible types in assignment (expression has type "Type[Profile]", base class "BasicAmazonAdsStream" defined the type as "CatalogModel") . Code line: model = Profile
python:mypy_has_type (MINOR) streams/profiles.py:27 Check that type of reference can be determined Cannot determine type of "model" . Code line: profile_id_obj = self.model.parse_obj(record)
python:mypy_has_type (MINOR) streams/profiles.py:48 Check that type of reference can be determined Cannot determine type of "model" . Code line: return [self.model.parse_obj(profile) for profile in self.read...
python:mypy_assignment (MINOR) report_streams/report_streams.py Check that assigned value is compatible with target Incompatible types in assignment (expression has type "str", variable has type "datetime") . Code line: next_date = next_date.strftime(ReportStream.REPORT_DATE_FO...
python:mypy_misc (MINOR) report_streams/report_streams.py Miscellaneous other checks Incompatible types in "yield" (actual type "datetime", expected type "str") . Code line: yield next_date
python:mypy_assignment (MINOR) report_streams/report_streams.py Check that assigned value is compatible with target Incompatible types in assignment (expression has type "datetime", variable has type "str") . Code line: report_date = datetime.strptime(report_date, ReportStream.REPO...
python:mypy_import (MINOR) report_streams/report_streams.py:17 Require that imported module can be found or has stubs Library stubs not installed for "pytz" (or incompatible with Python 3.9) . Code line: import pytz
python:mypy_import (MINOR) report_streams/report_streams.py:18 Require that imported module can be found or has stubs Library stubs not installed for "requests" (or incompatible with Python 3.9) . Code line: import requests
python:mypy_attr_defined (MINOR) report_streams/report_streams.py:111 Check that attribute exists Module has no attribute "parse" . Code line: self._start_date = pendulum.parse(config.start_date).set(tz="U...
python:mypy_no_any_return (MINOR) report_streams/report_streams.py:116 Reject returning value with "Any" type if return type is not "Any" Returning Any from function declared to return "CatalogModel" . Code line: return self._model
python:mypy_attr_defined (MINOR) report_streams/report_streams.py:382 Check that attribute exists "str" has no attribute "astimezone" . Code line: profile_time = report_date.astimezone(profile_tz)
python:mypy_no_any_return (MINOR) report_streams/report_streams.py:397 Reject returning value with "Any" type if return type is not "Any" Returning Any from function declared to return "List[Dict[Any, Any]]" . Code line: return json.loads(raw_string)
python:mypy_assignment (MINOR) streams/sponsored_brands.py:16 Check that assigned value is compatible with target Incompatible types in assignment (expression has type "Type[BrandsCampaign]", base class "BasicAmazonAdsStream" defined the type as "CatalogModel") . Code line: model = BrandsCampaign
python:mypy_assignment (MINOR) streams/sponsored_brands.py:29 Check that assigned value is compatible with target Incompatible types in assignment (expression has type "Type[BrandsAdGroup]", base class "BasicAmazonAdsStream" defined the type as "CatalogModel") . Code line: model = BrandsAdGroup
python:mypy_assignment (MINOR) streams/sponsored_brands.py:42 Check that assigned value is compatible with target Incompatible types in assignment (expression has type "Type[BrandsAdGroup]", base class "BasicAmazonAdsStream" defined the type as "CatalogModel") . Code line: model = BrandsAdGroup
python:mypy_assignment (MINOR) streams/sponsored_display.py:16 Check that assigned value is compatible with target Incompatible types in assignment (expression has type "Type[DisplayCampaign]", base class "BasicAmazonAdsStream" defined the type as "CatalogModel") . Code line: model = DisplayCampaign
python:mypy_assignment (MINOR) streams/sponsored_display.py:29 Check that assigned value is compatible with target Incompatible types in assignment (expression has type "Type[DisplayAdGroup]", base class "BasicAmazonAdsStream" defined the type as "CatalogModel") . Code line: model = DisplayAdGroup
python:mypy_assignment (MINOR) streams/sponsored_display.py:42 Check that assigned value is compatible with target Incompatible types in assignment (expression has type "Type[DisplayProductAds]", base class "BasicAmazonAdsStream" defined the type as "CatalogModel") . Code line: model = DisplayProductAds
python:mypy_assignment (MINOR) streams/sponsored_display.py:55 Check that assigned value is compatible with target Incompatible types in assignment (expression has type "Type[DisplayTargeting]", base class "BasicAmazonAdsStream" defined the type as "CatalogModel") . Code line: model = DisplayTargeting
python:mypy_assignment (MINOR) streams/sponsored_products.py:16 Check that assigned value is compatible with target Incompatible types in assignment (expression has type "Type[ProductCampaign]", base class "BasicAmazonAdsStream" defined the type as "CatalogModel") . Code line: model = ProductCampaign
python:mypy_assignment (MINOR) streams/sponsored_products.py:29 Check that assigned value is compatible with target Incompatible types in assignment (expression has type "Type[ProductAdGroups]", base class "BasicAmazonAdsStream" defined the type as "CatalogModel") . Code line: model = ProductAdGroups
python:mypy_assignment (MINOR) streams/sponsored_products.py:42 Check that assigned value is compatible with target Incompatible types in assignment (expression has type "Type[Keywords]", base class "BasicAmazonAdsStream" defined the type as "CatalogModel") . Code line: model = Keywords
python:mypy_assignment (MINOR) streams/sponsored_products.py:55 Check that assigned value is compatible with target Incompatible types in assignment (expression has type "Type[NegativeKeywords]", base class "BasicAmazonAdsStream" defined the type as "CatalogModel") . Code line: model = NegativeKeywords
python:mypy_assignment (MINOR) streams/sponsored_products.py:68 Check that assigned value is compatible with target Incompatible types in assignment (expression has type "Type[ProductAd]", base class "BasicAmazonAdsStream" defined the type as "CatalogModel") . Code line: model = ProductAd
python:mypy_assignment (MINOR) streams/sponsored_products.py:81 Check that assigned value is compatible with target Incompatible types in assignment (expression has type "Type[ProductTargeting]", base class "BasicAmazonAdsStream" defined the type as "CatalogModel") . Code line: model = ProductTargeting

Coverage (97.1%)

File Coverage File Coverage
source_amazon_ads/init.py 100.0 source_amazon_ads/constants.py 100.0
source_amazon_ads/schemas/init.py 100.0 source_amazon_ads/schemas/common.py 100.0
source_amazon_ads/schemas/profile.py 100.0 source_amazon_ads/schemas/sponsored_brands.py 100.0
source_amazon_ads/schemas/sponsored_display.py 100.0 source_amazon_ads/schemas/sponsored_products.py 100.0
source_amazon_ads/source.py 97.1 source_amazon_ads/spec.py 100.0
source_amazon_ads/streams/init.py 100.0 source_amazon_ads/streams/common.py 98.7
source_amazon_ads/streams/profiles.py 100.0 source_amazon_ads/streams/report_streams/init.py 100.0
source_amazon_ads/streams/report_streams/brands_report.py 100.0 source_amazon_ads/streams/report_streams/brands_video_report.py 100.0
source_amazon_ads/streams/report_streams/display_report.py 100.0 source_amazon_ads/streams/report_streams/products_report.py 100.0
source_amazon_ads/streams/report_streams/report_streams.py 97.0 source_amazon_ads/streams/sponsored_brands.py 100.0
source_amazon_ads/streams/sponsored_display.py 100.0 source_amazon_ads/streams/sponsored_products.py 100.0

Please sign in to comment.