Source Exchange Rates - Refactor for API Layer #15884

jonathanneo · 2022-08-23T14:13:43Z

What

The Rates API used for the Airbyte demo for Python CDK (HTTP) connectors has been deprecated.

The Rates API is now being served by API Layer. Here is the new API documentation.

This PR resolves the following issues:

How

Change url_base to the new url: https://api.apilayer.com/exchangerates_data/
Change spec.yaml to use api_key instead of access_key so that it aligns with the new API terminology
Add request_headers() to pass a header with apikey to the endpoint

🚨 User Impact 🚨

The spec now uses api_key instead of access_key so that it aligns with the new API terminology.

Pre-merge Checklist

Expand the relevant checklist and delete the others.

Updating a connector

Community member or Airbyter

Grant edit access to maintainers (instructions)
Secrets in the connector's spec are annotated with airbyte_secret
Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
Code reviews completed
Documentation updated
- Connector's README.md
- Connector's bootstrap.md. See description and examples
- Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.
/test connector=connectors/<name> command is passing
New Connector version released on Dockerhub and connector version bumped by running the /publish command described here

Tests

Unit

No unit tests were created by the original airbyte author.

Integration

Running integration tests locally from the root of the repo using the command ./gradlew :airbyte-integrations:connectors:source-exchange-rates:integrationTest results in the following error:

FAILURE: Build failed with an exception.

* Where:
Build file '/Users/jonathanneo/code/jonathanneo/airbyte/build.gradle' line: 198

* What went wrong:
A problem occurred evaluating root project 'airbyte'.
> java.lang.UnsupportedClassVersionError: com/bmuschko/gradle/docker/DockerRemoteApiPlugin has been compiled by a more recent version of the Java Runtime (class file version 55.0), this version of the Java Runtime only recognizes class file versions up to 52.0

Acceptance

Running docker run --rm -v $(pwd)/integration_tests:/integration_tests airbyte/source-exchange-rates:dev read --config /integration_tests/config.json --catalog /integration_tests/configured_catalog.json resulted in a successful sync:

{"type": "RECORD", "record": {"stream": "exchange_rates", "data": {"success": true, "timestamp": 1661265424, "historical": true, "base": "USD", "date": "2022-08-23", "rates": {"AED": 3.673103, "AFN": 88.999668, "ALL": 117.874989, "AMD": 404.929874, "ANG": 1.802425, "AOA": 429.28555, "ARS": 136.884802, "AUD": 1.442294, "AWG": 1.8055, "AZN": 1.702342, "BAM": 1.970947, "BBD": 2.019288, "BDT": 95.032196, "BGN": 1.960328, "BHD": 0.376956, "BIF": 2046, "BMD": 1, "BND": 1.396195, "BOB": 6.910705, "BRL": 5.092601, "BSD": 1.000071, "BTC": 4.6440968e-05, "BTN": 79.83403, "BWP": 12.83827, "BYN": 2.524362, "BYR": 19600, "BZD": 2.015862, "CAD": 1.29701, "CDF": 2001.000153, "CHF": 0.963275, "CLF": 0.03355, "CLP": 925.749715, "CNY": 6.836402, "COP": 4379.9, "CRC": 646.377717, "CUC": 1, "CUP": 26.5, "CVE": 111.11615, "CZK": 24.703608, "DJF": 178.04561, "DKK": 7.45388, "DOP": 53.25008, "DZD": 141.204424, "EGP": 19.171398, "ERN": 15, "ETB": 52.890421, "EUR": 1.002205, "FJD": 2.20175, "FKP": 0.823605, "GBP": 0.84506, "GEL": 2.789737, "GGP": 0.823605, "GHS": 9.824967, "GIP": 0.823605, "GMD": 54.424996, "GNF": 8754.999711, "GTQ": 7.738252, "GYD": 209.232723, "HKD": 7.84732, "HNL": 24.550132, "HRK": 7.529299, "HTG": 128.512692, "HUF": 414.357027, "IDR": 14850.4, "ILS": 3.262375, "IMP": 0.823605, "INR": 79.783499, "IQD": 1460, "IRR": 42350.000177, "ISK": 140.429707, "JEP": 0.823605, "JMD": 150.949785, "JOD": 0.709028, "JPY": 136.409007, "KES": 119.849744, "KGS": 80.328902, "KHR": 4114.50145, "KMF": 493.94986, "KPW": 900.00035, "KRW": 1336.929896, "KWD": 0.30775, "KYD": 0.833392, "KZT": 466.909875, "LAK": 15279.999655, "LBP": 1505.652441, "LKR": 365.277775, "LRD": 153.64994, "LSL": 16.989678, "LTL": 2.95274, "LVL": 0.60489, "LYD": 4.920557, "MAD": 10.503503, "MDL": 19.351829, "MGA": 4116.000089, "MKD": 62.090614, "MMK": 2100.158916, "MNT": 3194.961563, "MOP": 8.083256, "MRO": 356.999828, "MUR": 44.149314, "MVR": 15.35044, "MWK": 1022.502229, "MXN": 20.01538, "MYR": 4.484996, "MZN": 63.82977, "NAD": 16.990023, "NGN": 421.210168, "NIO": 35.949729, "NOK": 9.73107, "NPR": 127.734166, "NZD": 1.606935, "OMR": 0.384958, "PAB": 1.000071, "PEN": 3.868042, "PGK": 3.514995, "PHP": 56.014963, "PKR": 216.695056, "PLN": 4.78277, "PYG": 6887.728881, "QAR": 3.641017, "RON": 4.894301, "RSD": 117.659966, "RUB": 60.005502, "RWF": 1031.5, "SAR": 3.755612, "SBD": 8.220417, "SCR": 12.740352, "SDG": 570.000198, "SEK": 10.613525, "SGD": 1.392245, "SHP": 1.377402, "SLL": 13994.999805, "SOS": 290.834995, "SRD": 24.4855, "STD": 20697.981008, "SVC": 8.751121, "SYP": 2512.53037, "SZL": 16.9905, "THB": 36.030103, "TJS": 10.231075, "TMT": 3.51, "TND": 3.1855, "TOP": 2.353903, "TRY": 18.112403, "TTD": 6.789105, "TWD": 30.172502, "TZS": 2332.227985, "UAH": 36.930457, "UGX": 3845.394173, "USD": 1, "UYU": 40.508702, "UZS": 10910.000175, "VND": 23435, "VUV": 116.056079, "WST": 2.66746, "XAF": 661.000433, "XAG": 0.052207, "XAU": 0.000572, "XCD": 2.70255, "XDR": 0.765642, "XOF": 654.501052, "XPF": 120.324979, "YER": 250.249645, "ZAR": 16.9472, "ZMK": 9001.198797, "ZMW": 16.169016, "ZWL": 321.999592}}, "emitted_at": 1661265436541}}
{"type": "STATE", "state": {"data": {"exchange_rates": {"date": "2022-08-23"}}}}
{"type": "LOG", "log": {"level": "INFO", "message": "Read 6 records from exchange_rates stream"}}
{"type": "LOG", "log": {"level": "INFO", "message": "Finished syncing exchange_rates"}}
{"type": "LOG", "log": {"level": "INFO", "message": "SourceExchangeRates runtimes:\nSyncing stream exchange_rates 0:00:03.158786"}}
{"type": "LOG", "log": {"level": "INFO", "message": "Finished syncing SourceExchangeRates"}}

CLAassistant · 2022-08-23T14:13:54Z

All committers have signed the CLA.

jonathanneo · 2022-08-23T14:22:06Z

/test connector=source-exchange-rates

natalyjazzviolin · 2022-08-24T08:45:47Z

/test connector=connectors/source-exchange-rates

🕑 connectors/source-exchange-rates https://github.com/airbytehq/airbyte/actions/runs/2917709277
❌ connectors/source-exchange-rates https://github.com/airbytehq/airbyte/actions/runs/2917709277
🐛 https://gradle.com/s/4pvx526s4lf64

Build Failed

Test summary info:

=========================== short test summary info ============================
FAILED test_core.py::TestSpec::test_config_match_spec[inputs0] - Failed: Conf...
FAILED test_core.py::TestSpec::test_backward_compatibility[inputs0] - source_...
FAILED test_core.py::TestSpec::test_additional_properties_is_true[inputs0] - ...
FAILED test_core.py::TestConnection::test_check[inputs0] - docker.errors.Cont...
FAILED test_core.py::TestConnection::test_check[inputs1] - docker.errors.Cont...
FAILED test_core.py::TestDiscovery::test_discover[inputs0] - docker.errors.Co...
ERROR test_core.py::TestDiscovery::test_defined_cursors_exist_in_schema[inputs0]
ERROR test_core.py::TestDiscovery::test_defined_refs_exist_in_schema[inputs0]
ERROR test_core.py::TestDiscovery::test_defined_keyword_exist_in_schema[inputs0-allOf]
ERROR test_core.py::TestDiscovery::test_defined_keyword_exist_in_schema[inputs0-not]
ERROR test_core.py::TestDiscovery::test_primary_keys_exist_in_schema[inputs0]
ERROR test_core.py::TestDiscovery::test_streams_has_sync_modes[inputs0] - doc...
ERROR test_core.py::TestDiscovery::test_additional_properties_is_true[inputs0]
ERROR test_core.py::TestDiscovery::test_backward_compatibility[inputs0] - doc...
ERROR test_core.py::TestBasicRead::test_read[inputs0] - docker.errors.Contain...
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:60: Skipping TestFullRefresh.test_sequential_reads because not found in the config
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:60: Skipping TestIncremental.test_two_sequential_reads because not found in the config
============== 6 failed, 10 passed, 2 skipped, 9 errors in 16.78s ==============

jonathanneo · 2022-08-24T15:25:31Z

Hi @natalyjazzviolin , the build has failed. What are the next steps?

natalyjazzviolin · 2022-08-24T15:34:50Z

@jonathanneo I'm debugging it, should have an update for you soon :)

natalyjazzviolin · 2022-08-24T16:10:50Z

/test connector=connectors/source-exchange-rates

🕑 connectors/source-exchange-rates https://github.com/airbytehq/airbyte/actions/runs/2920630783
❌ connectors/source-exchange-rates https://github.com/airbytehq/airbyte/actions/runs/2920630783
🐛 https://gradle.com/s/lz523souay2fc

Build Failed

Test summary info:

=========================== short test summary info ============================
FAILED test_core.py::TestSpec::test_backward_compatibility[inputs0] - source_...
FAILED test_core.py::TestSpec::test_additional_properties_is_true[inputs0] - ...
FAILED test_core.py::TestConnection::test_check[inputs0] - AssertionError: as...
FAILED test_core.py::TestConnection::test_check[inputs1] - docker.errors.Cont...
FAILED test_core.py::TestBasicRead::test_read[inputs0] - docker.errors.Contai...
ERROR test_core.py::TestDiscovery::test_backward_compatibility[inputs0] - doc...
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:60: Skipping TestFullRefresh.test_sequential_reads because not found in the config
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:60: Skipping TestIncremental.test_two_sequential_reads because not found in the config
============== 5 failed, 19 passed, 2 skipped, 1 error in 14.17s ===============

jonathanneo · 2022-08-30T04:03:23Z

Hi @natalyjazzviolin , any updates?

sajarin · 2022-08-30T15:59:13Z

@jonathanneo the integration tests are breaking on our end since we haven't updated our credentials with a new api key from APILayer. Just made an issue requesting for one to be made so please stay tuned. Appreciate the contribution, hope to get this merged as soon as it's unblocked.

Let me know if you have questions :)

…onalProperties: true in reference to #14196.

natalyjazzviolin · 2022-08-31T22:58:08Z

@jonathanneo our integration tests are also failing due to changes in spec.yaml. Specifically: changing the access_key variable makes the backwards compatibility tests fail (see #136).

Hopefully we can get this tested and merged in soon, thank you so much for the fix!

jonathanneo · 2022-09-01T01:59:09Z

Awesome! Thanks both for testing the change :)

jonathanneo · 2022-09-08T04:25:41Z

Hi @sajarin just following up :)

sajarin · 2022-09-08T18:17:38Z

/test connector=source-exchange-rates

🕑 source-exchange-rates https://github.com/airbytehq/airbyte/actions/runs/3017347155
✅ source-exchange-rates https://github.com/airbytehq/airbyte/actions/runs/3017347155
Python tests coverage:

	 Name                                                 Stmts   Miss  Cover   Missing
	 ----------------------------------------------------------------------------------
	 source_acceptance_test/base.py                          10      4    60%   15-18
	 source_acceptance_test/config.py                        83      6    93%   78-80, 84-86
	 source_acceptance_test/conftest.py                     164    164     0%   6-282
	 source_acceptance_test/plugin.py                        48     48     0%   6-104
	 source_acceptance_test/tests/test_core.py              329    111    66%   39, 50-58, 63-70, 74-75, 79-80, 164, 202-219, 228-236, 240-245, 251, 284-289, 327-334, 374-376, 379, 439-448, 477-478, 484, 487, 520-530, 543-568, 573-577
	 source_acceptance_test/tests/test_full_refresh.py       52      2    96%   34, 65
	 source_acceptance_test/tests/test_incremental.py       121     25    79%   21-23, 29-31, 36-43, 48-61, 208-216
	 source_acceptance_test/utils/asserts.py                 37      2    95%   57-58
	 source_acceptance_test/utils/common.py                  77     17    78%   15-16, 24-30, 47-54, 64, 67
	 source_acceptance_test/utils/compare.py                 62     23    63%   21-51, 68, 97-99
	 source_acceptance_test/utils/connector_runner.py       110     48    56%   23-26, 32, 36, 39-64, 67-69, 72-74, 77-79, 82-84, 87-89, 92-110, 144-146
	 source_acceptance_test/utils/json_schema_helper.py     105     13    88%   30-31, 38, 41, 65-68, 96, 120, 190-192
	 ----------------------------------------------------------------------------------
	 TOTAL                                                 1322    463    65%

Build Passed

Test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:60: Skipping TestFullRefresh.test_sequential_reads because not found in the config
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:60: Skipping TestIncremental.test_two_sequential_reads because not found in the config
======================== 25 passed, 2 skipped in 38.54s ========================

…onathanneo/airbyte into source-exchange-rates-apilayer

docs/integrations/sources/exchangeratesapi.md

marcosmarxm · 2022-09-09T13:34:26Z

/publish connector=source-exchange-rates

🕑 Publishing the following connectors:
source-exchange-rates
https://github.com/airbytehq/airbyte/actions/runs/3022796979

Connector	Did it publish?	Were definitions generated?
source-exchange-rates	❌	❌

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

marcosmarxm · 2022-09-09T13:39:44Z

/publish connector=connectors/source-exchange-rates

🕑 Publishing the following connectors:
connectors/source-exchange-rates
https://github.com/airbytehq/airbyte/actions/runs/3022830369

Connector	Did it publish?	Were definitions generated?
connectors/source-exchange-rates	✅	✅

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

* refactor for api layer * changelog * Reverts variable name for backwards compatability tests & sets additionalProperties: true in reference to airbytehq#14196. * fix: bump dockerfile * fix: update changelog version to match Dockerfile * fix: actually update changelog version to match Dockerfile * auto-bump connector version [ci skip] Co-authored-by: Nataly Merezhuk <65251165+natalyjazzviolin@users.noreply.github.com> Co-authored-by: Sajarin <sajarindider@gmail.com> Co-authored-by: nataly <nataly@airbyte.io> Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>

refactor for api layer

2858324

octavia-squidington-iii added community connectors/source/exchange-rates labels Aug 23, 2022

github-actions bot added the area/connectors Connector related issues label Aug 23, 2022

jonathanneo marked this pull request as ready for review August 23, 2022 14:39

octavia-squidington-iii added the bounty label Aug 23, 2022

This was referenced Aug 23, 2022

Source Exhange Rate API: Invalid API Access key #15524

Closed

Exchange Rates API - set up source ('You have not supplied a valid API Access Key.') #14779

Closed

changelog

205e7eb

github-actions bot added the area/documentation Improvements or additions to documentation label Aug 23, 2022

sajarin added the bounty-L Maintainer program: claimable large bounty PR label Aug 24, 2022

Merge branch 'master' into source-exchange-rates-apilayer

467ec48

Merge branch 'master' into source-exchange-rates-apilayer

ed6864c

sajarin mentioned this pull request Aug 30, 2022

Sandbox Account for Exchange Rates Data API #16133

Closed

sajarin added blocked and removed bounty bounty-L Maintainer program: claimable large bounty PR labels Aug 30, 2022

octavia-squidington-iii added the bounty label Aug 30, 2022

sajarin added internal and removed bounty labels Aug 30, 2022

Reverts variable name for backwards compatability tests & sets additi…

2efce34

…onalProperties: true in reference to #14196.

natalyjazzviolin and others added 2 commits September 1, 2022 10:47

Merge branch 'master' into source-exchange-rates-apilayer

dfdc2e7

Merge branch 'master' into source-exchange-rates-apilayer

28a9c94

Merge branch 'master' into source-exchange-rates-apilayer

9300c76

Merge branch 'master' into source-exchange-rates-apilayer

39ec5ed

sajarin approved these changes Sep 8, 2022

View reviewed changes

sajarin added 2 commits September 8, 2022 15:10

fix: bump dockerfile

e2c22b3

Merge branch 'source-exchange-rates-apilayer' of https://github.com/j…

ac8346c

…onathanneo/airbyte into source-exchange-rates-apilayer

marcosmarxm requested changes Sep 8, 2022

View reviewed changes

docs/integrations/sources/exchangeratesapi.md Outdated Show resolved Hide resolved

sajarin added 3 commits September 8, 2022 15:26

fix: update changelog version to match Dockerfile

a6b253c

fix: actually update changelog version to match Dockerfile

2fa5fb7

Merge branch 'master' into source-exchange-rates-apilayer

2460a8b

auto-bump connector version [ci skip]

81bcdd5

marcosmarxm approved these changes Sep 9, 2022

View reviewed changes

marcosmarxm merged commit 6b4c523 into airbytehq:master Sep 9, 2022

octavia-squidington-iii mentioned this pull request Sep 12, 2022

Bump Airbyte version from 0.40.4 to 0.40.5 #16610

Merged

Ezziddeen-Abdulwahhab mentioned this pull request Dec 10, 2022

Exchange rates source description needs update #20340

Closed

mr-W3dnesday mentioned this pull request Oct 18, 2023

Update exchange-rates.md #31566

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Source Exchange Rates - Refactor for API Layer #15884

Source Exchange Rates - Refactor for API Layer #15884

jonathanneo commented Aug 23, 2022 •

edited

Loading

CLAassistant commented Aug 23, 2022 •

edited

Loading

jonathanneo commented Aug 23, 2022

natalyjazzviolin commented Aug 24, 2022 •

edited by github-actions bot

Loading

jonathanneo commented Aug 24, 2022

natalyjazzviolin commented Aug 24, 2022

natalyjazzviolin commented Aug 24, 2022 •

edited by github-actions bot

Loading

jonathanneo commented Aug 30, 2022

sajarin commented Aug 30, 2022

natalyjazzviolin commented Aug 31, 2022

jonathanneo commented Sep 1, 2022

jonathanneo commented Sep 8, 2022

sajarin commented Sep 8, 2022 •

edited by github-actions bot

Loading

marcosmarxm commented Sep 9, 2022 •

edited by github-actions bot

Loading

marcosmarxm commented Sep 9, 2022 •

edited by github-actions bot

Loading

Source Exchange Rates - Refactor for API Layer #15884

Source Exchange Rates - Refactor for API Layer #15884

Conversation

jonathanneo commented Aug 23, 2022 • edited Loading

What

How

Recommended reading order

🚨 User Impact 🚨

Pre-merge Checklist

Community member or Airbyter

Airbyter

Tests

CLAassistant commented Aug 23, 2022 • edited Loading

jonathanneo commented Aug 23, 2022

natalyjazzviolin commented Aug 24, 2022 • edited by github-actions bot Loading

Build Failed

jonathanneo commented Aug 24, 2022

natalyjazzviolin commented Aug 24, 2022

natalyjazzviolin commented Aug 24, 2022 • edited by github-actions bot Loading

Build Failed

jonathanneo commented Aug 30, 2022

sajarin commented Aug 30, 2022

natalyjazzviolin commented Aug 31, 2022

jonathanneo commented Sep 1, 2022

jonathanneo commented Sep 8, 2022

sajarin commented Sep 8, 2022 • edited by github-actions bot Loading

Build Passed

marcosmarxm commented Sep 9, 2022 • edited by github-actions bot Loading

marcosmarxm commented Sep 9, 2022 • edited by github-actions bot Loading

jonathanneo commented Aug 23, 2022 •

edited

Loading

CLAassistant commented Aug 23, 2022 •

edited

Loading

natalyjazzviolin commented Aug 24, 2022 •

edited by github-actions bot

Loading

natalyjazzviolin commented Aug 24, 2022 •

edited by github-actions bot

Loading

sajarin commented Sep 8, 2022 •

edited by github-actions bot

Loading

marcosmarxm commented Sep 9, 2022 •

edited by github-actions bot

Loading

marcosmarxm commented Sep 9, 2022 •

edited by github-actions bot

Loading