🐛 Source Salesforce: Write binary file without decoding #16684

Nakachi-S · 2022-09-14T00:48:45Z

What

#15950
#14659

https://discuss.airbyte.io/t/problem-in-salesforce-chunk-decoding/1757

How

In #11692 PR, the download_data method implements saving to a temporary file by chunk size.

I propose a way to handle multibyte strings by saving the data as a binary file instead of decoding by chunk size.

🚨 User Impact 🚨

No impact expected.

Pre-merge Checklist

Expand the relevant checklist and delete the others.

Updating a connector

Community member or Airbyter

Grant edit access to maintainers (instructions)
Secrets in the connector's spec are annotated with airbyte_secret
Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
Code reviews completed
Documentation updated
- Connector's README.md
- Connector's bootstrap.md. See description and examples
- Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.
/test connector=connectors/<name> command is passing
New Connector version released on Dockerhub and connector version bumped by running the /publish command described here

Tests

Unit

Integration

Put your integration tests output here.

Acceptance

Put your acceptance tests output here.

sajarin · 2022-09-14T17:39:04Z

/test connector=connectors/source-salesforce

🕑 connectors/source-salesforce https://github.com/airbytehq/airbyte/actions/runs/3054866558
✅ connectors/source-salesforce https://github.com/airbytehq/airbyte/actions/runs/3054866558
Python tests coverage:

Name                                 Stmts   Miss  Cover
--------------------------------------------------------
source_salesforce/__init__.py            2      0   100%
source_salesforce/exceptions.py          8      1    88%
source_salesforce/api.py               151     19    87%
source_salesforce/streams.py           307     71    77%
source_salesforce/rate_limiting.py      22      6    73%
source_salesforce/source.py             91     35    62%
source_salesforce/utils.py               8      7    12%
--------------------------------------------------------
TOTAL                                  589    139    76%
Name                                 Stmts   Miss  Cover
--------------------------------------------------------
source_salesforce/utils.py               8      0   100%
source_salesforce/__init__.py            2      0   100%
source_salesforce/source.py             91      8    91%
source_salesforce/api.py               151     14    91%
source_salesforce/exceptions.py          8      1    88%
source_salesforce/rate_limiting.py      22      3    86%
source_salesforce/streams.py           307     42    86%
--------------------------------------------------------
TOTAL                                  589     68    88%
	 Name                                                 Stmts   Miss  Cover   Missing
	 ----------------------------------------------------------------------------------
	 source_acceptance_test/base.py                          10      4    60%   15-18
	 source_acceptance_test/config.py                        83      6    93%   78-80, 84-86
	 source_acceptance_test/conftest.py                     164    164     0%   6-282
	 source_acceptance_test/plugin.py                        48     48     0%   6-104
	 source_acceptance_test/tests/test_core.py              329    111    66%   39, 50-58, 63-70, 74-75, 79-80, 164, 202-219, 228-236, 240-245, 251, 284-289, 327-334, 374-376, 379, 439-448, 477-478, 484, 487, 520-530, 543-568, 573-577
	 source_acceptance_test/tests/test_full_refresh.py       52      2    96%   34, 65
	 source_acceptance_test/tests/test_incremental.py       121     25    79%   21-23, 29-31, 36-43, 48-61, 208-216
	 source_acceptance_test/utils/asserts.py                 37      2    95%   57-58
	 source_acceptance_test/utils/common.py                  77     17    78%   15-16, 24-30, 47-54, 64, 67
	 source_acceptance_test/utils/compare.py                 62     23    63%   21-51, 68, 97-99
	 source_acceptance_test/utils/connector_runner.py       110     48    56%   23-26, 32, 36, 39-64, 67-69, 72-74, 77-79, 82-84, 87-89, 92-110, 144-146
	 source_acceptance_test/utils/json_schema_helper.py     105     13    88%   30-31, 38, 41, 65-68, 96, 120, 190-192
	 ----------------------------------------------------------------------------------
	 TOTAL                                                 1325    463    65%

Build Passed

Test summary info:

All Passed

Nakachi-S · 2022-09-15T11:58:52Z

@sajarin
Thanks for running the tests!

@grishick @marcosmarxm @tuliren
Could you review this PR?

sajarin · 2022-09-19T14:50:23Z

assigned to @koconder

vincentkoc

Hi @Nakachi-S I'll be taking a look at this PR for you

vincentkoc

@Nakachi-S few quick things:

Have you tested this directly yourself? The unit tests have no encoding or read/write unit tests, not sure if you are able to write a unit test on the read_with_chunks
See feedback on how chunks are written, based on the code i believe only the last chunk will be written
You need to update the changelog in docs/integrations//.md including changelog. See changelog example
You need to bump the docker image version for your change

airbyte/airbyte-integrations/connectors/source-salesforce/Dockerfile

Line 16 in dc328ca

LABEL io.airbyte.version=1.0.15

using semantic versioning, so this change would be 1.0.16

vincentkoc · 2022-09-19T20:43:58Z

airbyte-integrations/connectors/source-salesforce/source_salesforce/streams.py

                for chunk in response.iter_content(chunk_size=chunk_size):
-                    data_file.writelines(self.filter_null_bytes(self.decode(chunk)))
+                    data_file.write(self.filter_null_bytes(chunk))


@Nakachi-S just worried that since the file was opened as wb and not ab the function writelines() that now needs to use write() which is what’s required for bytes might overwrite the file at each chunk. Other than that everything looks OK here for this particular file

@koconder
Thanks your review.

A temporary file is created for each URL, and chunks are continuously written to the file.
Therefore, it is not necessary to use "aw," which is an append mode.

Also about

See feedback on how chunks are written, based on the code i believe only the last chunk will be written

if you write "write" in the with clause as shown below, the result is not overwriting the file, but appending to it.

with open("tmp.txt", "wb") as f: f.wirte("Hello\n") f.write("World.)

this result is

Hello World

Nakachi-S · 2022-09-20T11:48:30Z

@koconder

Have you tested this directly yourself?

Yes, I tested this method and there were no problems.

You need to update the changelog in docs/integrations//.md including changelog. See changelog example
You need to bump the docker image version for your change

I bump docker image version and updated document.
Also, I have given the version of the seed file as shown in the following commit, is this necessary?
93c66e8

vincentkoc · 2022-09-20T12:11:05Z

@Nakachi-S please revert the seed file changes, only keep to changes in the connector folder

This reverts commit 93c66e8.

Nakachi-S · 2022-09-21T00:58:47Z

@koconder
I have reverted the seed file changes.

airbyte-config/init/src/main/resources/seed/source_specs.yaml

airbyte-config/init/src/main/resources/seed/source_definitions.yaml

vincentkoc · 2022-09-21T04:10:33Z

PR approved, pending release.

sajarin · 2022-09-21T15:09:18Z

/test connector=connectors/source-salesforce

🕑 connectors/source-salesforce https://github.com/airbytehq/airbyte/actions/runs/3098949493
❌ connectors/source-salesforce https://github.com/airbytehq/airbyte/actions/runs/3098949493
🐛 https://gradle.com/s/xq33jx6uszdvs

Build Failed

Test summary info:

	 =========================== short test summary info ============================
	 FAILED unit_tests/api_test.py::test_rate_limit_bulk - TypeError: _read_stream...
	 FAILED unit_tests/api_test.py::test_rate_limit_rest - TypeError: _read_stream...
	 �[31m============ �[31m�[1m2 failed�[0m, �[32m50 passed�[0m, �[33m153 warnings�[0m�[31m in 94.35s (0:01:34)�[0m�[31m =============�[0m

vincentkoc · 2022-09-22T11:49:36Z

@Nakachi-S please resolve conflicts, changes made in #17001

sajarin · 2022-09-23T19:17:15Z

Hey @Nakachi-S.

Thank you for making this PR. One of our engineers has pushed a change that should fix the underlying issue; #17001. We look forward to more PRs in the future!

🐛 Write binary file without decoding.

65e518a

github-actions bot added the area/connectors Connector related issues label Sep 14, 2022

octavia-squidington-iii added bounty community connectors/source/salesforce labels Sep 14, 2022

Nakachi-S changed the title ~~🐛 Write binary file without decoding.~~ 🐛 Write binary file without decoding on Salesforce Sep 14, 2022

Merge branch 'master' into 15950-modify-decode

a4be27c

Nakachi-S marked this pull request as draft September 14, 2022 01:24

Nakachi-S changed the title ~~🐛 Write binary file without decoding on Salesforce~~ 🐛 Source Salesforce: Write binary file without decoding Sep 14, 2022

Nakachi-S marked this pull request as ready for review September 14, 2022 02:03

sajarin added the bounty-S Maintainer program: claimable small bounty PR label Sep 19, 2022

vincentkoc reviewed Sep 19, 2022

View reviewed changes

vincentkoc suggested changes Sep 19, 2022

View reviewed changes

Nakachi-S and others added 3 commits September 20, 2022 15:24

Merge branch 'master' into 15950-modify-decode

cb257ab

⬆️ Bump docker image version. Update docs.

48cd425

⬆️ Bump of seed yaml.

93c66e8

github-actions bot added the area/documentation Improvements or additions to documentation label Sep 20, 2022

Revert ":arrow_up: Bump of seed yaml."

d42ce9c

This reverts commit 93c66e8.

vincentkoc approved these changes Sep 21, 2022

View reviewed changes

airbyte-config/init/src/main/resources/seed/source_specs.yaml Outdated Show resolved Hide resolved

airbyte-config/init/src/main/resources/seed/source_definitions.yaml Outdated Show resolved Hide resolved

sajarin changed the title ~~🐛 Source Salesforce: Write binary file without decoding~~ 🐛 Source Salesforce: Write binary file without decoding Sep 21, 2022

davydov-d mentioned this pull request Sep 21, 2022

Source salesforce: handle japanese characters #17001

Merged

sajarin requested review from sherifnada and removed request for sherifnada September 23, 2022 15:46

sajarin closed this Sep 23, 2022

bleonard added the reward-50 label Oct 31, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🐛 Source Salesforce: Write binary file without decoding #16684

🐛 Source Salesforce: Write binary file without decoding #16684

Nakachi-S commented Sep 14, 2022 •

edited

Loading

sajarin commented Sep 14, 2022 •

edited by github-actions bot

Loading

Nakachi-S commented Sep 15, 2022

sajarin commented Sep 19, 2022

vincentkoc left a comment

vincentkoc left a comment

vincentkoc Sep 19, 2022

Nakachi-S Sep 20, 2022

Nakachi-S commented Sep 20, 2022

vincentkoc commented Sep 20, 2022

Nakachi-S commented Sep 21, 2022

vincentkoc commented Sep 21, 2022

sajarin commented Sep 21, 2022 •

edited by github-actions bot

Loading

vincentkoc commented Sep 22, 2022

sajarin commented Sep 23, 2022

🐛 Source Salesforce: Write binary file without decoding #16684

🐛 Source Salesforce: Write binary file without decoding #16684

Conversation

Nakachi-S commented Sep 14, 2022 • edited Loading

What

How

Recommended reading order

🚨 User Impact 🚨

Pre-merge Checklist

Community member or Airbyter

Airbyter

Tests

sajarin commented Sep 14, 2022 • edited by github-actions bot Loading

Build Passed

Nakachi-S commented Sep 15, 2022

sajarin commented Sep 19, 2022

vincentkoc left a comment

Choose a reason for hiding this comment

vincentkoc left a comment

Choose a reason for hiding this comment

vincentkoc Sep 19, 2022

Choose a reason for hiding this comment

Nakachi-S Sep 20, 2022

Choose a reason for hiding this comment

Nakachi-S commented Sep 20, 2022

vincentkoc commented Sep 20, 2022

Nakachi-S commented Sep 21, 2022

vincentkoc commented Sep 21, 2022

sajarin commented Sep 21, 2022 • edited by github-actions bot Loading

Build Failed

vincentkoc commented Sep 22, 2022

sajarin commented Sep 23, 2022

Nakachi-S commented Sep 14, 2022 •

edited

Loading

sajarin commented Sep 14, 2022 •

edited by github-actions bot

Loading

sajarin commented Sep 21, 2022 •

edited by github-actions bot

Loading