Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Transformer dumps JSON incompatible string #136

Open
daigotanaka opened this issue Jan 6, 2021 · 0 comments
Open

Transformer dumps JSON incompatible string #136

daigotanaka opened this issue Jan 6, 2021 · 0 comments

Comments

@daigotanaka
Copy link

daigotanaka commented Jan 6, 2021

When Transformer recognizes the type to be str, it will convert the (sub) object to str type. The issue is, if such (sub) object's original type is dict, the current method of converting to str produces JSON incompatible string:

return True, str(data)

This results in the conversion from a dict

{'active': True, 'note': None}

to

"{'active': True, 'note': None}"

instead of

'{"active": true, "note": null}'

str(data) conversion seems to produce problems with escape characters as well.

I am wondering if it is acceptable to replace str(data) with json.dumps(data)

One may argue that the tap should fully specify the schema so that the (sub)object is written out as dict. However, many of the Rest API often includes the field whose schema is not static.
An example is Github API's event object. event.payload is a (dict) object, but the schema depends on the event type.
https://docs.github.com/en/free-pro-team@latest/developers/webhooks-and-events/github-event-types#event-object-common-properties
In fact, I discovered this issue while I was debugging tap-github's usage of Transformer:
https://github.com/singer-io/tap-github/blob/master/tap_github/__init__.py#L361

If the str conversion was done through json.dumps, it would have been possible to parse JSON in the target datastore such as BigQuery and Redshift.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant