Need a sample target_bigquery.yml

[anbarasuvyn]

Hi all,

Could you please share a version of the target_bigquery.yml that is working?

I am trying to tap-github into target-bigquery using pipelinewise.

Thanks in advance.

[jmriego]

no problem!
The better place is to check the PipelineWise documentation for that: https://transferwise.github.io/pipelinewise/connectors/targets/bigquery.html

The difference is that this repo explains how to use this target outside of PipelineWise, but definitely using this yaml is easier.
Hope this helps!

[anbarasuvyn]

I would be happy to use this repo.

In that case can you share a sample config.json which is working. I am getting a schema load error with my config.json.

[jmriego]

You should write s JSON file that has a dictionary with at least the project config specified in the README

{
  "project_id": "mygbqproject"
}

That error sounds to me like you might be missing a comma, quote or similar. Maybe you can use a JSON online checker or something your text editor includes?

[anbarasuvyn]

My json is this

{
"project_id": "ddata",
"dataset_id": "batch16",
"table_id": "commits",
"default_target_schema": true,
"validate_records": true
}

And I run this command to pipeline from tap-github using the command

tap-github --config config_github.json --properties properties.json | target-bigquery -c config_bigquery.json config_bigquery.json

And the error is:

INFO Starting sync of repository: akx/walletool
INFO METRIC: {"type": "timer", "metric": "http_request_duration", "value": 0.5208690166473389, "tags": {"endpoint": "commits", "http_status_code": 200, "status": "succeeded"}}
INFO METRIC: {"type": "counter", "metric": "record_count", "value": 8, "tags": {"endpoint": "commits"}}
INFO Starting sync of repository: alexbosworth/lightning
INFO Sending version information to stitchdata.com. To disable sending anonymous usage data, set the config parameter "disable_collection" to true
INFO METRIC: {"type": "timer", "metric": "http_request_duration", "value": 0.4820370674133301, "tags": {"endpoint": "commits", "http_status_code": 200, "status": "succeeded"}}
INFO METRIC: {"type": "timer", "metric": "http_request_duration", "value": 0.4103388786315918, "tags": {"endpoint": "commits", "http_status_code": 200, "status": "succeeded"}}
{'type': ['string']}
Traceback (most recent call last):
File "/Users/anbarasu/Library/Python/3.8/bin/target-bigquery", line 8, in
sys.exit(main())
File "/Users/anbarasu/Library/Python/3.8/lib/python/site-packages/target_bigquery.py", line 301, in main
state = persist_lines_stream(config['project_id'], config['dataset_id'], input, validate_records=validate_records)

File "/Users/anbarasu/Library/Python/3.8/lib/python/site-packages/target_bigquery.py", line 240, in persist_lines_stream
tables[table] = bigquery.Table(dataset.table(table), schema=build_schema(schemas[table]))
File "/Users/anbarasu/Library/Python/3.8/lib/python/site-packages/target_bigquery.py", line 101, in build_schema
schema_name, schema_type, schema_mode, schema_description, schema_fields = define_schema(schema['properties'][key], key)
File "/Users/anbarasu/Library/Python/3.8/lib/python/site-packages/target_bigquery.py", line 74, in define_schema
schema_type = field['type'][1]
IndexError: list index out of range
INFO METRIC: {"type": "counter", "metric": "record_count", "value": 38, "tags": {"endpoint": "commits"}}
CRITICAL [Errno 32] Broken pipe
Traceback (most recent call last):
File "/Users/anbarasu/Library/Python/3.8/bin/tap-github", line 8, in
sys.exit(main())
File "/Users/anbarasu/Library/Python/3.8/lib/python/site-packages/singer/utils.py", line 229, in wrapped
return fnc(*args, **kwargs)
File "/Users/anbarasu/Library/Python/3.8/lib/python/site-packages/tap_github/init.py", line 1190, in main
do_sync(args.config, args.state, catalog)
File "/Users/anbarasu/Library/Python/3.8/lib/python/site-packages/tap_github/init.py", line 1154, in do_sync
state = sync_func(stream_schema, repo, state, mdata, start_date)
File "/Users/anbarasu/Library/Python/3.8/lib/python/site-packages/tap_github/init.py", line 970, in get_all_commits
singer.write_record('commits', rec, time_extracted=extraction_time)
File "/Users/anbarasu/Library/Python/3.8/lib/python/site-packages/singer/messages.py", line 226, in write_record
write_message(RecordMessage(stream=(stream_alias or stream_name),
File "/Users/anbarasu/Library/Python/3.8/lib/python/site-packages/singer/messages.py", line 218, in write_message
sys.stdout.flush()
BrokenPipeError: [Errno 32] Broken pipe
Exception ignored in: <_io.TextIOWrapper name='' mode='w' encoding='utf-8'>
BrokenPipeError: [Errno 32] Broken pipe

Should I change anything in the config file?Will bigquery use the same schema if the schema is not mentioned explicitly?

[jmriego]

That actually looks like an issue in either the tap not following the singer standard or the target.

Do you mind sending the results of tap-github --config config_github.json --properties properties.json?
I think the rows of SCHEMA type might have an schema this target is not accounting for. What repo are you using for the GitHub tap?

[anbarasuvyn]

results of tap-github --config config_github.json --properties properties.json? is

{"type": "STATE", "value": {}}
{"type": "SCHEMA", "stream": "commits", "schema": {"selected": true, "type": ["null", "object"], "additionalProperties": false, "properties": {"_sdc_repository": {"type": ["string"]}, "sha": {"type": ["null", "string"], "description": "The git commit hash"}, "url": {"type": ["null", "string"]}, "parents": {"type": ["null", "array"], "items": {"type": ["null", "object"], "additionalProperties": false, "properties": {"sha": {"type": ["null", "string"], "description": "The git hash of the parent commit"}, "url": {"type": ["null", "string"], "description": "The URL to the parent commit"}, "html_url": {"type": ["null", "string"], "description": "The HTML URL to the parent commit"}}}}, "files": {"type": ["null", "array"], "items": {"type": ["null", "object"], "properties": {"filename": {"type": ["null", "string"]}, "additions": {"type": ["null", "number"]}, "deletions": {"type": ["null", "number"]}, "changes": {"type": ["null", "number"]}, "status": {"type": ["null", "string"]}, "raw_url": {"type": ["null", "string"]}, "blob_url": {"type": ["null", "string"]}, "patch": {"type": ["null", "string"]}}}}, "html_url": {"type": ["null", "string"], "description": "The HTML URL to the commit"}, "comments_url": {"type": ["null", "string"], "description": "The URL to the commit's comments page"}, "commit": {"type": ["null", "object"], "additionalProperties": false, "properties": {"url": {"type": ["null", "string"], "description": "The URL to the commit"}, "tree": {"type": ["null", "object"], "additionalProperties": false, "properties": {"sha": {"type": ["null", "string"]}, "url": {"type": ["null", "string"]}}}, "author": {"type": ["null", "object"], "additionalProperties": false, "properties": {"date": {"type": ["null", "string"], "format": "date-time", "description": "The date the author committed the change"}, "name": {"type": ["null", "string"], "description": "The author's name"}, "email": {"type": ["null", "string"], "description": "The author's email"}, "login": {"type": ["null", "string"], "description": "The author's login"}}}, "message": {"type": ["null", "string"], "description": "The commit message"}, "committer": {"type": ["null", "object"], "additionalProperties": false, "properties": {"date": {"type": ["null", "string"], "format": "date-time", "description": "The date the committer committed the change"}, "name": {"type": ["null", "string"], "description": "The committer's name"}, "email": {"type": ["null", "string"], "description": "The committer's email"}, "login": {"type": ["null", "string"], "description": "The committer's login"}}}, "comment_count": {"type": ["null", "integer"], "description": "The number of comments on the commit"}}}, "pr_number": {"type": ["null", "integer"]}, "pr_id": {"type": ["null", "string"]}, "id": {"type": ["null", "string"]}}}, "key_properties": ["sha"]}
{"type": "STATE", "value": {}}

I am using https://github.com/singer-io/tap-github.

[jmriego]

hi @anbarasuvyn , sorry, there must be some other issue here. I just copied that text to a file and piped it to this target:

cat tapgit.txt | target-bigquery --config config.json
time=2022-03-11 16:55:08 name=singer level=INFO message=Table 'jvalenzuela.commits' does not exist. Creating...
time=2022-03-11 16:55:08 name=target_bigquery level=INFO message=Emitting state {}
{}

It seems to be working correctly

Is that the same test you did? which version of the target are you using? Anything you can do to help me reproduce this would help