Read data from a BigQuery table from C++ code with minimum dependencies.

[tzovas]

My understanding is that, when reading data from a BQ table, they are serialized using either the AVRO or the ARROW format. In C++, in at least one example that I found online, people use the C++ avro library to parse rows from a table. There is also the equivalent arrow library. Is there a way to read/parse data from a BQ table without using either of those libraries? They have heavy dependencies (like boost) and I will like to avoid installing boost just to read from a table. Is there maybe some lightweight library? Does google offer another format?

[coryan]

Does google offer another format?

Yes, but we have not implemented a C++ SDK for it. You can read data using the REST API for BigQuery:

https://cloud.google.com/bigquery/docs/reference/rest/v2/tabledata/list

We expect that this is fairly slow for large data sets. The service has to format the data as JSON and then your application (or a future C++ SDK) would need to parse it. Reading data in pages also impacts throughput, as each page is a new HTTP request.

Is there a way to read/parse data from a BQ table without using either of those libraries?

When using the BigQuery Storage API, those are the only possible download formats:

https://github.com/googleapis/googleapis/blob/01ebb1901fae079b01d0a41035d01fa52035391c/google/cloud/bigquery/storage/v1/stream.proto#L34-L45

https://github.com/googleapis/googleapis/blob/01ebb1901fae079b01d0a41035d01fa52035391c/google/cloud/bigquery/storage/v1/stream.proto#L154-L156

[tzovas]

Thanks a lot @coryan.
Using curl, and the curl library, I think I will be able to stitch something together to make the appropriate call. Trying this from command line, it works. For future reference, the command that worked was

curl -X GET \
    -H "Authorization: Bearer $(gcloud auth print-access-token)" \
    https://bigquery.googleapis.com/bigquery/v2/projects/{projectId}/datasets/{datasetId}/tables/{tableId}/data;maxResults=10

The response does not contain the column name though (is this the schema?), everything is named "v" as mentioned in https://cloud.google.com/bigquery/docs/reference/rest/v2/tabledata/list#response-body.
Is there a way to retrieve the column names?

[coryan]

With the caveat that I am not an expert in BigQuery, I think this is what you are looking for:

https://cloud.google.com/bigquery/docs/reference/rest/v2/tables/get

For example:

curl -X GET     -H "Authorization: Bearer $(gcloud auth print-access-token)"  \ 
    'https://bigquery.googleapis.com/bigquery/v2/projects/bigquery-public-data/datasets/samples/tables/shakespeare'

[tzovas]

Just a follow up. The curl approach seemed to work. But in the end, I decided to use the Avro C library that has much lighter dependencies. Thanks for your help.