Support for Google BigQuery

[n-riesco]

From https://community.plot.ly/t/support-for-google-bigquery/8894

---

This is a good first issue, since @google-cloud/bigquery provides all the needed functionality:

• query
• list of all datasets
• list of all tables
• column types

[dkapitan]

Have never contributed to js projects, but since the feature request is mine I am willing to help wherever I can. Looking at the codebase of falcon, it lookst like a new bigquery.js persistent datastore is needed. That means wrapping Bigquery node.js client libraries.

Is that right?

[shannonlal]

@dkapitan I can help you with this. I did the Athena connector and can walk you through the process. @n-riesco helped me a lot with setting this up. I would clone the project and look at the Contributing.md and New_connection.md as good starting points. Feel free to ask questions.

[n-riesco]

@dkapitan As @shannonlal says, the best place to start is https://github.com/plotly/falcon-sql-client/blob/master/NEW_CONNECTION.md

If you get stuck, at any point, just, give us a shout.

And to begin simple, I'd require that users have to set the dataset in the connection form (we can drop this requirement later).

[dkapitan]

@shannonlal and @n-riesco
Off topic, but to deepen my understanding: the Sequelize library is used to create the connections. I need to interface with MS SQL via ODBC as well, which would require integrating sequelize-odbc-mssql as a new dialect.

Is this correct?

[n-riesco]

@dkapitan

the Sequelize library is used to create the connections.

only some of the connectors use Sequelize

I need to interface with MS SQL via ODBC as well, which would require integrating sequelize-odbc-mssql as a new dialect.

Is this correct?

I'm not familiar with sequelize-odbc-mssql. The main difficulties I see:

• it requires an ODBC driver (see configuration of dialectOptions.driver)
• and it's only for windows.

If you want to experiment with sequelize-odbc-mssql, you'd need to edit these lines in Sql.js.

And I guess, you'll also have to add sequelize-odbc-mssql and @ratanakvlun/node-odbc to webpack.config.base.js.

[shannonlal]

@dkapitan I think the Plot.ly team was swamped the last couple of weeks fixing some bugs and getting the latest Falcon release working. I am looking at using Big Query at work soon so I could make sometime to help with you this. For Big Query I don't think it makes sense to use sequelize and I would look at using the https://www.npmjs.com/package/@google-cloud/bigquery project. For Athena, we already had the AWS SDK and I just wrote the Athena wrapper for this. If your still available you can ping me directly and I can give you a lot of advice on this. @n-riesco helped me a lot to get my first Connector up and running and I would be more then willing to help you.

[dkapitan]

@shannonlal thanks for your help. I am currently swamped myself, and in the process of implementing a new data science platform on Google Cloud. So I don't expect to have time in the coming months. Maybe over summer I could start this, when things are a bit less hectic at work.

[n-riesco]

@shannonlal has started to look into implementing this connector. Unfortunately, there are a few difficulties to overcome:

• webpack fails to bundle @google-cloud/bigquery (see issues webpack & google-cloud googleapis/google-cloud-node#1821 and What should the story be for a client-side Javascript gcloud library? googleapis/google-cloud-node#531)
• and more worryingly (judging by the error logs in webpack & google-cloud googleapis/google-cloud-node#1821), @google-cloud/bigquery depends on native modules that use node-pre-gyp.

[sumedhsakdeo]

Any updates on this integration? What are the rough timelines, when we can expect google bigquery integration with plotly?

[shannonlal]

@sumedhsakdeo I am only about 10 hours away from getting this working just need to finish this off. I was having a problem with using the official npm for bigquery but @n-riesco helped me resolve it. I have the UI and stubbed code for backend just need to get everything completed and tested. Shouldn't be too long

[ramiouanes]

Any news about this?