Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Connectors: Dedicated forms for OLAP engines #4934

Merged
merged 39 commits into from
Jun 3, 2024
Merged

Connectors: Dedicated forms for OLAP engines #4934

merged 39 commits into from
Jun 3, 2024

Conversation

ericpgreen2
Copy link
Contributor

@ericpgreen2 ericpgreen2 commented May 21, 2024 •
edited
Loading

This PR provides an in-app guided flow for connecting to external OLAP engines (currently: ClickHouse, Druid, DuckDB, and Pinot).

Specifically, this PR:

  • Reframes the "Add source" modal as the "Add data" modal, and splits it into two sections: Sources and OLAP engines
  • Adds forms for connecting to ClickHouse, Druid, and Pinot
  • Compiles the OLAP engine forms into connectors/<connector-name>.yaml files
  • Writes any secrets provided in the form to the project's .env file
  • Sets the olap_connector key in the rill.yaml file to the newly configured connector

Future PRs will:

image

image

image

image

image

@ericpgreen2 ericpgreen2 self-assigned this May 21, 2024
@nishantmonu51
Copy link
Collaborator

For CH form, instead of DSN, we would like to break it up in separate inputs for the user..

@nishantmonu51 nishantmonu51 added the blocker A release blocker issue that should be resolved before a new release label May 30, 2024
@ericpgreen2 ericpgreen2 marked this pull request as ready for review June 1, 2024 05:15
@ericpgreen2 ericpgreen2 merged commit bf22972 into main Jun 3, 2024
6 checks passed
@ericpgreen2 ericpgreen2 deleted the olap-connector-forms branch June 3, 2024 16:43
ericpgreen2 added a commit that referenced this pull request Jun 3, 2024
@ericpgreen2 @esevastyanov
@briangregoryholmes @nishantmonu51
Connectors: Dedicated forms for OLAP engines (#4934)
4f8f91f 
* Split out an OLAP section in the "Add data" modal

* Add basic ClickHouse and Pinot forms

* Remove SQL from DuckDB form

* Clean up header styles

* Remove code for CLI instructions

* Dedicated connectors

* Tests

* Removed unused functions

* empty rill.yaml

* Tests and corresponding fixes

* Connector analyzer should not fail in case of invalid connector

* Write connector secrets to `.env`

* Update `rill.yaml` with the name of the new connector

* Mark Pinot's DSN as a secret

* Add Druid form

* Disable autosave for the `connectors` directory

* Add Druid form (cont.)

* Refactor & use query cache

* UpdateInstanceWithRillYAML on connector reconcile

* Pass through `queryClient`

* Add terminology for a reconciling connector

* Fix perpetual reconciling

* Fix type warning

* flip bool

* Add Connector icon

* Consolidate unit tests

* Reference secrets in `.env` file

* Revert "UpdateInstanceWithRillYAML on connector reconcile"

This reverts commit 47072bc.

---------

Co-authored-by: e.sevastyanov <eugene.sevastianov@rilldata.com>
Co-authored-by: Brian Holmes <briangregoryholmes@gmail.com>
Co-authored-by: Nishant Bangarwa <nishant.monu51@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@nishantmonu51 nishantmonu51 nishantmonu51 approved these changes

Assignees

@ericpgreen2 ericpgreen2

Labels
blocker A release blocker issue that should be resolved before a new release
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@ericpgreen2 @nishantmonu51 @esevastyanov @briangregoryholmes