Adding Apache Pinot Query Runner #5798

xiangfu0 · 2022-07-18T23:50:37Z

What type of PR is this?

Description

Adding Apache Pinot query runner.

How is this tested?

Unit tests (pytest, jest)
E2E Tests (Cypress)
Manually
N/A

1. Start redash docker-compose

docker-compose up -d && docker-compose run --rm server create_db && docker-compose run --rm postgres psql -h postgres -U postgres -c "create database tests"

2. Start Pinot Quickstart

docker run \
    --name pinot-quickstart \
    --network redash_default \
    -p 9000:9000 -d \
    apachepinot/pinot:latest QuickStart \
    -type batch

This docker run create a Pinot cluster with controller at http://pinot-quickstart:9000 and broker athttp://pinot-quickstart:8000

3. Create data source:

4. Query Builder

Related Tickets & Documents

Mobile & Desktop Screenshots/Recordings (if there are UI changes)

xiangfu0 · 2022-07-19T00:08:28Z

Just realized the duplicated effort #5724

susodapop · 2022-07-19T00:24:55Z

I'm not seeing any movement on #5724 from that contributor. So I'm happy to move forward with this one assuming we can incorporate the feedback from the two previous attempts. The big question was around data type mapping, as I recall.

xiangfu0 · 2022-07-19T08:35:02Z

Adding more type support:

xiangfu0 · 2022-07-19T08:37:04Z

Also tested with auth-zk pinot quickstart:

docker run --name pinot-quickstart --network redash_default \
    -p 9000:9000 -p 8000:8000  -d\
    apachepinot/pinot:latest QuickStart \
    -type AUTH-ZK

With Data Source

susodapop · 2022-07-19T19:08:00Z

Please let me know when you feel this is ready for review 👌

xiangfu0 · 2022-07-19T20:29:43Z

Please let me know when you feel this is ready for review 👌

Yes, please. Logic wise it's pretty much there.

xiangfu0 · 2022-07-19T20:31:21Z

I've tried to capture the feedbacks for type mapping from #5724 and #5446

xiangfu0 · 2022-08-01T19:15:05Z

@susodapop can you start reviewing this PR, thanks!

susodapop · 2022-08-01T19:17:03Z

Thanks for the ping! I will review it this week.

xiangfu0 · 2022-08-17T23:34:05Z

Thanks for the ping! I will review it this week.

Kindly bump up the thread :p

xiangfu0 · 2022-08-23T21:53:47Z

Guess I forgot to @susodapop last week :p

susodapop

Thank you so much for your patience. This looks great overall! Please see my comments and let me know if you have questions.

requirements_all_ds.txt

redash/query_runner/pinot.py

susodapop · 2022-08-24T13:39:05Z

redash/query_runner/pinot.py

+        for schema_name in self.get_schema_names():
+            for table_name in self.get_table_names():
+                schema_table_name = "{}.{}".format(schema_name, table_name)
+                if table_name not in schema:
+                    schema[schema_table_name] = {"name": schema_table_name, "columns": []}
+                table_schema =self.get_pinot_table_schema(table_name)
+
+                for column in table_schema.get("dimensionFieldSpecs", []) + table_schema.get(
+                    "metricFieldSpecs", []) + table_schema.get("dateTimeFieldSpecs", []):
+                    c = {
+                            "name": column["name"],
+                            "type": PINOT_TYPES_MAPPING[column["dataType"]],
+                        }
+                    schema[schema_table_name]["columns"].append(c)


Nit: is there a way to pull the entire schema in one roundtrip to Pinot? This approach of looping through schemas -> tables -> columns will work but scales linearly as a schema grows. This isn't a blocker to merge but perhaps a way to incrementally improve in the future.

Looking at the Pinot docs it seems we could pull the entire schema in one roundtrip: https://docs.pinot.apache.org/users/tutorials/schema-evolution#get-the-existing-schema

Pinot doesn't support fetch all schemas at once, so this is the only way.

susodapop · 2022-08-24T13:43:56Z

redash/query_runner/pinot.py

+        return list(schema.values())
+
+    def get_schema_names(self):
+        return ["default"]


Does Pinot support schema names other than default? If so we should make this configurable or fetch them directly from the database.

Right now there is no concept of schema, so just use default for now.
Later we may consider adding tenant names as part of the schemas.

redash/query_runner/pinot.py

susodapop · 2022-08-24T13:49:12Z

Guess I forgot to @susodapop last week :p

I appreciate you pinging me so reliably. I've been hard at work on a couple other open source projects for the past month. But I swear I haven't forgotten about you :)

xiangfu0 · 2022-09-08T08:05:58Z

Guess I forgot to @susodapop last week :p

I appreciate you pinging me so reliably. I've been hard at work on a couple other open source projects for the past month. But I swear I haven't forgotten about you :)

Thanks for your comments!

Seems the CI cannot pass due to some visual change, can you help check what's going on?

xiangfu0 · 2022-09-24T09:48:46Z

@susodapop another round :p

xiangfu0 · 2022-09-25T09:42:48Z

Seems like the CI passed!
@susodapop can you go through this again ?

xiangfu0 · 2022-09-28T06:58:13Z

@susodapop ping :p

xiangfu0 · 2022-09-30T19:01:23Z

kindly Friday ping 🔢
@susodapop

susodapop · 2022-09-30T19:02:30Z

Thank you for the pings :) I have been out sick this week. Will review ASAP.

xiangfu0 · 2022-09-30T19:13:15Z

Thank you for the pings :) I have been out sick this week. Will review ASAP.

Oh, hope you get well soon and take your time!

arikfr · 2022-10-07T06:40:14Z

Merged. Thank you for your patience with this, @xiangfu0 !

xiangfu0 · 2022-10-07T18:01:16Z

Many thanks @arikfr and @susodapop !! Great team work!

* Adding Apache Pinot integration * address comments

xiangfu0 changed the title ~~Adding Apache Pinot integration~~ Adding Apache Pinot Query Runner Jul 18, 2022

xiangfu0 force-pushed the pinot-integration branch 2 times, most recently from 085ca51 to 45ab4f2 Compare July 19, 2022 08:34

mneedham mentioned this pull request Jul 19, 2022

Add Apache Pinot as a data source/query runner #5724

Closed

5 tasks

xiangfu0 force-pushed the pinot-integration branch from 45ab4f2 to 153bab5 Compare July 19, 2022 09:31

susodapop requested changes Aug 24, 2022

View reviewed changes

xiangfu0 force-pushed the pinot-integration branch 2 times, most recently from e3058c9 to 03ea39c Compare September 8, 2022 03:10

xiangfu0 requested a review from susodapop September 8, 2022 07:54

Adding Apache Pinot integration

3d8c86b

xiangfu0 force-pushed the pinot-integration branch from 03ea39c to 4df0c9c Compare September 24, 2022 09:46

xiangfu0 closed this Sep 24, 2022

xiangfu0 reopened this Sep 24, 2022

address comments

66402ca

xiangfu0 force-pushed the pinot-integration branch from 4df0c9c to 66402ca Compare September 24, 2022 16:27

arikfr merged commit a863c8c into getredash:master Oct 7, 2022

xiangfu0 deleted the pinot-integration branch October 7, 2022 18:00

xiangfu0 mentioned this pull request Oct 7, 2022

Adding Pinot Redash integration apache/pinot#6297

Closed

blarghmatey pushed a commit to mitodl/redash that referenced this pull request Oct 27, 2022

Adding Apache Pinot Query Runner (getredash#5798)

fe2933d

* Adding Apache Pinot integration * address comments

AkaashK pushed a commit to tharzeez/redash that referenced this pull request Jul 18, 2023

Adding Apache Pinot Query Runner (getredash#5798)

88db2ee

* Adding Apache Pinot integration * address comments

harveyrendell pushed a commit to pushpay/redash that referenced this pull request Jan 8, 2025

Adding Apache Pinot Query Runner (getredash#5798)

6a92a76

* Adding Apache Pinot integration * address comments

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adding Apache Pinot Query Runner #5798

Adding Apache Pinot Query Runner #5798

xiangfu0 commented Jul 18, 2022 •

edited

Loading

xiangfu0 commented Jul 19, 2022

susodapop commented Jul 19, 2022

xiangfu0 commented Jul 19, 2022

xiangfu0 commented Jul 19, 2022

susodapop commented Jul 19, 2022

xiangfu0 commented Jul 19, 2022

xiangfu0 commented Jul 19, 2022

xiangfu0 commented Aug 1, 2022

susodapop commented Aug 1, 2022

xiangfu0 commented Aug 17, 2022

xiangfu0 commented Aug 23, 2022

susodapop left a comment

susodapop Aug 24, 2022

xiangfu0 Sep 7, 2022

susodapop Aug 24, 2022

xiangfu0 Sep 7, 2022

susodapop commented Aug 24, 2022

xiangfu0 commented Sep 8, 2022

xiangfu0 commented Sep 24, 2022

xiangfu0 commented Sep 25, 2022

xiangfu0 commented Sep 28, 2022

xiangfu0 commented Sep 30, 2022

susodapop commented Sep 30, 2022

xiangfu0 commented Sep 30, 2022

arikfr commented Oct 7, 2022

xiangfu0 commented Oct 7, 2022 •

edited

Loading

Adding Apache Pinot Query Runner #5798

Adding Apache Pinot Query Runner #5798

Conversation

xiangfu0 commented Jul 18, 2022 • edited Loading

What type of PR is this?

Description

How is this tested?

1. Start redash docker-compose

2. Start Pinot Quickstart

3. Create data source:

4. Query Builder

Related Tickets & Documents

Mobile & Desktop Screenshots/Recordings (if there are UI changes)

xiangfu0 commented Jul 19, 2022

susodapop commented Jul 19, 2022

xiangfu0 commented Jul 19, 2022

xiangfu0 commented Jul 19, 2022

susodapop commented Jul 19, 2022

xiangfu0 commented Jul 19, 2022

xiangfu0 commented Jul 19, 2022

xiangfu0 commented Aug 1, 2022

susodapop commented Aug 1, 2022

xiangfu0 commented Aug 17, 2022

xiangfu0 commented Aug 23, 2022

susodapop left a comment

Choose a reason for hiding this comment

susodapop Aug 24, 2022

Choose a reason for hiding this comment

xiangfu0 Sep 7, 2022

Choose a reason for hiding this comment

susodapop Aug 24, 2022

Choose a reason for hiding this comment

xiangfu0 Sep 7, 2022

Choose a reason for hiding this comment

susodapop commented Aug 24, 2022

xiangfu0 commented Sep 8, 2022

xiangfu0 commented Sep 24, 2022

xiangfu0 commented Sep 25, 2022

xiangfu0 commented Sep 28, 2022

xiangfu0 commented Sep 30, 2022

susodapop commented Sep 30, 2022

xiangfu0 commented Sep 30, 2022

arikfr commented Oct 7, 2022

xiangfu0 commented Oct 7, 2022 • edited Loading

xiangfu0 commented Jul 18, 2022 •

edited

Loading

xiangfu0 commented Oct 7, 2022 •

edited

Loading