Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Runtime : Support multiple schemas with one live connection #4444

Merged
merged 28 commits into from
Apr 6, 2024
Merged
Show file tree
Hide file tree
Changes from 12 commits
Commits
Show all changes
28 commits
Select commit Hold shift + click to select a range
0970bf1
initial commit
k-anshul Mar 26, 2024
d5d0d32
fixing other APIs as well
k-anshul Mar 26, 2024
150f49f
Merge remote-tracking branch 'origin/main' into multiple_schemas
k-anshul Mar 26, 2024
90cd137
fixes in backend to split tbl name
k-anshul Mar 26, 2024
d8b201c
cover all table queries
k-anshul Mar 27, 2024
d21e6e9
fix unit tests
k-anshul Mar 27, 2024
fe61f9c
information_schema fix for duckdb + rename
k-anshul Mar 27, 2024
dba3da6
Merge remote-tracking branch 'origin/main' into multiple_schemas
k-anshul Mar 27, 2024
6fed4a4
don't need db and schema support for duckdb for now
k-anshul Mar 27, 2024
ef1b38a
revert temporary UI changes
k-anshul Mar 27, 2024
ceea597
Merge remote-tracking branch 'origin/main' into multiple_schemas
k-anshul Mar 27, 2024
7db9d8d
fix field rename
k-anshul Mar 27, 2024
64d895d
added support for database and schema in other profiling APIs
k-anshul Mar 29, 2024
df9a311
small self review
k-anshul Mar 29, 2024
ec47fbd
observability
k-anshul Mar 29, 2024
513a56a
review comments
k-anshul Mar 29, 2024
bc5e625
review comments
k-anshul Mar 29, 2024
ecf165f
review comments simplify information_schema calls
k-anshul Mar 29, 2024
9b6fe4c
review comments simplify information_schema calls
k-anshul Mar 29, 2024
ca5c4c9
small rename
k-anshul Mar 29, 2024
c80671f
missing changes
k-anshul Mar 29, 2024
653436d
Merge remote-tracking branch 'origin/main' into multiple_schemas
k-anshul Mar 29, 2024
779d812
Apply suggestions from code review
k-anshul Mar 31, 2024
d0bb5e5
nit
k-anshul Mar 31, 2024
a7c2082
Merge remote-tracking branch 'origin/main' into multiple_schemas
k-anshul Apr 5, 2024
bb58c43
UI to support multiple schemas in an OLAP engine (#4515)
ericpgreen2 Apr 5, 2024
3d9d92c
missing database and schema passing in new code
k-anshul Apr 6, 2024
f243936
Merge remote-tracking branch 'origin/main' into multiple_schemas
k-anshul Apr 6, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1,441 changes: 730 additions & 711 deletions proto/gen/rill/runtime/v1/api.pb.go

Large diffs are not rendered by default.

4 changes: 4 additions & 0 deletions proto/gen/rill/runtime/v1/api.pb.validate.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

415 changes: 222 additions & 193 deletions proto/gen/rill/runtime/v1/connectors.pb.go

Large diffs are not rendered by default.

6 changes: 6 additions & 0 deletions proto/gen/rill/runtime/v1/connectors.pb.validate.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

816 changes: 437 additions & 379 deletions proto/gen/rill/runtime/v1/queries.pb.go

Large diffs are not rendered by default.

12 changes: 12 additions & 0 deletions proto/gen/rill/runtime/v1/queries.pb.validate.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1,113 changes: 567 additions & 546 deletions proto/gen/rill/runtime/v1/resources.pb.go

Large diffs are not rendered by default.

4 changes: 4 additions & 0 deletions proto/gen/rill/runtime/v1/resources.pb.validate.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

56 changes: 56 additions & 0 deletions proto/gen/rill/runtime/v1/runtime.swagger.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -120,6 +120,14 @@ paths:
in: query
required: false
type: string
- name: database
in: query
required: false
type: string
- name: dbSchema
in: query
required: false
type: string
- name: table
in: query
required: false
Expand Down Expand Up @@ -711,6 +719,10 @@ paths:
properties:
connector:
type: string
database:
type: string
dbSchema:
type: string
table:
type: string
path:
Expand Down Expand Up @@ -1065,6 +1077,14 @@ paths:
in: query
required: false
type: string
- name: database
in: query
required: false
type: string
- name: dbSchema
in: query
required: false
type: string
- name: priority
in: query
required: false
Expand Down Expand Up @@ -1831,6 +1851,14 @@ paths:
in: query
required: false
type: string
- name: database
in: query
required: false
type: string
- name: dbSchema
in: query
required: false
type: string
- name: limit
in: query
required: false
Expand Down Expand Up @@ -1946,6 +1974,14 @@ paths:
in: query
required: false
type: string
- name: database
in: query
required: false
type: string
- name: dbSchema
in: query
required: false
type: string
- name: priority
in: query
required: false
Expand Down Expand Up @@ -4340,6 +4376,12 @@ definitions:
connector:
type: string
title: Connector containing the table
database:
type: string
title: Name of the database where table is located (optional)
dbSchema:
type: string
title: Name of the database schema where table is located (optional)
table:
type: string
title: Name of the table the metrics view is based on
Expand Down Expand Up @@ -5530,6 +5572,10 @@ definitions:
type: string
connector:
type: string
database:
type: string
dbSchema:
type: string
tableName:
type: string
title: Required
Expand All @@ -5550,6 +5596,10 @@ definitions:
type: string
connector:
type: string
database:
type: string
dbSchema:
type: string
tableName:
type: string
priority:
Expand All @@ -5572,6 +5622,8 @@ definitions:
properties:
database:
type: string
dbSchema:
type: string
name:
type: string
hasUnsupportedDataTypes:
Expand All @@ -5584,6 +5636,10 @@ definitions:
type: string
connector:
type: string
database:
type: string
dbSchema:
type: string
tableName:
type: string
limit:
Expand Down
2 changes: 2 additions & 0 deletions proto/rill/runtime/v1/api.proto
Original file line number Diff line number Diff line change
Expand Up @@ -504,6 +504,8 @@ message UnpackEmptyResponse {}
message GenerateMetricsViewFileRequest {
string instance_id = 1 [(validate.rules).string = {pattern: "^[_\\-a-zA-Z0-9]+$"}];
string connector = 2;
string database = 6;
string db_schema = 7;
string table = 3;
k-anshul marked this conversation as resolved.
Show resolved Hide resolved
string path = 4;
bool use_ai = 5;
Expand Down
3 changes: 3 additions & 0 deletions proto/rill/runtime/v1/connectors.proto
Original file line number Diff line number Diff line change
Expand Up @@ -183,6 +183,7 @@ message OLAPListTablesResponse {

message TableInfo {
string database = 1;
string db_schema = 4;
string name = 2;
// has_unsupported_data_types indicates if the underlying table has any column with an unsupported OLAP engine datatype
bool has_unsupported_data_types = 3;
Expand All @@ -191,6 +192,8 @@ message TableInfo {
message OLAPGetTableRequest {
string instance_id = 1;
string connector = 2;
string database = 4;
string db_schema = 5;
string table = 3;
}

Expand Down
6 changes: 6 additions & 0 deletions proto/rill/runtime/v1/queries.proto
Original file line number Diff line number Diff line change
Expand Up @@ -938,6 +938,8 @@ message TimeSeriesValue {
message TableCardinalityRequest {
string instance_id = 1;
string connector = 4;
string database = 5;
string db_schema = 6;
k-anshul marked this conversation as resolved.
Show resolved Hide resolved
// Required
string table_name = 2 [(validate.rules).string.min_len = 1];
int32 priority = 3;
Expand All @@ -951,6 +953,8 @@ message TableCardinalityResponse {
message TableColumnsRequest {
string instance_id = 1;
string connector = 4;
string database = 5;
string db_schema = 6;
string table_name = 2 [(validate.rules).string.min_len = 1];
int32 priority = 3;
}
Expand All @@ -972,6 +976,8 @@ message ProfileColumn {
message TableRowsRequest {
string instance_id = 1;
string connector = 5;
string database = 6;
string db_schema = 7;
string table_name = 2 [(validate.rules).string.min_len = 1];
int32 limit = 3 [(validate.rules).int32.gte = 0];
int32 priority = 4;
Expand Down
4 changes: 4 additions & 0 deletions proto/rill/runtime/v1/resources.proto
Original file line number Diff line number Diff line change
Expand Up @@ -179,6 +179,10 @@ message MetricsViewSpec {
}
// Connector containing the table
string connector = 1;
// Name of the database where table is located (optional)
string database = 21;
// Name of the database schema where table is located (optional)
string db_schema = 22;
// Name of the table the metrics view is based on
string table = 2;
// User friendly label for the dashboard
Expand Down
4 changes: 4 additions & 0 deletions runtime/compilers/rillv1/parse_metrics_view.go
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,8 @@ type MetricsViewYAML struct {
DisplayName string `yaml:"display_name"` // Backwards compatibility
Description string `yaml:"description"`
Model string `yaml:"model"`
Database string `yaml:"database"`
DBSchema string `yaml:"db_schema"`
Table string `yaml:"table"`
TimeDimension string `yaml:"timeseries"`
Watermark string `yaml:"watermark"`
Expand Down Expand Up @@ -416,6 +418,8 @@ func (p *Parser) parseMetricsView(node *Node) error {

spec.Connector = node.Connector
spec.Table = table
spec.DbSchema = tmp.DBSchema
spec.Database = tmp.Database
spec.Title = tmp.Title
spec.Description = tmp.Description
spec.TimeDimension = tmp.TimeDimension
Expand Down
Loading
Loading