-
Notifications
You must be signed in to change notification settings - Fork 3.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
docs: sql unnest and cleanup unnest datasource #13736
Merged
Merged
Changes from all commits
Commits
Show all changes
48 commits
Select commit
Hold shift + click to select a range
d2e244e
docs: sql unnest and cleanup datasource unnest
317brian 87adfd7
words
317brian d6b735e
caps
317brian 58eb0df
fix
317brian d4ba9d8
address comments
317brian e96ef27
address comments
317brian adf1a71
address comments from vtlim
317brian 6b274a6
address comments from somu
317brian 0f4c3cc
address comments from Victoria
317brian 4f60c8e
fix filename
317brian fafafd4
Merge branch 'master' into unnest-sql
317brian 3439ba2
update spelling file
317brian b494f2e
Fix a link problem (#13876)
writer-jill ddc9971
docs: fix html nits (#13835)
317brian ed21780
Add Post Aggregators for Tuple Sketches (#13819)
anshu-makkar 351dcfc
do not run non sql compatible tests for all jdk flavours (#13875)
abhishekagarwal87 c1b20e6
Bump CycloneDX module to address POM errors (#13878)
imply-elliott da7e72b
Allow druid-kubernetes-overlord-extensions to be loaded in any druid …
9d64618
use getProperty in MSQDurableStorageModule (#13881)
cf3456e
Python Druid API for use in notebooks (#13787)
paul-rogers 5a200cd
Fix durable storage cleanup (#13853)
rohangarg 05a27f1
Adding forbidden api for Properties#get() and Properties#getOrDefault…
cryptoe 7fdce23
Suggested memory calculation in case NOT_ENOUGH_MEMORY_FAULT is throw…
cryptoe d7652cf
Stream Kubernetes Job Logs (#13869)
81506fc
Web console: Compaction history dialog (#13861)
vogievetsky de2c820
Add warning comments to Granularity.getIterable. (#13888)
gianm deab298
fix SQL in segment card (#13895)
vogievetsky 385f383
use Calcites.getColumnTypeForRelDataType for SQL CAST operator conver…
clintropolis 04a1eac
Use base task dir in kubernetes task runner (#13880)
8895cf8
Add validation for aggregations on __time (#13793)
adarshsanjeev 43b77fc
Improved error message when topic name changes within same supervisor…
abhishekagarwal87 34fc9b0
fix ci (#13901)
clintropolis 5f8590f
Fix start-druid for indexers. (#13891)
gianm f219371
Sort-merge join and hash shuffles for MSQ. (#13506)
gianm 27a56c5
fix KafkaInputFormat when used with Sampler API (#13900)
clintropolis a0d4204
Fix for OOM in the Tombstone generating logic in MSQ (#13893)
LakshSingla 1921fc4
Avoid creating new RelDataTypeFactory during SQL planning. (#13904)
gianm 5c96758
use native nvl expression for SQL NVL and 2 argument COALESCE (#13897)
clintropolis b337537
Merge branch 'master' into unnest-sql
317brian c8b3f3a
merge master
317brian 011047d
updates to unnest for latest changes
317brian 3282888
remve mention of allowlist
317brian 87fc458
add name to virtualcolumn block for unnest datasource
317brian 62ad960
add show/hide
317brian e3e8753
fix missing anchor link
317brian aea3e9a
Merge branch 'master' into unnest-sql
317brian 4d5c24b
Apply suggestions from code review
317brian f9beef2
Apply suggestions from code review
317brian File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -55,6 +55,7 @@ Druid SQL supports SELECT queries with the following structure: | |
[ WITH tableName [ ( column1, column2, ... ) ] AS ( query ) ] | ||
SELECT [ ALL | DISTINCT ] { * | exprs } | ||
FROM { <table> | (<subquery>) | <o1> [ INNER | LEFT ] JOIN <o2> ON condition } | ||
[, UNNEST(source_expression) as table_alias_name(column_alias_name) ] | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This is the right one, we should do the previous thing the same name <table_alias_name> |
||
[ WHERE expr ] | ||
[ GROUP BY [ exprs | GROUPING SETS ( (exprs), ... ) | ROLLUP (exprs) | CUBE (exprs) ] ] | ||
[ HAVING expr ] | ||
|
@@ -82,6 +83,43 @@ FROM clause, metadata tables are not considered datasources. They exist only in | |
For more information about table, lookup, query, and join datasources, refer to the [Datasources](datasource.md) | ||
documentation. | ||
|
||
## UNNEST | ||
|
||
> The UNNEST SQL function is [experimental](../development/experimental.md). Its API and behavior are subject | ||
> to change in future releases. It is not recommended to use this feature in production at this time. | ||
|
||
The UNNEST clause unnests array values. It's the SQL equivalent to the [unnest datasource](./datasource.md#unnest). The source for UNNEST can be an array or an input that's been transformed into an array, such as with helper functions like MV_TO_ARRAY or ARRAY. | ||
|
||
The following is the general syntax for UNNEST, specifically a query that returns the column that gets unnested: | ||
|
||
```sql | ||
SELECT column_alias_name FROM datasource, UNNEST(source_expression1) AS table_alias_name1(column_alias_name1), UNNEST(source_expression2) AS table_alias_name2(column_alias_name2), ... | ||
``` | ||
|
||
* The `datasource` for UNNEST can be any Druid datasource, such as the following: | ||
* A table, such as `FROM a_table`. | ||
* A subset of a table based on a query, a filter, or a JOIN. For example, `FROM (SELECT columnA,columnB,columnC from a_table)`. | ||
* The `source_expression` for the UNNEST function must be an array and can come from any expression. If the dimension you are unnesting is a multi-value dimension, you have to specify `MV_TO_ARRAY(dimension)` to convert it to an implicit ARRAY type. You can also specify any expression that has an SQL array datatype. For example, you can call UNNEST on the following: | ||
* `ARRAY[dim1,dim2]` if you want to make an array out of two dimensions. | ||
* `ARRAY_CONCAT(dim1,dim2)` if you want to concatenate two multi-value dimensions. | ||
* The `AS table_alias_name(column_alias_name)` clause is not required but is highly recommended. Use it to specify the output, which can be an existing column or a new one. Replace `table_alias_name` and `column_alias_name` with a table and column name you want to alias the unnested results to. If you don't provide this, Druid uses a nondescriptive name, such as `EXPR$0`. | ||
|
||
Keep the following things in mind when writing your query: | ||
|
||
- You must include the context parameter `"enableUnnest": true`. | ||
- You can unnest multiple source expressions in a single query. | ||
- Notice the comma between the datasource and the UNNEST function. This is needed in most cases of the UNNEST function. Specifically, it is not needed when you're unnesting an inline array since the array itself is the datasource. | ||
- If you view the native explanation of a SQL UNNEST, you'll notice that Druid uses `j0.unnest` as a virtual column to perform the unnest. An underscore is added for each unnest, so you may notice virtual columns named `_j0.unnest` or `__j0.unnest`. | ||
- UNNEST preserves the ordering of the source array that is being unnested. | ||
|
||
For examples, see the [Unnest arrays tutorial](../tutorials/tutorial-unnest-arrays.md). | ||
|
||
The UNNEST function has the following limitations: | ||
|
||
- The function does not remove any duplicates or nulls in an array. Nulls will be treated as any other value in an array. If there are multiple nulls within the array, a record corresponding to each of the nulls gets created. | ||
- Arrays inside complex JSON types are not supported. | ||
- You cannot perform an UNNEST at ingestion time, including SQL-based ingestion using the MSQ task engine. | ||
|
||
## WHERE | ||
|
||
The WHERE clause refers to columns in the FROM table, and will be translated to [native filters](filters.md). The | ||
|
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should we be constant and call it Unnest or Unnesting at all places ?
multi-value-dimensions.md
calls this UnnestingThere was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Based on Clint's comment, I've removed it from the mvd page.