Add hidden $partition column to Hive connector #3582
Conversation
| @@ -384,9 +385,12 @@ public int getIndex() | |||
| } | |||
| } | |||
| else { | |||
| String partitionKeyValues = String.join("/", partitionKeys.stream() | |||
| .map(partitionKey -> format("%s=%s", partitionKey.getName(), partitionKey.getValue())) | |||
There was a problem hiding this comment.
Do we need some escaping of / or = here?
There was a problem hiding this comment.
Also wouln't MAP be a better type here instead of VARCHAR?
There was a problem hiding this comment.
I think escaping or encoding is needed. Let me update it.
Exactly, MAP type is also a candidate for this column.
@martint Do you have any opinion?
There was a problem hiding this comment.
there might be also be :-escaping and canonicalization of representation.
but, since we know partition name before hand, why dont we just pass it here, instead of reconsructing?
did you consider this approach?
cc @electrum
There was a problem hiding this comment.
Replaced with partitionName provided by Hive split.
We may still need to add manual escaping in tests.
ebyhr
force-pushed
the
hive-partition-column
branch
3 times, most recently
from
dcb8c53 to
57ce58c
Compare
| assertEquals(results.getRowCount(), 9); | ||
|
|
||
| assertUpdate("DROP TABLE test_partition_hidden_column"); | ||
| assertFalse(getQueryRunner().tableExists(getSession(), "test_partition_hidden_column")); |
| assertEquals(getPartitions("test_partition_hidden_column").size(), 9); | ||
|
|
||
| MaterializedResult results = computeActual(format("SELECT *, \"%s\" FROM test_partition_hidden_column", PARTITION_COLUMN_NAME)); | ||
| for (int i = 0; i < results.getRowCount(); i++) { |
There was a problem hiding this comment.
You can drop index from loop. IMO this looks a bit cleaner:
for (MaterializedRow row : results.getMaterializedRows()) {
String actualPartition = (String) row.getField(3);
String expectedPartition = format("col1=%s/col2=%s", row.getField(1), row.getField(2));
assertEquals(actualPartition, expectedPartition);
}
| ColumnMetadata columnMetadata = columnMetadatas.get(i); | ||
| assertEquals(columnMetadata.getName(), columnNames.get(i)); | ||
| if (columnMetadata.getName().equals(PARTITION_COLUMN_NAME)) { | ||
| // $partition should be hidden column |
ebyhr
force-pushed
the
hive-partition-column
branch
from
57ce58c to
5b6eaa9
Compare
Initial support for #5