[HUDI-915] Add missing partititonpath to records COW #8666
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jonvex
force-pushed
the
fix_bootstrap_filegroup_update_lost_partition
branch
from
7510bce to
5d1b90a
Compare
codope
reviewed
May 15, 2023
...nt/hudi-client-common/src/main/java/org/apache/hudi/table/action/commit/BaseMergeHelper.java
Outdated
Show resolved
Hide resolved
...lient/hudi-flink-client/src/main/java/org/apache/hudi/table/HoodieFlinkCopyOnWriteTable.java
Outdated
Show resolved
Hide resolved
...-client/hudi-spark-client/src/main/java/org/apache/hudi/client/bootstrap/PartitionUtils.java
Outdated
Show resolved
Hide resolved
...-client/hudi-spark-client/src/main/java/org/apache/hudi/client/bootstrap/PartitionUtils.java
Outdated
Show resolved
Hide resolved
...n/java/org/apache/hudi/client/clustering/run/strategy/MultipleSparkJobExecutionStrategy.java
Outdated
Show resolved
Hide resolved
...-client/hudi-spark-client/src/main/java/org/apache/hudi/client/bootstrap/PartitionUtils.java
Outdated
Show resolved
Hide resolved
hudi-common/src/main/java/org/apache/hudi/io/storage/HoodieBootstrapRecordIterator.java
Outdated
Show resolved
Hide resolved
hudi-common/src/test/java/org/apache/hudi/common/testutils/HoodieTestUtils.java
Show resolved
Hide resolved
...-spark-datasource/hudi-spark/src/test/java/org/apache/hudi/functional/TestBootstrapRead.java
Outdated
Show resolved
Hide resolved
|
|
||
| private static Stream<Arguments> testArgs() { | ||
| Stream.Builder<Arguments> b = Stream.builder(); | ||
| //TODO: add dash partitions false with [HUDI-4944] |
There was a problem hiding this comment.
please add more details here.
Also, you could try adding a sleep of 2s and check if timeline server returns correct fs view.
There was a problem hiding this comment.
- fixed
- I added a 5 second delay between bootstrapping and the update and was still getting failures at the same rate
There was a problem hiding this comment.
Ok let's tackle that issue separately but I don't see more details in the comment.
There was a problem hiding this comment.
I changed the line below to:
Boolean[] dashPartitions = {true/,false/};
The code is already there to test that, it just doesn't work currently because of the double decoding issue
codope
reviewed
May 16, 2023
...nt/hudi-client-common/src/main/java/org/apache/hudi/table/action/commit/BaseMergeHelper.java
Outdated
Show resolved
Hide resolved
hudi-common/src/main/java/org/apache/hudi/io/storage/HoodieBootstrapRecordIterator.java
Outdated
Show resolved
Hide resolved
hudi-common/src/test/java/org/apache/hudi/common/testutils/HoodieTestUtils.java
Show resolved
Hide resolved
|
|
||
| private static Stream<Arguments> testArgs() { | ||
| Stream.Builder<Arguments> b = Stream.builder(); | ||
| //TODO: add dash partitions false with [HUDI-4944] |
There was a problem hiding this comment.
Ok let's tackle that issue separately but I don't see more details in the comment.
codope
approved these changes
May 17, 2023
4 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Change Logs
In a bootstrapped table, If there was an upsert in a file group, the records in the file group that were not upserted would have null values for their partition column.
Impact
Now works correctly
Risk level (write none, low medium or high below)
Medium
Lots of tests written
Documentation Update
N/A
Contributor's checklist