[HUDI-4093] Fix NPE when insert records that partition column is null #5573
Conversation
|
high level question/thoughts. |
|
@watermelon12138 : can you respond to my question above. |
nsivabalan
added
priority:major
@nsivabalan I am so sorry，I didn't respond in time. This problem is about partition column value being null. The solution is that when the value of the partition column is null, the |
watermelon12138
force-pushed
the
fixInsertNPE
branch
3 times, most recently
from
068c83d to
d6b556c
Compare
… the insert operation throws NPE.
watermelon12138
force-pushed
the
fixInsertNPE
branch
from
d6b556c to
6c89bfb
Compare
| userSpecifiedDataTypes.keySet.foreach { name => | ||
| val dataType = userSpecifiedDataTypes.get(name).getOrElse("") | ||
| if (!dataType.isInstanceOf[StringType]) { | ||
| partitionStr = partitionStr.replace(s"$name=default", s"$name=__HIVE_DEFAULT_PARTITION__") |
There was a problem hiding this comment.
Why this is needed only for Spark3?
There was a problem hiding this comment.
Please also consider adding a test for this scenario.
| @@ -610,7 +610,7 @@ public static Option<String> getNullableValAsString(GenericRecord rec, String fi | |||
| * @return field value either converted (for certain data types) or as it is. | |||
| */ | |||
| public static Object convertValueForSpecificDataTypes(Schema fieldSchema, Object fieldValue, boolean consistentLogicalTimestampEnabled) { | |||
| if (fieldSchema == null) { | |||
| if (fieldSchema == null || fieldValue == null) { | |||
codope
changed the title
|
@watermelon12138 I suggest you to try out latest master as #7349 has been merged. It should fix the NPE. |
|
@watermelon12138 : If this is not an issue in current master, can you kindly close it ? |
… into partition hudi table
Tips
What is the purpose of the pull request
(For example: This pull request adds quick-start document.)
Brief change log
(for example:)
Verify this pull request
(Please pick either of the following options)
This pull request is a trivial rework / code cleanup without any test coverage.
(or)
This pull request is already covered by existing tests, such as (please describe tests).
(or)
This change added tests and can be verified as follows:
(example:)
Committer checklist
Has a corresponding JIRA in PR title & commit
Commit message is descriptive of the change
CI is green
Necessary doc changes done or have another open PR
For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.