Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[HUDI-5244] Fix bugs in schema evolution client with lost operation field and not found schema #7248

Merged
merged 2 commits into from
Nov 21, 2022

Conversation

trushev
Copy link
Contributor

@trushev trushev commented Nov 20, 2022

Change Logs

This PR fixes 2 issues in schema evolution client api:

  1. Lost operation field in avro schema
  2. Not found schema for table
org.apache.hudi.exception.HoodieException: cannot find schema for current table: /tmp/hudi
	at org.apache.hudi.client.BaseHoodieWriteClient.getInternalSchemaAndMetaClient(BaseHoodieWriteClient.java:1767)
	at org.apache.hudi.client.BaseHoodieWriteClient.addColumn(BaseHoodieWriteClient.java:1673)
	at org.apache.hudi.sink.TestWriteCopyOnWrite.test(TestWriteCopyOnWrite.java:454)

Introduced new test TestSchemaEvolutionClientfor schema evolution client api.

Flink schema evolution #5830 requires these fixes

Impact

Fixed lost operation field in avro schema and not found schema for table

Risk level

Low

Documentation Update

None

Contributor's checklist

  • Read through contributor's guide
  • Change Logs and Impact were stated clearly
  • Adequate tests were added if applicable
  • CI passed

@hudi-bot
Copy link

CI report:

Bot commands @hudi-bot supports the following commands:
  • @hudi-bot run azure re-run the last Azure build

@xiarixiaoyao
Copy link
Contributor

LGTM
@trushev
Does Spark have this problem?

@trushev
Copy link
Contributor Author

trushev commented Nov 21, 2022

LGTM @trushev Does Spark have this problem?

"Not found schema for table" -- no it is just java client api bug
"Lost operation field in avro schema" -- I believe yes because saveInternalSchema is used in spark as well as in flink

@xiarixiaoyao xiarixiaoyao merged commit 4c21823 into apache:master Nov 21, 2022
@trushev
Copy link
Contributor Author

trushev commented Nov 21, 2022

@xiarixiaoyao
This PR have been merged right after #7250 which consists of conflicted code removing HoodieJavaClientTestBase
Need to revert and rework PR

trushev added a commit to trushev/hudi that referenced this pull request Nov 21, 2022
…ration field and not found schema (apache#7248)"

This reverts commit 4c21823.
satishkotha pushed a commit to satishkotha/incubator-hudi that referenced this pull request Dec 12, 2022
…ield and not found schema (apache#7248)

* [HUDI-5244] Fix bugs in schema evolution client with lost operation field and not found schema
alexeykudinkin pushed a commit to onehouseinc/hudi that referenced this pull request Dec 14, 2022
…ield and not found schema (apache#7248)

* [HUDI-5244] Fix bugs in schema evolution client with lost operation field and not found schema
alexeykudinkin pushed a commit to onehouseinc/hudi that referenced this pull request Dec 14, 2022
…ield and not found schema (apache#7248)

* [HUDI-5244] Fix bugs in schema evolution client with lost operation field and not found schema
alexeykudinkin pushed a commit to onehouseinc/hudi that referenced this pull request Dec 14, 2022
…ield and not found schema (apache#7248)

* [HUDI-5244] Fix bugs in schema evolution client with lost operation field and not found schema
alexeykudinkin pushed a commit to onehouseinc/hudi that referenced this pull request Dec 14, 2022
…ield and not found schema (apache#7248)

* [HUDI-5244] Fix bugs in schema evolution client with lost operation field and not found schema
fengjian428 pushed a commit to fengjian428/hudi that referenced this pull request Apr 5, 2023
…ield and not found schema (apache#7248)

* [HUDI-5244] Fix bugs in schema evolution client with lost operation field and not found schema
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Archived in project
Development

Successfully merging this pull request may close these issues.

3 participants