Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
[SPARK-46990][SQL] Fix loading empty Avro files emitted by event-hubs
### What changes were proposed in this pull request? This PR fixes a regression introduced by [SPARK-46633](https://issues.apache.org/jira/browse/SPARK-46633), commit: apache@3a6b9ad where one could not read an empty Avro file as the reader would be stuck in an infinite loop. I reverted the reader code to the pre-SPARK-46633 version and updated handling for empty blocks. When reading empty blocks in Avro, `blockRemaining` could still be read as 0. Call to `hasNext` status would load the next block but would still return false because of the final check `blockRemaining != 0`. Calling the method again picks up the next non-empty block and seems to fix the issue. ### Why are the changes needed? Fixes a regression introduced in [SPARK-46633](https://issues.apache.org/jira/browse/SPARK-46633). ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? I added a unit test to verify that empty files can be read correctly. ### Was this patch authored or co-authored using generative AI tooling? No. Closes apache#45578 from sadikovi/SPARK-46990. Authored-by: Ivan Sadikov <ivan.sadikov@databricks.com> Signed-off-by: Wenchen Fan <wenchen@databricks.com>
- Loading branch information