You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Describe the bug
The AVRO coalescing reading, introduced in #5306, will produce a big AVRO file with different sync markers when the sync marker varies in the original files. But one AVRO file should have the same sync marker in each sync block.
This bug exists because the reader simply concatenates the blocks from different files. We choose to do this is the implementation is quite simple for the quick start, and the cuDF AVRO reader ignores the sync markers for now.
Expected behavior
We should write the same sync marker into the coalesced AVRO file, no matter whether the original files have different sync markers.
The text was updated successfully, but these errors were encountered:
so to clarify this isn't really a problem right now because CUDF ignores the sync markers right now, but we should clean it up in case they do, correct?
so to clarify this isn't really a problem right now because CUDF ignores the sync markers right now, but we should clean it up in case they do, correct?
sameerz
changed the title
[BUG]The coalesced AVRO file may contain different sync markers if the sync marker varies in the avro files being coalesced.
[BUG] The coalesced AVRO file may contain different sync markers if the sync marker varies in the avro files being coalesced.
Jun 6, 2022
Describe the bug
The AVRO coalescing reading, introduced in #5306, will produce a big AVRO file with different sync markers when the sync marker varies in the original files. But one AVRO file should have the same sync marker in each sync block.
This bug exists because the reader simply concatenates the blocks from different files. We choose to do this is the implementation is quite simple for the quick start, and the cuDF AVRO reader ignores the sync markers for now.
Expected behavior
We should write the same sync marker into the coalesced AVRO file, no matter whether the original files have different sync markers.
The text was updated successfully, but these errors were encountered: