Fix reading ORC/PARQUET over empty clipped schema #5674
Conversation
Signed-off-by: sperlingxx <lovedreamf@gmail.com>
|
build |
| // not reading any data, so return a degenerate ColumnarBatch with the row count | ||
| if (currentChunkMeta.numTotalRows == 0) { | ||
| None | ||
| } else { | ||
| val rows = currentChunkMeta.numTotalRows.toInt | ||
| // Someone is going to process this data, even if it is just a row count | ||
| GpuSemaphore.acquireIfNecessary(TaskContext.get(), metrics(SEMAPHORE_WAIT_TIME)) |
There was a problem hiding this comment.
This is inconsistent with the semantics of the ORC scan below. In this case, it will acquire the semaphore despite returning a row-count-only batch, while the ORC case below will avoid acquiring the semaphore unless it actually generates null columns.
There was a problem hiding this comment.
Aligned all GPU semaphore acquisitions for empty batches.
| case meta: HostMemoryEmptyMetaData => | ||
| // Not reading any data, but add in partition data if needed | ||
| // Someone is going to process this data, even if it is just a row count | ||
| GpuSemaphore.acquireIfNecessary(TaskContext.get(), metrics(SEMAPHORE_WAIT_TIME)) |
There was a problem hiding this comment.
Same inconsistent acquisition on empty batches issue here.
| // Not reading any data, but add in partition data if needed | ||
| // Someone is going to process this data, even if it is just a row count | ||
| GpuSemaphore.acquireIfNecessary(TaskContext.get(), metrics(SEMAPHORE_WAIT_TIME)) |
There was a problem hiding this comment.
Same semaphore acquisition on empty batch consistency here.
|
build |
|
@sperlingxx does this also fix #5694? If not can you please take a look at that issue. |
sperlingxx
requested review from
revans2,
GaryShen2008,
NvTimLiu and
zhanga5
as code owners
sperlingxx
force-pushed
the
fix_read_empty_clipped_schema
branch
from
55791f2 to
eeb3a61
Compare
sperlingxx
removed request for
revans2,
zhanga5,
GaryShen2008,
NvTimLiu and
firestarman
Yes, @tgravescs. It will fix #5694 as well. |
|
I apologize that I did an upmerge to 22.08 by mistake. To fix the blunder, I had to run a force update to shift the base. |
|
build |
2 similar comments
|
build |
|
build |
|
build |
Fixes #5672
Fixes #5694
Fix reading ORC/PARQUET over empty clipped schema through bypassing the call of cuDF table reader.
Signed-off-by: sperlingxx lovedreamf@gmail.com