Skip to content

Commit

Permalink
[SPARK-23358][CORE] When the number of partitions is greater than 2^2…
Browse files Browse the repository at this point in the history
…8, it will result in an error result

## What changes were proposed in this pull request?
In the `checkIndexAndDataFile`,the `blocks` is the ` Int` type,  when it is greater than 2^28, `blocks*8` will overflow, and this will result in an error result.
In fact, `blocks` is actually the number of partitions.

## How was this patch tested?
Manual test

Author: liuxian <liu.xian3@zte.com.cn>

Closes apache#20544 from 10110346/overflow.
  • Loading branch information
10110346 authored and srowen committed Feb 9, 2018
1 parent 4b4ee26 commit f77270b
Showing 1 changed file with 1 addition and 1 deletion.
Original file line number Diff line number Diff line change
Expand Up @@ -84,7 +84,7 @@ private[spark] class IndexShuffleBlockResolver(
*/
private def checkIndexAndDataFile(index: File, data: File, blocks: Int): Array[Long] = {
// the index file should have `block + 1` longs as offset.
if (index.length() != (blocks + 1) * 8) {
if (index.length() != (blocks + 1) * 8L) {
return null
}
val lengths = new Array[Long](blocks)
Expand Down

0 comments on commit f77270b

Please sign in to comment.