Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PARQUET-41: Add bloom filter #757

Merged
merged 18 commits into from
Feb 26, 2020
Merged

PARQUET-41: Add bloom filter #757

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
d473d17
PARQUET-1328: Add Bloom filter reader and writer (#587)
chenjunjiedada Jan 21, 2019
96c2fef
PARQUET-1516: Store Bloom filters near to footer (#608)
chenjunjiedada Feb 12, 2019
dd7e655
PARQUET-1391: Integrate Bloom filter logic (#619)
chenjunjiedada Mar 19, 2019
21c45ed
Merge branch 'master' into bloom-filter
chenjunjiedada May 30, 2019
1fc2733
Merge branch 'master' into bloom-filter
chenjunjiedada Sep 20, 2019
ba28686
PARQUET-1660: align Bloom filter implementation with format (#686)
chenjunjiedada Jan 7, 2020
1e44aa4
Merge remote-tracking branch 'apache/master' into bloom-filter
chenjunjiedada Jan 9, 2020
e396d15
Merge branch 'master' into bloom-filter
chenjunjiedada Feb 12, 2020
6260ca4
remove some useless changes
chenjunjiedada Feb 12, 2020
a67480d
address comments first round
chenjunjiedada Feb 16, 2020
28216ce
address comments second round
chenjunjiedada Feb 17, 2020
1519967
address comments, add unit tests, shade jar
chenjunjiedada Feb 20, 2020
6a8e641
Add xxhash correctness unit test, update shading, upper bound and som…
chenjunjiedada Feb 22, 2020
069cc22
add more unit tests, some minor fixes
chenjunjiedada Feb 24, 2020
c16e8e8
minor updates
chenjunjiedada Feb 25, 2020
8f04bb8
minor updates
chenjunjiedada Feb 25, 2020
5a98e78
fix build
chenjunjiedada Feb 26, 2020
373d811
Merge branch 'master' into bloom-filter
gszadovszky Feb 26, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 0 additions & 4 deletions ...umn/src/main/java/org/apache/parquet/column/values/bloomfilter/BlockSplitBloomFilter.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,6 @@
import java.nio.ByteBuffer;
import java.nio.ByteOrder;
import java.nio.IntBuffer;
import java.util.Arrays;

/*
* This Bloom filter is implemented using block-based Bloom filter algorithm from Putze et al.'s
Expand Down Expand Up @@ -223,8 +222,6 @@ public void writeTo(OutputStream out) throws IOException {
}

private int[] setMask(int key) {
Arrays.fill(mask, 0);

// The following three loops are written separately so that they could be
// optimized for vectorization.
for (int i = 0; i < BITS_SET_PER_BLOCK; ++i) {
gszadovszky marked this conversation as resolved.
Show resolved Hide resolved
Expand Down Expand Up @@ -316,7 +313,6 @@ public long hash(Object value) {
return hashFunction.hashBytes(((Binary) value).getBytes());
}

cacheBuffer.clear();
if (value instanceof Integer) {
gszadovszky marked this conversation as resolved.
Show resolved Hide resolved
cacheBuffer.putInt((Integer)value);
} else if (value instanceof Long) {
Expand Down
6 changes: 3 additions & 3 deletions ...src/test/java/org/apache/parquet/column/values/bloomfilter/TestBlockSplitBloomFilter.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -77,11 +77,11 @@ public void testBloomFilterForPrimitives() {
}

private void testBloomFilterForPrimitives(long seed) {
Random random = new Random(seed);
final int numValues = 1024 * 1024;
final int numBytes = BlockSplitBloomFilter.optimalNumOfBits(numValues , 0.01) / 8;

final int numBytes = BlockSplitBloomFilter.optimalNumOfBits(numValues , random.nextDouble()/10) / 8;
gszadovszky marked this conversation as resolved.
Show resolved Hide resolved
BloomFilter bloomFilter = new BlockSplitBloomFilter(numBytes);
Random random = new Random(seed);

Set<Object> values = new HashSet<>();

for (int i = 0; i < numValues; i++) {
Expand Down