Reducing size

[hengfeiyang]

• compress storedFields with chunk
• use zstd replace snappy
• packed int for docNum, offset of docValue
• packed int for TF, Loc of posting list
• compress chunk of posting list
• RunOptimize for roaring.Bitmap

After that,
we can reduce index size from 28 MB -> 9 MB with the same test data.
on a bigger data set,
we can reduce index size from 900 MB -> 160 MB with the same test data.

test data:

https://github.com/zinclabs/zinc/releases/download/v0.1.1/olympics.ndjson.gz

[hengfeiyang]

I fixed CI

[hengfeiyang]

Lint Error is not my fault, The golang-ci version is too old, not compatible with go 1.18.1.

please upgrade Github action.

i run golang-ci in my local is ok.

[mschoch]

Lint Error is not my fault, The golang-ci version is too old, not compatible with go 1.18.1.

please upgrade Github action.

i run golang-ci in my local is ok.

Don't worry, we are aware of the issues, have them on the main Bluge project as well: blugelabs/bluge#112

Hopefully I can get that sorted out, and apply it here as well.

I plan to start reviewing this weekend.

[voldyman]

Looks good! Is this going to be backwards compatible with existing indexes? or would users need to rebuild their index?

[hengfeiyang]

Looks good! Is this going to be backwards compatible with existing indexes? or would users need to rebuild their index?

Yes, the MR not compatible old version.

[mschoch]

OK, this is going to take longer than I originally thought, but I did an overview today and it all seems reasonable. A few initial questions/concerns, meanwhile I'll start trying to go feature by feature and be more thorough.

• reducing the index size is great, but I'm wondering if on the zinc side you have any performance tests? It's reasonable to wonder if there are some higher latency or reduced throughput that come with the trade-offs here.
• you say "compress storedFields with trunk" and "compress trunk of posting list" and also the code/comments make multiple references to 'trunk'. what does this word mean in this context? I am not familiar with how you are using it.

Thanks again for the contribution, I will dig deeper into the individual features next.

[hengfeiyang]

1. Sorry, trunk -> chunk, i update my description.
2. As we test, after compressed, the performance has be a bit better, i think because the size of file much reduced, so the read time reduced to.

[mschoch]

Ah, it is chunk, thanks, that makes sense.

[voldyman]

looks good to me, thanks for the great work @hengfeiyang !

[hengfeiyang]

fix panic when search in memory

[mschoch]

This looks great. I have done as much review as possible, thank you @voldyman for assisting. I think there is more value in getting this merged, so more people can use it, than there is value in waiting longer for me to review completely.

I will starting fixing the CI so we can proceed.

[mschoch]

Can we rename to chunkI

[hengfeiyang]

renamed

[mschoch]

OK, so my plan is to do the following tomorrow (please comment if you think there are other steps):

• Tag the existing master branch as v1.0.0 (this is more a formality to have a clean semantic version number to refer to what bluge has been using for the past year)
• Merge this PR
• Tag master as v2.0.0
• Update bluge config.go to support both ice v1 and v2
• Release new version of Bluge, with default to new format

[mschoch]

Alright I don't have perms to fix the lint issue on this branch, so I'll clean it up after merging

[hengfeiyang]

Alright I don't have perms to fix the lint issue on this branch, so I'll clean it up after merging

i merged ci udpate from master

[FZambia]

@hengfeiyang hello, seems like this PR introduced several critical issues in Bluge, see discussion in blugelabs/bluge#123, so the latest version is broken.

[hengfeiyang]

@FZambia i create a new pr sync our version in zinc.

[FZambia]

Thanks! Do you mean that in zinc you already have a version of Bluge which already solves those panics/errors?

[hengfeiyang]

I'm not sure fixed your case, we just fixed some panic. Also i can do more test with your given test code. blugelabs/bluge#123 (comment)

[hengfeiyang]

@FZambia I tested you test code, it work well with my PR #18