-
Notifications
You must be signed in to change notification settings - Fork 516
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Zstandard support #95
base: 2.5.5-dev2
Are you sure you want to change the base?
Conversation
Hello, Thank you for the PR. I don't use this type of compression but seems very promising. I think I will have the user manually install the zstandard lib with pip3 (will be documented in the wiki), and check if the lib is installed when using this option. Thanks again, much appreciated 👍 |
Sounds fair.
Yep, I compress all my local data with zstd. zstd is basically a better alternative to gzip/zlib. It creates smaller archives that decompress much faster. In fact, I get better performance searching over a zstd compressed file than searching over a decompressed file because I get limited by my disk io in both cases. Thanks for creating h8mail btw! It makes life much easier. I especially love the fact that it supports multi-threading when searching over local data. |
@@ -3,9 +3,9 @@ | |||
</h1> | |||
|
|||
[![platforms](https://img.shields.io/badge/platforms-Windows%20%7C%20Linux%20%7C%20OSX-success.svg)](https://pypi.org/project/h8mail/) [![PyPI version](https://badge.fury.io/py/h8mail.svg)](https://badge.fury.io/py/h8mail) | |||
[![PyPI - Python Version](https://img.shields.io/pypi/pyversions/h8mail.svg)](https://pypi.org/project/h8mail/) [![Downloads](https://pepy.tech/badge/h8mail)](https://pepy.tech/project/h8mail) [![travis](https://img.shields.io/travis/khast3x/h8mail.svg)](https://travis-ci.org/khast3x/h8mail) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@
h8mail/utils/run.py
Outdated
if user_args.single_file: | ||
local_found = local_search_single_zstd(res, targets) | ||
else: | ||
local_found = local_zstd_search(res, targets) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Zstandard, or
zstd
as short version, is a fast lossless compression algorithm, targeting real-time compression scenarios at zlib-level and better compression ratios.This PR adds support for searching through zst files using the zstandard Python bindings.
It works and gets the job done for now. Additional things that can be done:
localzstdsearch.py
andlocalgzipsearch.py
to reduce code duplicationutf-8