Use quickxorhash module if available

[wienand]

If available use quickxorhash module instead of subprocess call to quickxorhash executable.

Install module with

pip install quickxorhash

[incognito1234]

Hi Wienand,

Thanks for this patch. Unfortunately, it does not work with large files. Here is the result of a test with a 5 Gb file:

>>> from lib.check_helper import *
>>> q=quickxorhash()
>>> q.quickxorhash("../largefile.dat")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/pi/src/odc/lib/check_helper.py", line 25, in quickxorhash
    h.update(open(filename, 'rb').read())
OverflowError: unbounded read returned more bytes than a Python bytes object can hold

[wienand]

You could read the file in chunks and call the method update for every chunk:

      h = qxh.quickxorhash()
      h.update(open(filename, 'rb').read())
      return base64.b64encode(h.digest()).decode('utf8')

e.g.

      chunksize = 1024 * 1024 * 5
      h = qxh.quickxorhash()
      with open(filename, 'rb') as datafile:
          while True:
              chunk = datafile.read(chunksize)
              if chunk == b'':
                  break
              h.update(chunk)
      return base64.b64encode(h.digest()).decode('utf8')

Let me know if I should update the merge requests or create a new one.

Anyway, thanks to the library. I transferred all photos (~10 GB) from Amazon Photos to Onedrive with it.

[incognito1234]

I added the patch to the last release.
Thanks