Uniquely identify image and differences with libjpeg and libjpeg-turbo

[qlimax86]

Hello!

To uniquely identify an image I'm calculating the image histogram with Pillow and then hash it with sha1
This histogram is calculated after some transformation of the image (crop, thumb)
This is still a valid way to uniquely identify images IMO.

The problem is that with Pillow >= 9.0.0 using libjpeg-turbo the resulting images are slightly different and in consequence the histogram of my images is different from the past calculated histograms.

Any ideas how to solve the problem?
Is there a way to tell Pillow to use libjpeg instead of libjpeg-turbo?
Any other ideas to uniquely identify an image by using Pillow?

Thank You
Q

[radarhere]

Hi. Pillow only uses libjpeg-turbo if you are installing from the wheel, or if libjpeg-turbo is installed.

To install Pillow with pip, but without using the wheel, you can run

python3 -m pip install --upgrade Pillow --no-binary :all:

You will likely have to install various dependencies, because you are now building from source. See https://pillow.readthedocs.io/en/stable/installation.html#building-from-source for more information.

[qlimax86]

thank you for your quick reply! I appreciated!

So this is what I did:

• compiled the library from the sources: https://www.ijg.org/files/jpegsrc.v9d.tar.gz:

./configure --prefix=/opt/jpeg
make
make install

• install Pillow with CFLAGS:
  CFLAGS="-I/opt/jpeg/include -L/opt/jpeg/lib" pip install --verbose --no-cache -U Pillow==9.1.1 --no-binary :all:
  at the end of this install (with --verbose) you get this nice recap, in which you can see if pillow is installed with all the stuff you need.
  in my case:

  --------------------------------------------------------------------
  PIL SETUP SUMMARY
  --------------------------------------------------------------------
  version      Pillow 9.1.1
  platform     linux 3.7.12 (default, Jan 29 2022, 06:07:21)
               [GCC 10.2.1 20210110]
  --------------------------------------------------------------------
  --- JPEG support available
  --- OPENJPEG (JPEG2000) support available (2.4)
  --- ZLIB (PNG/ZIP) support available
  *** LIBIMAGEQUANT support not available
  --- LIBTIFF support available
  --- FREETYPE2 support available
  *** RAQM (Text shaping) support not available
  --- LITTLECMS2 support available
  *** WEBP support not available
  *** WEBPMUX support not available
  --- XCB (X protocol) support available
  --------------------------------------------------------------------

• now, before the python interpreter is started, add this environment variable to tell python where to find the compiled library.
  In alternative configure prefix with /usr before make & make install
  LD_LIBRARY_PATH="/opt/jpeg/lib" python my_script.py

Ok, it works but it's a bit tricky..
It would have been great to choose which library to use at runtime so that the resulting image would have been identical/backward-compatible with the images generated with Pillow < 9.0.0 ... but I think I'm the only one on the planet who catalogs the images using this technique to avoid duplicates (scope: web uploads)

Anyway, any other ideas to uniquely fingerprint an image?
Again, thank you for the answer
Q

[radarhere]

If I understand your situation correctly, then you might want to look at comments in #3120. That is an open feature request for a perceptual hash in Pillow, and there are some suggestions in there.