Allow gmic.eu for files download at build time

[myselfhimself]

Details

• Read the Docs project URL: https://gmic-py.readthedocs.io/en/latest/
• Build URL (if applicable): https://readthedocs.org/projects/gmic-py/builds/12455625/
• Read the Docs username (if applicable): myselfhimself

Expected Result

G'MIC (C++) / gmic-py (Python) is a framework for generating images and applying filters. One of its commands sp or sample grabs an image from gmic.eu, with URLs such as https://gmic.eu/img/sample_earth.png
I use this command a lot to prevent to pre-download and push sample image files onto gmic-py's Github repository.

Actual Result

Currently the URL seems to be blocked by the readthedocs.org buildbot environment, when it is being tentatively accessed by gmic-py (which embeds libcurl and uses libcurl for such remote file retrievals). It could be that gmic.eu is no allowed domain, but I am unsure of this.

Would you agree to whitelist gmic.eu , if that may solve the problem?

Thank you in advance for your assistance!

[stsewd]

Don't think we block any resources in the builders. Could you link me to where this is executed? Didn't find anything obvious in https://github.com/myselfhimself/gmic-py/blob/master/docs/conf.py

[myselfhimself]

Hello, this is executed by the gmic-sphinx gmicpic directive using the G'MIC sample or sp command:

.. gmicpic:: sample earth display

For example in docs/tutorials.rst

the above line triggers a gmic libcurl download call to https://gmic.eu/img/sample_earth.png

Note I face a similar download error for the gmic module (without sphinx) error in Jupyter and Google Colab environments. It could be that G'MIC's libcurl does not autoconfigure itself properly.

Do you have any proxy environment variable that libcurl should be fed with at call time ?

[myselfhimself]

Here is the libcurl configuration in use. The curl executable does work on Google Colab (where a similar libcurl download error happens as in the readthedocs buildbot environment). I think that G'MIC has to patch some of its network code...

(from https://raw.githubusercontent.com/dtschump/CImg/master/CImg.h)

#ifdef cimg_use_curl
      const unsigned int omode = cimg::exception_mode();
      cimg::exception_mode(0);
      try {
        CURL *curl = 0;
        CURLcode res;
        curl = curl_easy_init();
        if (curl) {
          file = cimg::fopen(filename_local,"wb");
          curl_easy_setopt(curl,CURLOPT_URL,url);
          curl_easy_setopt(curl,CURLOPT_WRITEFUNCTION,0);
          curl_easy_setopt(curl,CURLOPT_WRITEDATA,file);
          curl_easy_setopt(curl,CURLOPT_SSL_VERIFYPEER,0L);
          curl_easy_setopt(curl,CURLOPT_SSL_VERIFYHOST,0L);
          curl_easy_setopt(curl,CURLOPT_FOLLOWLOCATION,1L);
          if (timeout) curl_easy_setopt(curl,CURLOPT_TIMEOUT,(long)timeout);
          if (std::strchr(url,'?')) curl_easy_setopt(curl,CURLOPT_HTTPGET,1L);
          if (referer) curl_easy_setopt(curl,CURLOPT_REFERER,referer);
          res = curl_easy_perform(curl);
          curl_easy_cleanup(curl);
          cimg::fseek(file,0,SEEK_END); // Check if file size is 0
          const cimg_ulong siz = cimg::ftell(file);
          cimg::fclose(file);
          if (siz>0 && res==CURLE_OK) {
            cimg::exception_mode(omode);
            return filename_local;
          } else std::remove(filename_local);
        }

[myselfhimself]

Reproduced:

docker run -it readthedocs/build

docs@b829f755733f:~$ python3.7 -m pip install gmic

gmic seems to be linked against curl:

docs@b829f755733f:~/.pyenv/versions/3.7.3/lib/python3.7/site-packages$ ldd gmic.cpython-37m-x86_64-linux-gnu.so  | grep curl
	libcurl-783997b8.so.4.3.0 => /home/docs/.pyenv/versions/3.7.3/lib/python3.7/site-packages/./gmic.libs/libcurl-783997b8.so.4.3.0 (0x00007f526b65c000)

Attempting gmic libcurl-based download:

docs@b829f755733f:~$ python3.7 -c "import gmic; gmic.run('sp leno'); print('bonsoir')"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
gmic.GmicException: *** Error in ./sp/_sample/*repeat/*if/*if/ *** Unreachable network file 'https://gmic.eu/img/sample_leno.png'.

Attempting pre-installed curl executable download:

docs@b829f755733f:~$ curl https://gmic.eu/img/sample_leno.png --output leno.png
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  166k  100  166k    0     0   763k      0 --:--:-- --:--:-- --:--:--  763k
docs@b829f755733f:~$ file leno.png 
leno.png: PNG image data, 512 x 512, 8-bit colormap, non-interlaced

[myselfhimself]

@dtschump feel free to comment if you have any idea

[myselfhimself]

I managed to install gmic 2.9.4 in the readthedocs docker image.

Sample image download using the gmic curl executable does work:

root@264e79af3c6b:/# gmic -v
[gmic]-0./ Start G'MIC interpreter.
[gmic]-0./ *** Error *** Command 'verbose': Invalid argument ''.
[gmic] Command 'verbose' has the following description: 

  gmic: GREYC's Magic for Image Computing: command-line interface
        Version 2.9.4
        (https://gmic.eu)

        Copyright (c) 2008-2020, David Tschumperlé / GREYC / CNRS.
        (https://www.greyc.fr)

  verbose (+):
      level |
      { + | - }

    Set or increment/decrement the verbosity level. Default level is 0.
    (equivalent to shortcut command 'v').

    When 'level'>0, G'MIC log messages are displayed on the standard error (stderr).

    Default value: 'level=1'.

root@264e79af3c6b:/# gmic sp leno
[gmic]-0./ Start G'MIC interpreter.
[gmic]-1./ Input sample image 'leno' (1 image 512x512x1x3).
[gmic]-1./ Print image [0] = 'leno'.
[0] = 'leno':
  size = (512,512,1,3) [3072 Kio of floats].
  data = (224,224,223,224,224,225,224,224,224,224,224,224,(...),69,85,85,79,87,79,85,90,77,77,79,84).
  min = 1, max = 255, mean = 128.318, std = 58.3599, coords_min = (508,71,0,1), coords_max = (124,189,0,0).
[gmic]-1./ End G'MIC interpreter.

[myselfhimself]

I will troubleshoot this myself with a stripped down libcurl piece of code, this is not a readthedocs issue I think. I can report back here.

[myselfhimself]

Hello, closing this as I found a solution: not linking the gmic module against libcurl, but using the curl executable, which by chance is also installed in your buildbot docker image.

Thanks for reading and supporting.
I will try readthedocs online builds with a bumped gmic release number later on this month. I will reopen this issue if needed.

Thank you for your kind time reading and support!

$ docker run -v `readlink -f wheelhouse`:/wheelhouse -it readthedocs/build bash
docs@26e0933478ec:/$ python3.7
Python 3.7.3 (default, Jan 24 2020, 02:22:02) 
[GCC 7.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> 
docs@26e0933478ec:/$ cd wheelhouse/
docs@26e0933478ec:/wheelhouse$ ls
gmic-2.9.1a5-cp36-cp36m-linux_x86_64.whl           gmic-2.9.1a5-cp37-cp37m-linux_x86_64.whl          out         wurlitzer-2.0.1-py2.py3-none-any.whl
gmic-2.9.1a5-cp36-cp36m-linux_x86_64.whl_repaired  gmic-2.9.1a5-cp37-cp37m-linux_x86_64.whl_old      rerepair
gmic-2.9.1a5-cp36-cp36m-manylinux2014_x86_64.whl   gmic-2.9.1a5-cp37-cp37m-manylinux2014_x86_64.whl  wheelhouse
docs@26e0933478ec:/wheelhouse$ pip
pip     pip2    pip2.7  pip3    pip3.5  pip3.6  pip3.7  pip3.8  
docs@26e0933478ec:/wheelhouse$ pip3.7  install gmic-2.9.1a5-cp37-cp37m-manylinux2014_x86_64.whl
Processing ./gmic-2.9.1a5-cp37-cp37m-manylinux2014_x86_64.whl
Collecting wurlitzer
  Downloading wurlitzer-2.0.1-py2.py3-none-any.whl (6.1 kB)
Installing collected packages: wurlitzer, gmic
Successfully installed gmic-2.9.1a5 wurlitzer-2.0.1
WARNING: You are using pip version 20.0.1; however, version 20.3.1 is available.
You should consider upgrading via the '/home/docs/.pyenv/versions/3.7.3/bin/python3.7 -m pip install --upgrade pip' command.
docs@26e0933478ec:/wheelhouse$ python3.7
Python 3.7.3 (default, Jan 24 2020, 02:22:02) 
[GCC 7.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import gmic
>>> gmic.run("sp print")
gmic-py: Working in display-less mode.
[gmic]-1./ Print image [0] = 'monkey'.
[0] = 'monkey':
  size = (640,600,1,3) [4500 Kio of floats].
  data = (33,35,35,35,35,35,35,35,35,35,35,30,(...),104,95,95,134,142,127,120,120,86,86,104,86).
  min = 2, max = 249, mean = 78.6035, std = 61.3154, coords_min = (80,0,0,0), coords_max = (428,127,0,1).
>>> import os
>>> os.listdir('/home/docs/.cache/gmic')
['sample_monkey.png']
>>> 
docs@26e0933478ec:/wheelhouse$ which curl
/usr/bin/curl