Merge pull request #123 from bird-house/proxy_read_timeout-config-sho…

…uld-be-configurable

proxy: proxy_read_timeout config should be configurable

We have a performance problem with the production deployment at Ouranos so we need a longer timeout.  Being a Ouranos specific need, it should not be hardcoded as in previous PR #122.

The previous increase was sometime not enough !

The value is now configurable via `env.local` as most other customizations.  Documentation updated.

Timeout in Prod:
```
WPS_URL=https://pavics.ouranos.ca/twitcher/ows/proxy/raven/wps FINCH_WPS_URL=https://pavics.ouranos.ca/twitcher/ows/proxy/finch/wps FLYINGPIGEON_WPS
_URL=https://pavics.ouranos.ca/twitcher/ows/proxy/flyingpigeon/wps pytest --nbval-lax --verbose docs/source/notebooks/Running_HMETS_with_CANOPEX_datas
et.ipynb --sanitize-with docs/source/output-sanitize.cfg --ignore docs/source/notebooks/.ipynb_checkpoints

HTTPError: 504 Server Error: Gateway Time-out for url: https://pavics.ouranos.ca/twitcher/ows/proxy/raven/wps

===================================================== 11 failed, 4 passed, 1 warning in 249.80s (0:04:09) ===========================================
```

Pass easily on my test VM with very modest hardware (10G ram, 2 cpu):
```
WPS_URL=https://lvupavicsmaster.ouranos.ca/twitcher/ows/proxy/raven/wps FINCH_WPS_URL=https://lvupavicsmaster.ouranos.ca/twitcher/ows/proxy/finch/wp
s FLYINGPIGEON_WPS_URL=https://lvupavicsmaster.ouranos.ca/twitcher/ows/proxy/flyingpigeon/wps pytest --nbval-lax --verbose docs/source/notebooks/Runni
ng_HMETS_with_CANOPEX_dataset.ipynb --sanitize-with docs/source/output-sanitize.cfg --ignore docs/source/notebooks/.ipynb_checkpoints

=========================================================== 15 passed, 1 warning in 33.84s ===========================================================
```

Pass against Medus:
```
WPS_URL=https://medus.ouranos.ca/twitcher/ows/proxy/raven/wps FINCH_WPS_URL=https://medus.ouranos.ca/twitcher/ows/proxy/finch/wps FLYINGPIGEON_WPS_URL=https://medus.ouranos.ca/twitcher/ows/proxy/flyingpigeon/wps pytest --nbval-lax --verbose docs/source/notebooks/Running_HMETS_with_CANOPEX_dataset.ipynb --sanitize-with docs/source/output-sanitize.cfg --ignore docs/source/notebooks/.ipynb_checkpoints

============================================== 15 passed, 1 warning in 42.44s =======================================================
```

Pass against `hirondelle.crim.ca`:
```
WPS_URL=https://hirondelle.crim.ca/twitcher/ows/proxy/raven/wps FINCH_WPS_URL=https://hirondelle.crim.ca/twitcher/ows/proxy/finch/wps FLYINGPIGEON_WPS_URL=https://hirondelle.crim.ca/twitcher/ows/proxy/flyingpigeon/wps pytest --nbval-lax --verbose docs/source/notebooks/Running_HMETS_with_CANOPEX_dataset.ipynb --sanitize-with docs/source/output-sanitize.cfg --ignore docs/source/notebooks/.ipynb_checkpoints

=============================================== 15 passed, 1 warning in 35.61s ===============================================
```

For comparison, a run on Prod without Twitcher (PR bird-house/birdhouse-deploy-ouranos#5):
```
WPS_URL=https://pavics.ouranos.ca/raven/wps FINCH_WPS_URL=https://pavics.ouranos.ca/twitcher/ows/proxy/finch/wps FLYINGPIGEON_WPS_URL=https://pavics
.ouranos.ca/twitcher/ows/proxy/flyingpigeon/wps pytest --nbval-lax --verbose docs/source/notebooks/Running_HMETS_with_CANOPEX_dataset.ipynb --sanitize
-with docs/source/output-sanitize.cfg --ignore docs/source/notebooks/.ipynb_checkpoints

HTTPError: 504 Server Error: Gateway Time-out for url: https://pavics.ouranos.ca/raven/wps

================================================ 11 failed, 4 passed, 1 warning in 248.99s (0:04:08) =================================================
```

A run on Prod without Twitcher and Nginx (direct hit Raven):
```
WPS_URL=http://pavics.ouranos.ca:8096/ FINCH_WPS_URL=https://pavics.ouranos.ca/twitcher/ows/proxy/finch/wps FLYINGPIGEON_WPS_URL=https://pavics.oura
nos.ca/twitcher/ows/proxy/flyingpigeon/wps pytest --nbval-lax --verbose docs/source/notebooks/Running_HMETS_with_CANOPEX_dataset.ipynb --sanitize-with
 docs/source/output-sanitize.cfg --ignore docs/source/notebooks/.ipynb_checkpoints

===================================================== 15 passed, 1 warning in 218.46s (0:03:38) ======================================================

birdhouse/README.rst

@@ -73,6 +73,10 @@ Note
   Nginx proxy, see `nginx.conf`_ (:download:`download <birdhouse/config/proxy/nginx.conf>`).
   Any WPS requests that will take longer should use the async mode.
 
+  Default value ``PROXY_READ_TIMEOUT_VALUE`` in `default.env`_ (:download:`download <birdhouse/default.env>`).
+
+  Overrideable in ``env.local`` file, as usual for all values in ``default.env`` file.
+
 
 Manual steps post deployment
 ----------------------------
@@ -241,3 +245,4 @@ Given a version number MAJOR.MINOR.PATCH, increment the:
 
 
 .. _nginx.conf: ./config/proxy/nginx.conf
+.. _default.env: ./default.env

birdhouse/config/proxy/.gitignore

@@ -0,0 +1 @@
+nginx.conf

birdhouse/config/proxy/nginx.conf → birdhouse/config/proxy/nginx.conf.template

@@ -31,7 +31,7 @@ http {
     client_body_timeout 600s;
 
     # timeout for reading a response from the proxied server
-    proxy_read_timeout 240s;  # default 60s
+    proxy_read_timeout ${PROXY_READ_TIMEOUT_VALUE};  # default 60s
 
     include /etc/nginx/conf.d/*.conf;
 

birdhouse/default.env

@@ -46,3 +46,7 @@ export AUTODEPLOY_PLATFORM_FREQUENCY="7 5 * * *"
 
 # Hourly
 export AUTODEPLOY_NOTEBOOK_FREQUENCY="@hourly"
+
+# Timeout for reading a response from the proxied server.
+# Any WPS processes taking longer than this should use async mode.
+export PROXY_READ_TIMEOUT_VALUE="240s"

birdhouse/pavics-compose.sh

@@ -47,6 +47,7 @@ OPTIONAL_VARS='
   $AUTODEPLOY_PLATFORM_FREQUENCY
   $AUTODEPLOY_NOTEBOOK_FREQUENCY
   $AUTODEPLOY_EXTRA_SCHEDULER_JOBS
+  $PROXY_READ_TIMEOUT_VALUE
 '
 
 # we switch to the real directory of the script, so it still works when used from $PATH