868 Incorporate X-Forwarded-Prefix into other GET multiendpoints v3

.env.dev

@@ -34,7 +34,6 @@ LAYMAN_GS_PATH=/geoserver/
 LAYMAN_GS_USER=layman
 LAYMAN_GS_PASSWORD=laymanpwd
 LAYMAN_GS_ROLE=LAYMAN_ROLE
-LAYMAN_GS_PROXY_BASE_URL=http://localhost:8000/geoserver/
 LAYMAN_GS_AUTHN_HTTP_HEADER_ATTRIBUTE=e58e4774e3dc7d6443ad59a8202c5ee0
 LAYMAN_GS_NORMALIZED_RASTER_DIRECTORY=normalized_raster_data_dev
 

.env.test

@@ -34,7 +34,6 @@ LAYMAN_GS_PATH=/geoserver/
 LAYMAN_GS_USER=layman_test
 LAYMAN_GS_PASSWORD=laymanpwd
 LAYMAN_GS_ROLE=LAYMAN_TEST_ROLE
-LAYMAN_GS_PROXY_BASE_URL=http://localhost:8000/geoserver/
 LAYMAN_GS_AUTHN_HTTP_HEADER_ATTRIBUTE=e58e4774e3dc7d6443ad59a8202c5ee0
 LAYMAN_GS_NORMALIZED_RASTER_DIRECTORY=normalized_raster_data_test
 
@@ -58,7 +57,7 @@ LAYMAN_SKIP_REDIS_LOADING=false
 
 # Flask-style server name (domain and port)
 LAYMAN_SERVER_NAME=layman_test_run_1:8000
-LAYMAN_PROXY_SERVER_NAME=enjoychallenge.tech
+LAYMAN_PROXY_SERVER_NAME=localhost:8000
 
 # internal settings
 LAYMAN_TIMGEN_URL=http://timgen:8080/

CHANGELOG.md

@@ -3,13 +3,15 @@
 ## v1.22.0
  {release-date}
 ### Upgrade requirements
+- Stop using environment variable `LAYMAN_GS_PROXY_BASE_URL`, it has no effect to Layman anymore.
+  - GeoServer's [Proxy Base URL](https://docs.geoserver.org/2.21.x/en/user/configuration/globalsettings.html) is now automatically set by Layman on each start. Value is automatically derived from environment variables [`LAYMAN_CLIENT_PUBLIC_URL`](doc/env-settings.md#layman_client_public_url) (protocol), [`LAYMAN_PROXY_SERVER_NAME`](doc/env-settings.md#layman_proxy_server_name) (domain and port), and [`LAYMAN_GS_PATH`](doc/env-settings.md#layman_gs_path) (path).
 ### Migrations and checks
 #### Schema migrations
 #### Data migrations
 - [#765](https://github.com/LayerManager/layman/issues/765) Fix `issuer_id` value in `users` table that was broken since v1.21.0.
 - [#765](https://github.com/LayerManager/layman/issues/765) Remove `authn.txt` files from workspace directories. The same information as in `authn.txt` files is saved in prime DB schema.
 ### Changes
-- [#868](https://github.com/LayerManager/layman/issues/868) Endpoint [GET Publications](doc/rest.md#get-publications) respects [HTTP header `X-Forwarded-Prefix`](doc/client-proxy.md#x-forwarded-prefix-http-header) of the request in response in `url` key.
+- [#868](https://github.com/LayerManager/layman/issues/868) Endpoints [GET Publications](doc/rest.md#get-publications), [GET Layers](doc/rest.md#get-layers), [GET Workspace Layers](doc/rest.md#get-workspace-layers) and [GET Maps](doc/rest.md#get-maps) [GET Workspace Maps](doc/rest.md#get-workspace-maps) respects [HTTP header `X-Forwarded-Prefix`](doc/client-proxy.md#x-forwarded-prefix-http-header) of the request in response in `url` key. Endpoint [GET Workspace Layer](doc/rest.md#get-workspace-layer) respects it in `url`, `thumbnail`.`url`, `metadata`.`comparison_url`, `wms`.`url`, `wfs`.`url` and `style`.`url` keys. Endpoint [GET Workspace Map](doc/rest.md#get-workspace-map) respects it in `url`, `file`.`url`, `thumbnail`.`url` and `metadata`.`comparison_url` keys.
 - [#880](https://github.com/LayerManager/layman/issues/880) Use Docker Compose v2 (`docker compose`) in Makefile without `compatibility` flag and remove `Makefile_docker-compose_v1` file. Docker containers are named according to Docker Compose v2 and may have different name after upgrade.
 - [#765](https://github.com/LayerManager/layman/issues/765) Stop saving OAuth2 claims in filesystem, use prime DB schema only.
 - [#893](https://github.com/LayerManager/layman/issues/893) It is possible to specify logging level by new environment variable [LAYMAN_LOGLEVEL](doc/env-settings.md#LAYMAN_LOGLEVEL). Default level is `INFO`.
@@ -733,7 +735,7 @@ There is a critical bug in this release, posting new layer breaks Layman: https:
 - [#74](https://github.com/LayerManager/layman/issues/74) Layman user and role at GeoServer defined by [LAYMAN_GS_USER](doc/env-settings.md#LAYMAN_GS_USER) and [LAYMAN_GS_ROLE](doc/env-settings.md#LAYMAN_GS_ROLE) are now created automatically on Layman's startup if an only if new environment variable [GEOSERVER_ADMIN_PASSWORD](doc/env-settings.md#GEOSERVER_ADMIN_PASSWORD) is provided. There is no need to set [GEOSERVER_ADMIN_PASSWORD](doc/env-settings.md#GEOSERVER_ADMIN_PASSWORD) for other reason than automatically creating Layman user and Layman role.
    - No change is required. If you are migrating existing instance, Layman user and role are already created, so you don't need to set [GEOSERVER_ADMIN_PASSWORD](doc/env-settings.md#GEOSERVER_ADMIN_PASSWORD). If this is your first Layman release, [GEOSERVER_ADMIN_PASSWORD](doc/env-settings.md#GEOSERVER_ADMIN_PASSWORD) is set in `.env` files starting with this version, so Layman user and role at GeoServer will be automatically created on startup.
    - No need to run command `make geoserver-reset-default-datadir` from now on. This command was removed from make options.
-- [#62](https://github.com/LayerManager/layman/issues/62) GeoServer [Proxy Base URL](https://docs.geoserver.org/2.21.x/en/user/configuration/globalsettings.html) is now automatically set on Layman's startup according to [LAYMAN_GS_PROXY_BASE_URL](doc/env-settings.md#LAYMAN_GS_PROXY_BASE_URL). If you do not set the variable, value is calculated as [LAYMAN_CLIENT_PUBLIC_URL](doc/env-settings.md#LAYMAN_CLIENT_PUBLIC_URL)+[LAYMAN_GS_PATH](doc/env-settings.md#LAYMAN_GS_PATH). If you set it to empty string, no change of Proxy Base URL will be done on GeoServer side.
+- [#62](https://github.com/LayerManager/layman/issues/62) GeoServer [Proxy Base URL](https://docs.geoserver.org/2.21.x/en/user/configuration/globalsettings.html) is now automatically set on Layman's startup according to [LAYMAN_GS_PROXY_BASE_URL](https://github.com/LayerManager/layman/blob/v1.21.1/doc/env-settings.md#LAYMAN_GS_PROXY_BASE_URL). If you do not set the variable, value is calculated as [LAYMAN_CLIENT_PUBLIC_URL](doc/env-settings.md#LAYMAN_CLIENT_PUBLIC_URL)+[LAYMAN_GS_PATH](doc/env-settings.md#LAYMAN_GS_PATH). If you set it to empty string, no change of Proxy Base URL will be done on GeoServer side.
 - [#83](https://github.com/LayerManager/layman/issues/89) All layers are created as `GEOMETRY` type, so any other type can be added (for example polygons can be added to points).
 - [#73](https://github.com/LayerManager/layman/issues/73) Layman users are automatically created on GeoServer (either at start up of Layman or when reserved) with separate role and workspace. Username is the same as in Layman, name of role is `"USER_"+username`, name of workspace is the same as username. Read and write permissions for workspace are set according to Layman's authorization (as of now read-everyone-write-everyone or read-everyone-write-owner).
 - New environment variables [LAYMAN_GS_USER_GROUP_SERVICE](doc/env-settings.md#LAYMAN_GS_USER_GROUP_SERVICE) and [LAYMAN_GS_ROLE_SERVICE](doc/env-settings.md#LAYMAN_GS_ROLE_SERVICE) enable to control which user/group and role services are used at GeoServer. Not setting these variables means to use default services. 

doc/client-proxy.md

@@ -43,3 +43,23 @@ For example, if you send request to `/layman-client-proxy/rest/publications` wit
 Currently, value of `X-Forwarded-Prefix` affects following URLs:
 * [GET Publications](rest.md#get-publications)
   * `url` key
+* [GET Layers](rest.md#get-layers)
+  * `url` key
+* [GET Workspace Layers](rest.md#get-workspace-layers)
+  * `url` key
+* [GET Maps](rest.md#get-maps)
+  * `url` key
+* [GET Workspace Maps](rest.md#get-workspace-maps)
+  * `url` key
+* [GET Workspace Layer](rest.md#get-workspace-layer)
+  * `url` key
+  * `wms`.`url` key
+  * `wfs`.`url` key
+  * `style`.`url` key
+  * `thumbnail`.`url` key
+  * `metadata`.`comparison_url` key
+* [GET Workspace Map](rest.md#get-workspace-map)
+  * `url` key
+  * `file`.`url` key
+  * `thumbnail`.`url` key
+  * `metadata`.`comparison_url` key

doc/env-settings.md

@@ -167,9 +167,6 @@ Password of [LAYMAN_GS_USER](#LAYMAN_GS_USER).
 ### LAYMAN_GS_ROLE
 Name of [GeoServer role](https://docs.geoserver.org/2.21.x/en/user/security/webadmin/ugr.html#edit-role-service) of [LAYMAN_GS_USER](#LAYMAN_GS_USER). The role is used to create explicit [access rule](https://docs.geoserver.org/2.21.x/en/user/security/layer.html) for all layers published by Layman. The LAYMAN_GS_ROLE must be another role than default [`ADMIN`](https://docs.geoserver.org/2.21.x/en/user/security/usergrouprole/roleservices.html#mapping-roles-to-system-roles) role (defined by `adminRoleName`)! The role will be created automatically if [GEOSERVER_ADMIN_PASSWORD](#GEOSERVER_ADMIN_PASSWORD) is provided.
 
-### LAYMAN_GS_PROXY_BASE_URL
-GeoServer [Proxy Base URL](https://docs.geoserver.org/2.21.x/en/user/configuration/globalsettings.html). It is automatically set on Layman's startup. If you do not set the variable, value is calculated as protocol, host, and port of [LAYMAN_CLIENT_PUBLIC_URL](#LAYMAN_CLIENT_PUBLIC_URL) followed by [LAYMAN_GS_PATH](#LAYMAN_GS_PATH). If you set it to empty string, no change of Proxy Base URL will be done on GeoServer side.
-
 ### LAYMAN_GS_USER_GROUP_SERVICE
 Name of [user/group service](https://docs.geoserver.org/2.21.x/en/user/security/usergrouprole/usergroupservices.html) used for managing users at GeoServer. If not set (default), the service named `default` is chosen. Usually it's [XML user/group service](https://docs.geoserver.org/2.21.x/en/user/security/usergrouprole/usergroupservices.html#xml-user-group-service).
 

src/layman/__init__.py

@@ -91,8 +91,7 @@
                         gs_util.ensure_user_role(settings.LAYMAN_GS_USER, 'ADMIN', settings.GEOSERVER_ADMIN_AUTH)
                         gs_util.ensure_user_role(settings.LAYMAN_GS_USER, settings.LAYMAN_GS_ROLE, settings.GEOSERVER_ADMIN_AUTH)
 
-                    if settings.LAYMAN_GS_PROXY_BASE_URL != '':
-                        gs_util.ensure_proxy_base_url(settings.LAYMAN_GS_PROXY_BASE_URL, settings.LAYMAN_GS_AUTH)
+                    gs_util.ensure_proxy_base_url(settings.LAYMAN_GS_PROXY_BASE_URL, settings.LAYMAN_GS_AUTH)
 
                     if not IN_UPGRADE_PROCESS:
                         logger.info(f'Adjusting GeoServer SRS')

src/layman/layer/filesystem/thumbnail.py

@@ -32,13 +32,14 @@ def ensure_layer_thumbnail_dir(workspace, layername):
     return thumbnail_dir
 
 
-def get_layer_info(workspace, layername):
+def get_layer_info(workspace, layername, *, x_forwarded_prefix=None):
     thumbnail_path = get_layer_thumbnail_path(workspace, layername)
     if os.path.exists(thumbnail_path):
         return {
             'thumbnail': {
                 'url': url_for('rest_workspace_layer_thumbnail.get', workspace=workspace,
-                               layername=layername),
+                               layername=layername,
+                               x_forwarded_prefix=x_forwarded_prefix),
                 'path': os.path.relpath(thumbnail_path, common_util.get_workspace_dir(workspace))
             },
             '_thumbnail': {

src/layman/layer/geoserver/sld.py

@@ -41,10 +41,10 @@ def delete_layer(workspace, layername):
     return result
 
 
-def get_layer_info(workspace, layername):
+def get_layer_info(workspace, layername, *, x_forwarded_prefix=None):
     response = get_style_response(workspace, layername, gs_util.headers_sld['1.0.0'], settings.LAYMAN_GS_AUTH)
     if response and response.status_code == 200:
-        url = url_for('rest_workspace_layer_style.get', workspace=workspace, layername=layername)
+        url = url_for('rest_workspace_layer_style.get', workspace=workspace, layername=layername, x_forwarded_prefix=x_forwarded_prefix)
         info = {
             'style': {
                 'url': url,

src/layman/layer/geoserver/util.py

@@ -19,6 +19,11 @@ def get_gs_proxy_base_url():
     return proxy_base_url
 
 
+def get_gs_proxy_server_url():
+    proxy_base_url = f'{settings.LAYMAN_PUBLIC_URL_SCHEME}://{settings.LAYMAN_PROXY_SERVER_NAME}'
+    return proxy_base_url
+
+
 def wms_proxy(wms_url, xml=None, version=None, headers=None):
     from layman.layer.geoserver.wms import VERSION
     version = version or VERSION

src/layman/layer/geoserver/wfs.py

@@ -9,7 +9,7 @@
 from layman import util as layman_util
 from layman.layer import LAYER_TYPE
 import requests_util.retry
-from .util import get_gs_proxy_base_url, get_external_db_store_name
+from .util import get_gs_proxy_server_url, get_external_db_store_name
 from . import wms
 
 FLASK_PROXY_KEY = f'{__name__}:PROXY:{{workspace}}'
@@ -58,8 +58,10 @@ def delete_layer(workspace, layername):
     return {}
 
 
-def get_wfs_url(workspace, external_url=False):
-    base_url = get_gs_proxy_base_url() if external_url else settings.LAYMAN_GS_URL
+def get_wfs_url(workspace, external_url=False, *, x_forwarded_prefix=None):
+    assert external_url or not x_forwarded_prefix
+    x_forwarded_prefix = x_forwarded_prefix or ''
+    base_url = urljoin(get_gs_proxy_server_url(), x_forwarded_prefix) + settings.LAYMAN_GS_PATH if external_url else settings.LAYMAN_GS_URL
     return urljoin(base_url, workspace + '/wfs')
 
 
@@ -122,11 +124,11 @@ def clear_cache(workspace):
     mem_redis.delete(key)
 
 
-def get_layer_info(workspace, layername):
+def get_layer_info(workspace, layername, *, x_forwarded_prefix=None):
     wfs = get_wfs_proxy(workspace)
     if wfs is None:
         return {}
-    wfs_proxy_url = get_wfs_url(workspace, external_url=True)
+    wfs_proxy_url = get_wfs_url(workspace, external_url=True, x_forwarded_prefix=x_forwarded_prefix)
 
     wfs_layername = f"{workspace}:{layername}"
     if wfs_layername not in wfs.contents:

src/layman/layer/geoserver/wms.py

@@ -14,7 +14,7 @@
 from layman.layer import LAYER_TYPE
 from layman.layer.filesystem import gdal
 import requests_util.retry
-from .util import get_gs_proxy_base_url, get_external_db_store_name
+from .util import get_gs_proxy_server_url, get_external_db_store_name
 
 FLASK_PROXY_KEY = f'{__name__}:PROXY:{{workspace}}'
 DEFAULT_WMS_QGIS_STORE_PREFIX = 'qgis'
@@ -82,9 +82,11 @@ def delete_layer(workspace, layername):
     return {}
 
 
-def get_wms_url(workspace, external_url=False):
+def get_wms_url(workspace, external_url=False, *, x_forwarded_prefix=None):
+    assert external_url or not x_forwarded_prefix
+    x_forwarded_prefix = x_forwarded_prefix or ''
     geoserver_workspace = get_geoserver_workspace(workspace)
-    base_url = get_gs_proxy_base_url() if external_url else settings.LAYMAN_GS_URL
+    base_url = urljoin(get_gs_proxy_server_url(), x_forwarded_prefix) + settings.LAYMAN_GS_PATH if external_url else settings.LAYMAN_GS_URL
     return urljoin(base_url, geoserver_workspace + '/ows')
 
 
@@ -167,11 +169,11 @@ def get_timeregex_props(workspace, layername):
     return result
 
 
-def get_layer_info(workspace, layername):
+def get_layer_info(workspace, layername, *, x_forwarded_prefix=None):
     wms = get_wms_proxy(workspace)
     if wms is None:
         return {}
-    wms_proxy_url = get_wms_url(workspace, external_url=True)
+    wms_proxy_url = get_wms_url(workspace, external_url=True, x_forwarded_prefix=x_forwarded_prefix)
 
     if layername not in wms.contents:
         return {}

src/layman/layer/micka/csw.py

@@ -28,7 +28,7 @@
 post_layer = empty_method
 
 
-def get_layer_info(workspace, layername):
+def get_layer_info(workspace, layername, *, x_forwarded_prefix=None):
     uuid = get_layer_uuid(workspace, layername)
     try:
         csw = common_util.create_csw()
@@ -50,7 +50,8 @@ def get_layer_info(workspace, layername):
                 'identifier': muuid,
                 'csw_url': settings.CSW_PROXY_URL,
                 'record_url': common_util.get_metadata_url(uuid, url_type=common_util.RecordUrlType.BASIC),
-                'comparison_url': url_for('rest_workspace_layer_metadata_comparison.get', workspace=workspace, layername=layername),
+                'comparison_url': url_for('rest_workspace_layer_metadata_comparison.get', workspace=workspace, layername=layername,
+                                          x_forwarded_prefix=x_forwarded_prefix),
             }
         }
     return {}

src/layman/layer/micka/util_test_filled_template.xml

@@ -122,7 +122,7 @@
       <gmd:graphicOverview>
         <gmd:MD_BrowseGraphic>
           <gmd:fileName>
-            <gco:CharacterString>http://enjoychallenge.tech/rest/workspaces/browser/layers/layer/thumbnail</gco:CharacterString>
+            <gco:CharacterString>http://localhost:8000/rest/workspaces/browser/layers/layer/thumbnail</gco:CharacterString>
           </gmd:fileName>
           <gmd:fileType>
             <gco:CharacterString>PNG</gco:CharacterString>
@@ -261,7 +261,7 @@
           <gmd:onLine>
             <gmd:CI_OnlineResource>
               <gmd:linkage>
-                <gmd:URL>http://enjoychallenge.tech/rest/workspaces/browser/layers/layer</gmd:URL>
+                <gmd:URL>http://localhost:8000/rest/workspaces/browser/layers/layer</gmd:URL>
               </gmd:linkage>
               <gmd:protocol>
                 <gmx:Anchor xlink:href="https://services.cuzk.cz/registry/codelist/OnlineResourceProtocolValue/WWW:LINK-1.0-http--link">WWW:LINK-1.0-http--link</gmx:Anchor>

src/layman/layer/qgis/wms.py

@@ -18,11 +18,11 @@
 get_publication_uuid = empty_method_returns_none
 
 
-def get_layer_info(workspace, layername):
+def get_layer_info(workspace, layername, *, x_forwarded_prefix=None):
     input_file_dir = qgis.get_layer_dir(workspace, layername)
     result = {}
     if os.path.exists(input_file_dir):
-        url = layman_util.url_for('rest_workspace_layer_style.get', workspace=workspace, layername=layername)
+        url = layman_util.url_for('rest_workspace_layer_style.get', workspace=workspace, layername=layername, x_forwarded_prefix=x_forwarded_prefix)
         result = {
             'name': layername,
             'style': {

src/layman/layer/rest_layers.py

@@ -1,5 +1,6 @@
 from flask import Blueprint, g, request, current_app as app
 
+from layman import util as layman_util
 from layman.authn import authenticate, get_authn_username
 from layman.authz import authorize_publications_decorator
 from layman.common import rest as rest_common
@@ -20,4 +21,5 @@ def get():
     app.logger.info(f"GET Layers, actor={g.user}")
 
     actor = get_authn_username()
-    return rest_common.get_publications(LAYER_TYPE, actor, request_args=request.args)
+    x_forwarded_prefix = layman_util.get_x_forwarded_prefix(request.headers)
+    return rest_common.get_publications(LAYER_TYPE, actor, request_args=request.args, x_forwarded_prefix=x_forwarded_prefix)

src/layman/layer/rest_workspace_layer.py

@@ -37,7 +37,8 @@ def get(workspace, layername):
     # pylint: disable=unused-argument
     app.logger.info(f"GET Layer, actor={g.user}")
 
-    info = util.get_complete_layer_info(workspace, layername)
+    x_forwarded_prefix = layman_util.get_x_forwarded_prefix(request.headers)
+    info = util.get_complete_layer_info(workspace, layername, x_forwarded_prefix=x_forwarded_prefix)
 
     return jsonify(info), 200
 

src/layman/layer/rest_workspace_layers.py

@@ -32,7 +32,8 @@ def get(workspace):
     app.logger.info(f"GET Layers, actor={g.user}")
 
     actor = get_authn_username()
-    return rest_common.get_publications(LAYER_TYPE, actor, request_args=request.args, workspace=workspace)
+    x_forwarded_prefix = layman_util.get_x_forwarded_prefix(request.headers)
+    return rest_common.get_publications(LAYER_TYPE, actor, request_args=request.args, workspace=workspace, x_forwarded_prefix=x_forwarded_prefix)
 
 
 @bp.route(f"/{LAYER_REST_PATH_NAME}", methods=['POST'])