868 Return also Vector-WFS relations in get_layers_from_json

.env.demo

@@ -30,7 +30,6 @@ LAYMAN_PRIME_SCHEMA=_prime_schema
 GEOSERVER_ADMIN_PASSWORD=geoserver
 LAYMAN_GS_HOST=geoserver
 LAYMAN_GS_PORT=8080
-LAYMAN_GS_PATH=/geoserver/
 LAYMAN_GS_USER=layman
 LAYMAN_GS_PASSWORD=laymanpwd
 LAYMAN_GS_ROLE=LAYMAN_ROLE

.env.dev

@@ -30,7 +30,6 @@ LAYMAN_PRIME_SCHEMA=_prime_schema
 GEOSERVER_ADMIN_PASSWORD=geoserver
 LAYMAN_GS_HOST=geoserver
 LAYMAN_GS_PORT=8080
-LAYMAN_GS_PATH=/geoserver/
 LAYMAN_GS_USER=layman
 LAYMAN_GS_PASSWORD=laymanpwd
 LAYMAN_GS_ROLE=LAYMAN_ROLE

.env.test

@@ -30,7 +30,6 @@ LAYMAN_PRIME_SCHEMA=_prime_schema
 GEOSERVER_ADMIN_PASSWORD=geoserver
 LAYMAN_GS_HOST=geoserver
 LAYMAN_GS_PORT=8080
-LAYMAN_GS_PATH=/geoserver/
 LAYMAN_GS_USER=layman_test
 LAYMAN_GS_PASSWORD=laymanpwd
 LAYMAN_GS_ROLE=LAYMAN_TEST_ROLE

CHANGELOG.md

@@ -4,14 +4,16 @@
  {release-date}
 ### Upgrade requirements
 - Stop using environment variable `LAYMAN_GS_PROXY_BASE_URL`, it has no effect to Layman anymore.
-  - GeoServer's [Proxy Base URL](https://docs.geoserver.org/2.21.x/en/user/configuration/globalsettings.html) is now automatically set by Layman on each start. Value is automatically derived from environment variables [`LAYMAN_CLIENT_PUBLIC_URL`](doc/env-settings.md#layman_client_public_url) (protocol), [`LAYMAN_PROXY_SERVER_NAME`](doc/env-settings.md#layman_proxy_server_name) (domain and port), and [`LAYMAN_GS_PATH`](doc/env-settings.md#layman_gs_path) (path).
+  - GeoServer's [Proxy Base URL](https://docs.geoserver.org/2.21.x/en/user/configuration/globalsettings.html) is now automatically set by Layman on each start. Value is automatically derived from environment variables [`LAYMAN_CLIENT_PUBLIC_URL`](doc/env-settings.md#layman_client_public_url) (protocol) and [`LAYMAN_PROXY_SERVER_NAME`](doc/env-settings.md#layman_proxy_server_name) (domain and port). URL path is always `/geoserver/`.
+- Stop using environment variable `LAYMAN_GS_PATH`, it has no effect to Layman anymore.
+  - GeoServer's URL path must be always `/geoserver/` (that is true for GeoServer shipped with Layman).
 ### Migrations and checks
 #### Schema migrations
 - [#868](https://github.com/LayerManager/layman/issues/868) Create new table `map_layer` in prime DB schema.
 #### Data migrations
 - [#765](https://github.com/LayerManager/layman/issues/765) Fix `issuer_id` value in `users` table that was broken since v1.21.0.
 - [#765](https://github.com/LayerManager/layman/issues/765) Remove `authn.txt` files from workspace directories. The same information as in `authn.txt` files is saved in prime DB schema.
-- [#868](https://github.com/LayerManager/layman/issues/868) Fill table `map_layer` with relations between maps and their layers published on this Layman instance. Relations to layers from other servers are not imported into the table.
+- [#868](https://github.com/LayerManager/layman/issues/868) Fill table `map_layer` with relations between maps and [internal layers](doc/models.md#internal-map-layer) (layers published on this Layman instance). Relations to [external layers](doc/models.md#internal-map-layer) (layers of other servers) are not imported into the table.
 ### Changes
 - [#868](https://github.com/LayerManager/layman/issues/868) Endpoints [GET Publications](doc/rest.md#get-publications), [GET Layers](doc/rest.md#get-layers), [GET Workspace Layers](doc/rest.md#get-workspace-layers), [GET Maps](doc/rest.md#get-maps), [GET Workspace Maps](doc/rest.md#get-workspace-maps), [GET Workspace Layer](doc/rest.md#get-workspace-layer), [GET Workspace Map](doc/rest.md#get-workspace-map), [POST Workspace Layers](doc/rest.md#post-workspace-layers), [DELETE Workspace Layer](doc/rest.md#delete-workspace-layer) and [DELETE Workspace Layers](doc/rest.md#delete-workspace-layers) respects [HTTP header `X-Forwarded-Prefix`](doc/client-proxy.md#x-forwarded-prefix-http-header) of the request in the response.
 - [#880](https://github.com/LayerManager/layman/issues/880) Use Docker Compose v2 (`docker compose`) in Makefile without `compatibility` flag and remove `Makefile_docker-compose_v1` file. Docker containers are named according to Docker Compose v2 and may have different name after upgrade.
@@ -737,7 +739,7 @@ There is a critical bug in this release, posting new layer breaks Layman: https:
 - [#74](https://github.com/LayerManager/layman/issues/74) Layman user and role at GeoServer defined by [LAYMAN_GS_USER](doc/env-settings.md#LAYMAN_GS_USER) and [LAYMAN_GS_ROLE](doc/env-settings.md#LAYMAN_GS_ROLE) are now created automatically on Layman's startup if an only if new environment variable [GEOSERVER_ADMIN_PASSWORD](doc/env-settings.md#GEOSERVER_ADMIN_PASSWORD) is provided. There is no need to set [GEOSERVER_ADMIN_PASSWORD](doc/env-settings.md#GEOSERVER_ADMIN_PASSWORD) for other reason than automatically creating Layman user and Layman role.
    - No change is required. If you are migrating existing instance, Layman user and role are already created, so you don't need to set [GEOSERVER_ADMIN_PASSWORD](doc/env-settings.md#GEOSERVER_ADMIN_PASSWORD). If this is your first Layman release, [GEOSERVER_ADMIN_PASSWORD](doc/env-settings.md#GEOSERVER_ADMIN_PASSWORD) is set in `.env` files starting with this version, so Layman user and role at GeoServer will be automatically created on startup.
    - No need to run command `make geoserver-reset-default-datadir` from now on. This command was removed from make options.
-- [#62](https://github.com/LayerManager/layman/issues/62) GeoServer [Proxy Base URL](https://docs.geoserver.org/2.21.x/en/user/configuration/globalsettings.html) is now automatically set on Layman's startup according to [LAYMAN_GS_PROXY_BASE_URL](https://github.com/LayerManager/layman/blob/v1.21.1/doc/env-settings.md#LAYMAN_GS_PROXY_BASE_URL). If you do not set the variable, value is calculated as [LAYMAN_CLIENT_PUBLIC_URL](doc/env-settings.md#LAYMAN_CLIENT_PUBLIC_URL)+[LAYMAN_GS_PATH](doc/env-settings.md#LAYMAN_GS_PATH). If you set it to empty string, no change of Proxy Base URL will be done on GeoServer side.
+- [#62](https://github.com/LayerManager/layman/issues/62) GeoServer [Proxy Base URL](https://docs.geoserver.org/2.21.x/en/user/configuration/globalsettings.html) is now automatically set on Layman's startup according to [LAYMAN_GS_PROXY_BASE_URL](https://github.com/LayerManager/layman/blob/v1.21.1/doc/env-settings.md#LAYMAN_GS_PROXY_BASE_URL). If you do not set the variable, value is calculated as [LAYMAN_CLIENT_PUBLIC_URL](doc/env-settings.md#LAYMAN_CLIENT_PUBLIC_URL)+[LAYMAN_GS_PATH](https://github.com/LayerManager/layman/blob/v1.21.1/doc/env-settings.md#LAYMAN_GS_PATH). If you set it to empty string, no change of Proxy Base URL will be done on GeoServer side.
 - [#83](https://github.com/LayerManager/layman/issues/89) All layers are created as `GEOMETRY` type, so any other type can be added (for example polygons can be added to points).
 - [#73](https://github.com/LayerManager/layman/issues/73) Layman users are automatically created on GeoServer (either at start up of Layman or when reserved) with separate role and workspace. Username is the same as in Layman, name of role is `"USER_"+username`, name of workspace is the same as username. Read and write permissions for workspace are set according to Layman's authorization (as of now read-everyone-write-everyone or read-everyone-write-owner).
 - New environment variables [LAYMAN_GS_USER_GROUP_SERVICE](doc/env-settings.md#LAYMAN_GS_USER_GROUP_SERVICE) and [LAYMAN_GS_ROLE_SERVICE](doc/env-settings.md#LAYMAN_GS_ROLE_SERVICE) enable to control which user/group and role services are used at GeoServer. Not setting these variables means to use default services. 

README.md

@@ -123,7 +123,7 @@ Also, anytime you change `.env` file, remember to rebuild docker images as some
 
 
 ## Dependencies
-Layman has [many dependencies](doc/dependencies.md). Most of them are shipped with Layman. However there are some **external dependencies** that should be treated carefully:
+Layman has [many dependencies](doc/dependencies.md). Most of them are shipped with Layman. However, there are some **external dependencies** that should be treated carefully:
 - PostgreSQL & PostGIS
 - QGIS Server
 - GeoServer
@@ -177,7 +177,7 @@ Within PostgreSQL, you need to provide one database for Layman and one database
 
 Within QGIS Server, you do not need to provide anything special.
 
-Within GeoServer, you need to provide either admin password [GEOSERVER_ADMIN_PASSWORD](doc/env-settings.md#GEOSERVER_ADMIN_PASSWORD), or one Layman user [LAYMAN_GS_USER](doc/env-settings.md#LAYMAN_GS_USER) and one layman role [LAYMAN_GS_ROLE](doc/env-settings.md#LAYMAN_GS_ROLE). If admin password is provided, Layman will create the Layman user and the Layman role automatically.
+Within GeoServer, you need to provide either admin password [GEOSERVER_ADMIN_PASSWORD](doc/env-settings.md#GEOSERVER_ADMIN_PASSWORD), or one Layman user [LAYMAN_GS_USER](doc/env-settings.md#LAYMAN_GS_USER) and one layman role [LAYMAN_GS_ROLE](doc/env-settings.md#LAYMAN_GS_ROLE). If admin password is provided, Layman will create the Layman user and the Layman role automatically. URL path of the GeoServer must be `/geoserver/`.
 
 Within Redis, you need to provide two databases, one for Layman, second for Layman Test Client. Connection strings are defined by [LAYMAN_REDIS_URL](doc/env-settings.md#LAYMAN_REDIS_URL) and [LTC_REDIS_URL](doc/env-settings.md#LTC_REDIS_URL).
 

doc/data-storage.md

@@ -51,7 +51,7 @@ Information about [maps](models.md#map) includes JSON definition.
 
 When user [publishes new map](rest.md#post-workspace-maps)
 - UUID and name is saved to [Redis](#redis) and [filesystem](#filesystem),
-- UUID, name, title and access rights are saved to [PostgreSQL](#postgresql),
+- UUID, name, title, access rights, and relations to [internal layers](models.md#internal-map-layer) are saved to [PostgreSQL](#postgresql),
 - JSON file is saved to [filesystem](#filesystem),
 - and asynchronous [tasks](#tasks) are saved in [Redis](#redis).
 

doc/env-settings.md

@@ -155,9 +155,6 @@ Internal URL host of GeoServer instance.
 ### LAYMAN_GS_PORT
 Internal URL port of GeoServer instance.
 
-### LAYMAN_GS_PATH
-URL path of GeoServer instance.
-
 ### LAYMAN_GS_USER
 Name of [GeoServer user](https://docs.geoserver.org/2.21.x/en/user/security/webadmin/ugr.html#add-user) that Layman uses for authentication and communication with GeoServer. The LAYMAN_GS_USER must be another user than default `admin` user. The LAYMAN_GS_USER user must have at least the [LAYMAN_GS_ROLE](#LAYMAN_GS_ROLE) and default [`ADMIN`](https://docs.geoserver.org/2.21.x/en/user/security/usergrouprole/roleservices.html#mapping-roles-to-system-roles) role (defined by `adminRoleName`). The user and his required roles will be created automatically on Layman's startup if [GEOSERVER_ADMIN_PASSWORD](#GEOSERVER_ADMIN_PASSWORD) is provided.
 

doc/models.md

@@ -41,7 +41,8 @@
 - Also referred to as **map composition**
 - Map is [publication](#publication) defined by JSON valid against [map-composition schema](https://github.com/hslayers/map-compositions) version 2 used by [Hslayers-ng](https://github.com/hslayers/hslayers-ng)
 - Map is collection of WMS layers and vector data
-- Maps composed from WMS layers only are fully supported
+- Maps composed of WMS layers only are fully supported
+- Each layer is either [internal](#internal-map-layer), or [external](#external-map-layer).
 - Documented [map publishing](publish-map.md) process 
 - Thumbnail image available
 - Map-related data is named and structured
@@ -55,6 +56,15 @@
   - one thumbnail per map
   - one metadata record per map
 
+### Internal map layer
+- Internal map layer is layer of the [map](#map) with valid [workspace](#workspace) name, valid [layer](#layer) name, and
+    - whose `className` is `WMS` (or ends with `.WMS`) and whose `url` points to the Layman instance,
+    - or whose `className` is `Vector` (or ends with `.Vector`), whose `protocol.format` is `WFS` (or ends with `.WFS`) and whose `protocol.url` points to the Layman instance.
+- Map layer is considered internal even if [layer](#layer) with given name in given [workspace](#workspace) does not currently exist in the Layman instance.
+
+### External map layer
+- External map layer is layer of the [map](#map) that is not [internal](#internal-map-layer).
+
 ## User
 - User is any person who communicates with Layman REST API through any client.
 - User can be either authenticated, or unauthenticated (i.e. anonymous).

sample/layman.map/internal_external_edge_cases.json

@@ -0,0 +1,104 @@
+{
+  "describedBy": "https://raw.githubusercontent.com/hslayers/map-compositions/2.0.0/schema.json",
+  "schema_version": "2.0.0",
+  "abstract": "Hranice",
+  "title": "Hranice",
+  "extent": [
+    -35.0,
+    -48.5,
+    179,
+    81.5
+  ],
+  "nativeExtent": [
+    -3896182.18,
+    -6190443.81,
+    19926188.85,
+    16579785.82
+  ],
+  "projection": "epsg:3857",
+  "layers": [
+    {
+      "className": "OpenLayers.Layer.Vector",
+      "dimensions": {},
+      "legends": [
+        ""
+      ],
+      "maxResolution": null,
+      "metadata": {},
+      "minResolution": 0,
+      "name": "testuser1:hranice",
+      "opacity": 1,
+      "path": "group2",
+      "protocol": {
+        "format": "hs.format.WFS",
+        "url": "http://localhost:8000/missing-geoserver-in-path/wfs"
+      },
+      "ratio": 1.5,
+      "singleTile": true,
+      "title": "Layer pointing to the same netloc but without `/geoserver/` in path",
+      "visibility": false,
+      "wmsMaxScale": 0
+    },
+    {
+      "className": "OpenLayers.Layer.Vector",
+      "dimensions": {},
+      "legends": [
+        ""
+      ],
+      "maxResolution": null,
+      "metadata": {},
+      "minResolution": 0,
+      "name": "test__user1:hranice",
+      "opacity": 1,
+      "path": "group2",
+      "protocol": {
+        "format": "hs.format.WFS",
+        "url": "http://localhost:8000/geoserver/wfs"
+      },
+      "ratio": 1.5,
+      "singleTile": true,
+      "title": "Layer pointing to the same Layman instance but workspace name not matching regex",
+      "visibility": false,
+      "wmsMaxScale": 0
+    },
+    {
+      "metadata": {},
+      "visibility": true,
+      "opacity": 1,
+      "title": "Layer pointing to the same Layman instance but layer name not matching regex",
+      "className": "HSLayers.Layer.WMS",
+      "singleTile": true,
+      "url": "http://localhost:8000/geoserver/testuser1_wms/ows",
+      "params": {
+        "LAYERS": "mis-ta",
+        "FORMAT": "image\/png"
+      }
+    },
+    {
+      "metadata": {},
+      "visibility": true,
+      "opacity": 1,
+      "title": "Layer pointing to the same Layman instance but without workspace",
+      "className": "HSLayers.Layer.WMS",
+      "singleTile": true,
+      "url": "http://localhost:8000/geoserver/ows",
+      "params": {
+        "LAYERS": "mista",
+        "FORMAT": "image\/png"
+      }
+    },
+    {
+      "metadata": {},
+      "visibility": true,
+      "opacity": 1,
+      "title": "Layer pointing to the same Layman instance but with 3 layer-name parts",
+      "className": "HSLayers.Layer.WMS",
+      "singleTile": true,
+      "url": "http://localhost:8000/geoserver/ows",
+      "params": {
+        "LAYERS": "part1:part2:mista",
+        "FORMAT": "image\/png"
+      }
+    }
+  ]
+}

sample/layman.map/internal_url.json

@@ -75,12 +75,12 @@
       "maxResolution": null,
       "metadata": {},
       "minResolution": 0,
-      "name": "Hranice",
+      "name": "hranice",
       "opacity": 1,
       "path": "group2",
       "protocol": {
         "format": "hs.format.WFS",
-        "url": "http://localhost:8000/geoserver/testuser1_wms/wfs"
+        "url": "http://localhost:8000/geoserver/testuser1/wfs"
       },
       "ratio": 1.5,
       "singleTile": true,

sample/layman.map/internal_url_wfs_workspace_in_layername.json

@@ -0,0 +1,43 @@
+{
+  "describedBy": "https://raw.githubusercontent.com/hslayers/map-compositions/2.0.0/schema.json",
+  "schema_version": "2.0.0",
+  "abstract": "Hranice",
+  "title": "Hranice",
+  "extent": [
+    -35.0,
+    -48.5,
+    179,
+    81.5
+  ],
+  "nativeExtent": [
+    -3896182.18,
+    -6190443.81,
+    19926188.85,
+    16579785.82
+  ],
+  "projection": "epsg:3857",
+  "layers": [
+    {
+      "className": "OpenLayers.Layer.Vector",
+      "dimensions": {},
+      "legends": [
+        ""
+      ],
+      "maxResolution": null,
+      "metadata": {},
+      "minResolution": 0,
+      "name": "testuser1:hranice",
+      "opacity": 1,
+      "path": "group2",
+      "protocol": {
+        "format": "hs.format.WFS",
+        "url": "http://localhost:8000/geoserver/wfs"
+      },
+      "ratio": 1.5,
+      "singleTile": true,
+      "title": "Hranice",
+      "visibility": false,
+      "wmsMaxScale": 0
+    }
+  ]
+}

src/layman/map/util.py

@@ -357,26 +357,55 @@ def find_maps_by_grep(regexp):
     return maps
 
 
+def _get_layer_url_from_wms_json(map_layer):
+    return map_layer.get('url')
+
+
+def _get_layer_names_from_wms_json(map_layer):
+    return [
+        n for n in map_layer.get('params', {}).get('LAYERS', '').split(',')
+        if len(n) > 0
+    ]
+
+
+def _get_layer_url_from_vector_json(map_layer):
+    protocol = map_layer.get('protocol', {})
+    return protocol.get('url') if protocol.get('format', '').split('.')[-1] == 'WFS' else None
+
+
+def _get_layer_names_from_vector_json(map_layer):
+    return [
+        n for n in map_layer.get('name', '').split(',')
+        if len(n) > 0
+    ]
+
+
 def get_layers_from_json(map_json):
     map_json = input_file.unquote_urls(map_json)
     gs_server_url = get_gs_proxy_server_url()
-    gs_wms_url_pattern = r'^' + re.escape(gs_server_url) + layman_util.CLIENT_PROXY_ONLY_PATTERN + \
-                         re.escape(layman_settings.LAYMAN_GS_PATH) + \
-                         r'(?:(?P<workspace>' + layman_util.WORKSPACE_NAME_ONLY_PATTERN + r')/)?' \
-                         + r'(?:ows|wms|wfs).*$'
+    gs_url_pattern = r'^' + re.escape(gs_server_url) + layman_util.CLIENT_PROXY_ONLY_PATTERN + \
+                     re.escape(layman_settings.LAYMAN_GS_PATH) + \
+                     r'(?:(?P<workspace>' + layman_util.WORKSPACE_NAME_ONLY_PATTERN + r')/)?' \
+                     + r'(?:ows|wms|wfs).*$'
     found_layers = set()
     for layer_idx, map_layer in enumerate(map_json['layers']):
-        layer_url = map_layer.get('url', None)
+        class_name = map_layer.get('className', '').split('.')[-1]
+        layer_url_getter = {
+            'WMS': _get_layer_url_from_wms_json,
+            'Vector': _get_layer_url_from_vector_json,
+        }.get(class_name)
+        layer_url = layer_url_getter(map_layer)
         if not layer_url:
             continue
-        match = re.match(gs_wms_url_pattern, layer_url)
+        match = re.match(gs_url_pattern, layer_url)
         if not match:
             continue
         url_geoserver_workspace = match.group('workspace')
-        layer_names = [
-            n for n in map_layer.get('params', {}).get('LAYERS', '').split(',')
-            if len(n) > 0
-        ]
+        layer_names_getter = {
+            'WMS': _get_layer_names_from_wms_json,
+            'Vector': _get_layer_names_from_vector_json,
+        }.get(class_name)
+        layer_names = layer_names_getter(map_layer)
         for full_layername in layer_names:
             if not url_geoserver_workspace:
                 layername_parts = full_layername.split(':')