Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix catch error on index #327

Merged
merged 1 commit into from
Jun 26, 2024
Merged

Conversation

PavloMalko
Copy link
Contributor

@PavloMalko PavloMalko commented Mar 28, 2024

When datasets have the wrong geospatial coverage value, they aren’t indexed so users won’t know of them and can’t update/ delete them:

ckan search-index rebuild -ei
2024-03-13 02:12:13,066 INFO  [ckan.cli] Using configuration file /etc/ckan/default/production.ini
2024-03-13 02:12:13,067 INFO  [ckan.config.environment] Loading static files from public-bs3
2024-03-13 02:12:13,267 WARNI [ckan.common] Option ckan.plugins is not declared
2024-03-13 02:12:13,494 INFO  [ckan.config.environment] Loading templates from /usr/lib/ckan/default/src/ckan/ckan/templates-bs3
2024-03-13 02:12:13,714 WARNI [ckan.common] Option ckanext.xloader.max_content_length is not declared
2024-03-13 02:12:13,714 WARNI [ckan.common] Option ckanext.xloader.max_type_guessing_length is not declared
2024-03-13 02:12:13,714 WARNI [ckan.common] Option ckanext.xloader.max_excerpt_lines is not declared
2024-03-13 02:12:14,704 WARNI [ckan.common] Option lang is not declared
2024-03-13 02:12:14,782 WARNI [ckanext.reclineview.plugin] The Recline-based views are deprecated andwill be removed in future versions
2024-03-13 02:12:14,825 INFO  [ckan.config.environment] Loading templates from /usr/lib/ckan/default/src/ckan/ckan/templates-bs3
2024-03-13 02:12:14,904 DEBUG [ckanext.harvest.model] Harvest tables defined in memory
2024-03-13 02:12:14,908 DEBUG [ckanext.harvest.model] Harvest tables already exist
2024-03-13 02:12:14,990 WARNI [ckan.common] Option ckanext_jmh.theme_name is not declared
2024-03-13 02:12:15,211 WARNI [ckan.config.middleware.flask_app] Extensions are excluded from CSRF protection! We allow extensions to run without CSRF protection but it will be forced future releases. Read the documentation for more information on how to add CSRF protection to your extension.
2024-03-13 02:12:15,301 INFO  [ckan.lib.search] Rebuilding search index...
2024-03-13 02:12:15,306 INFO  [ckan.lib.search] Rebuilding the whole index...
Indexing dataset 1/32024-03-13 02:12:15,335 ERROR [ckan.lib.search] Error while indexing dataset ffaedeec-6345-4925-920f-da60ef617e48: TypeError("'float' object is not iterable")
2024-03-13 02:12:15,346 ERROR [ckan.lib.search] Traceback (most recent call last):
  File "/usr/lib/ckan/default/src/ckan/ckan/lib/search/__init__.py", line 253, in rebuild
    package_index.update_dict(
  File "/usr/lib/ckan/default/src/ckan/ckan/lib/search/index.py", line 106, in update_dict
    self.index_package(pkg_dict, defer_commit)
  File "/usr/lib/ckan/default/src/ckan/ckan/lib/search/index.py", line 290, in index_package
    pkg_dict = item.before_dataset_index(pkg_dict)
  File "/usr/lib/ckan/default/src/ckanext-spatial/ckanext/spatial/plugin/__init__.py", line 175, in before_dataset_index
    pkg_dict = search_backends[search_backend]().index_dataset(pkg_dict)
  File "/usr/lib/ckan/default/src/ckanext-spatial/ckanext/spatial/search/__init__.py", line 178, in index_dataset
    shape = self.shape_from_geometry(geometry)
  File "/usr/lib/ckan/default/src/ckanext-spatial/ckanext/spatial/search/__init__.py", line 37, in shape_from_geometry
    shape = shapely.geometry.shape(geometry)
  File "/usr/lib/ckan/default/lib/python3.8/site-packages/shapely/geometry/geo.py", line 102, in shape
    return Polygon(ob["coordinates"][0], ob["coordinates"][1:])
  File "/usr/lib/ckan/default/lib/python3.8/site-packages/shapely/geometry/polygon.py", line 230, in __new__
    shell = LinearRing(shell)
  File "/usr/lib/ckan/default/lib/python3.8/site-packages/shapely/geometry/polygon.py", line 93, in __new__
    coordinates = np.array([_coords(o) for o in coordinates])
TypeError: 'float' object is not iterable
Indexing dataset 2/32024-03-13 02:12:15,357 ERROR [ckan.lib.search] Error while indexing dataset d6ea3142-0746-4b43-8fdf-cdd2a75643f7: TypeError("'float' object is not iterable")
2024-03-13 02:12:15,363 ERROR [ckan.lib.search] Traceback (most recent call last):
  File "/usr/lib/ckan/default/src/ckan/ckan/lib/search/__init__.py", line 253, in rebuild
    package_index.update_dict(
  File "/usr/lib/ckan/default/src/ckan/ckan/lib/search/index.py", line 106, in update_dict
    self.index_package(pkg_dict, defer_commit)
  File "/usr/lib/ckan/default/src/ckan/ckan/lib/search/index.py", line 290, in index_package
    pkg_dict = item.before_dataset_index(pkg_dict)
  File "/usr/lib/ckan/default/src/ckanext-spatial/ckanext/spatial/plugin/__init__.py", line 175, in before_dataset_index
    pkg_dict = search_backends[search_backend]().index_dataset(pkg_dict)
  File "/usr/lib/ckan/default/src/ckanext-spatial/ckanext/spatial/search/__init__.py", line 178, in index_dataset
    shape = self.shape_from_geometry(geometry)
  File "/usr/lib/ckan/default/src/ckanext-spatial/ckanext/spatial/search/__init__.py", line 37, in shape_from_geometry
    shape = shapely.geometry.shape(geometry)
  File "/usr/lib/ckan/default/lib/python3.8/site-packages/shapely/geometry/geo.py", line 108, in shape
    return MultiPolygon([[c[0], c[1:]] for c in ob["coordinates"]])
  File "/usr/lib/ckan/default/lib/python3.8/site-packages/shapely/geometry/multipolygon.py", line 81, in __new__
    p = polygon.Polygon(shell, holes)
  File "/usr/lib/ckan/default/lib/python3.8/site-packages/shapely/geometry/polygon.py", line 230, in __new__
    shell = LinearRing(shell)
  File "/usr/lib/ckan/default/lib/python3.8/site-packages/shapely/geometry/polygon.py", line 93, in __new__
    coordinates = np.array([_coords(o) for o in coordinates])
TypeError: 'float' object is not iterable
Indexing dataset 3/32024-03-13 02:12:15,376 ERROR [ckan.lib.search] Error while indexing dataset de972abb-6e84-4bcb-a4d2-cf23a7e306b9: TypeError("'float' object is not iterable")
2024-03-13 02:12:15,381 ERROR [ckan.lib.search] Traceback (most recent call last):
  File "/usr/lib/ckan/default/src/ckan/ckan/lib/search/__init__.py", line 253, in rebuild
    package_index.update_dict(
  File "/usr/lib/ckan/default/src/ckan/ckan/lib/search/index.py", line 106, in update_dict
    self.index_package(pkg_dict, defer_commit)
  File "/usr/lib/ckan/default/src/ckan/ckan/lib/search/index.py", line 290, in index_package
    pkg_dict = item.before_dataset_index(pkg_dict)
  File "/usr/lib/ckan/default/src/ckanext-spatial/ckanext/spatial/plugin/__init__.py", line 175, in before_dataset_index
    pkg_dict = search_backends[search_backend]().index_dataset(pkg_dict)
  File "/usr/lib/ckan/default/src/ckanext-spatial/ckanext/spatial/search/__init__.py", line 178, in index_dataset
    shape = self.shape_from_geometry(geometry)
  File "/usr/lib/ckan/default/src/ckanext-spatial/ckanext/spatial/search/__init__.py", line 37, in shape_from_geometry
    shape = shapely.geometry.shape(geometry)
  File "/usr/lib/ckan/default/lib/python3.8/site-packages/shapely/geometry/geo.py", line 108, in shape
    return MultiPolygon([[c[0], c[1:]] for c in ob["coordinates"]])
  File "/usr/lib/ckan/default/lib/python3.8/site-packages/shapely/geometry/multipolygon.py", line 81, in __new__
    p = polygon.Polygon(shell, holes)
  File "/usr/lib/ckan/default/lib/python3.8/site-packages/shapely/geometry/polygon.py", line 230, in __new__
    shell = LinearRing(shell)
  File "/usr/lib/ckan/default/lib/python3.8/site-packages/shapely/geometry/polygon.py", line 93, in __new__
    coordinates = np.array([_coords(o) for o in coordinates])
TypeError: 'float' object is not iterable
2024-03-13 02:12:15,382 INFO  [ckan.lib.search] Finished rebuilding search index.`

@amercader amercader merged commit 237d033 into ckan:master Jun 26, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants