-
Notifications
You must be signed in to change notification settings - Fork 12
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Error accessing Spatial data #335
Comments
@steffilazerte I am getting the exact same error. I have not been able to query geodata since yesterday morning. |
Interesting that it's so recent! I was trying this yesterday afternoon for the first time since March, so thought it might have been a longer term issue. Perhaps it's a BC data portal issue? |
Thanks for reporting. This is definitely brand new, and smells like a catalogue/wfs issue. I'll look into this today |
I made a change yesterday to start caching the 'base' (all layers) WMS and WFS documents, when a specific layer is not requested. e.g. https://openmaps.gov.bc.ca/geo/pub/wfs?request=GetCapabilities but not https://openmaps.gov.bc.ca/geo/pub/WHSE_FOREST_VEGETATION.VEG_COMP_LYR_R1_POLY/wfs?request=GetCapabilities. If a cached document is hit, you'll get a response header The cached documents should only be served to requests that wanted the base document. One URL I added to redirect to the cached GetCap doc which might be causing issues is https://openmaps.gov.bc.ca/geo/pub/wfs ? |
Currently the server is not accepting
Returns the entire
|
Thanks @rustyjux - that could be related? As I just posted (at the same time as you I think!), all |
Yep that's got to be it. I will try and exclude all POST requests |
I think that could work, I can't think off the top of my head where we're sending queries without a layer name that looks like that... OR excluding all POSTs might be more robust. Or all POST requests that contain a query string at all? Happy to test any/all of the potential solutions on delivery. Just let me know when to try |
I tried rolling back the caching entirely on delivery. Using @ateucher's request:
I get this response:
Presumably that isn't what you were looking for (though it isn't the full GetCap documnet) |
Haha, not quite! This works though (content sent via POST as data rather than as a form):
|
Let me test in the package |
@ateucher yep that one is better. It would be good to know what the package is doing. Thanks for bringing this to my attention |
Thanks @rustyjux all passing now! Really appreciate the quick attention :) The http client we are using in the package sends the query as data via In general, I have tried to document how are using DSS web services here: https://github.com/bcgov/bcdata/blob/main/vignettes/service_documentation.Rmd |
No problem, sorry for the trouble. |
I did a bit more reading on *By "that style" I mean |
@rustyjux Anytime you want or need testing just open an issue and we will test. |
thanks @stephhazlitt @ateucher seems unlikely the package is using Hoping this provides a good proxy for the app:
|
That for sure gets the basic case.. I can try to create a curl command than also involves sending data for a WFS query... might be a week or two before I get to it though |
Ah it was an easy fix and had to do specifically with rewrites on requests to /geo/pub/wfs with no query and the need for an exception for POST there. Which is what I suspected and noted in my first comment... but I was on a call so didn't look closely at that piece. So much for multi-tasking. If you can confirm using the package that all is well on delivery.openmaps.gov.bc.ca, I will re-implement caching in production with the added exception. |
All is well from my perspective, thanks! |
All tests pass using |
And we're live in production. Again, my apologies. On the bright side, 5xx errors (mostly due to timeouts) on requests for the full GetCapabilities doc have dropped from ~15% of requests to 0.25% |
No worries, thanks @rustyjux! I think it's smart to cache that document, and that's great to see that many fewer errors. We cache it locally within a session already, because we query it a lot. I guess not a lot of users are using |
Looks like the service is down as Simon's bcdata fails too (ex. Hoping it is just a temporary glitch but posting here now anyway. Interestingly - about an hour ago bcdata::bcdc_get_data("WHSE_IMAGERY_AND_BASE_MAPS.GSR_AIRPORTS_SVW")
#> Error: There was an issue sending this WFS request
#> ▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄
#> Request:
#> URL: https://openmaps.gov.bc.ca/geo/pub/wfs
#> POST fields:
#> SERVICE=WFS&VERSION=2.0.0&REQUEST=DescribeFeatureType&typeNames=WHSE_IMAGERY_AND_BASE_MAPS.GSR_AIRPORTS_SVW
#> Content-Type: application/x-www-form-urlencoded
#> Accept-Encoding: gzip, deflate
#> Accept: application/json, text/xml, application/xml, */*
#> User-Agent: https://github.com/bcgov/bcdata
#> Response:
#> status: HTTP/1.1 400
#> date: Thu, 04 Jul 2024 16:15:39 GMT
#> server: Apache
#> x-frame-options: allow-from (null)
#> x-control-flow-delay-ms: 0
#> content-type: application/xml
#> set-cookie: GS_FLOW_CONTROL=GS_CFLOW_-43486bae:1907e5f61f3:-7ff5
#> access-control-allow-origin: (null)
#> access-control-allow-credentials: true
#> access-control-allow-methods: POST, GET, OPTIONS, HEAD
#> access-control-allow-headers: X-Requested-With, Referer, Origin, Content-Type, SOAPAction, Authorization, Accept
#> access-control-max-age: 1000
#> connection: close
#> transfer-encoding: chunked Created on 2024-07-04 with reprex v2.1.0 |
Services were down this morning. Should be back up. In future you can consult the Data Systems and Services Uptime page (look in MAP group) or more specifically the WMS and WFS checks. Oddly the WFS checks didn't show an outage but WMS did: |
Thank you for the response. hmm. new error with uptime showing that we are good to go. Tried system restart and fresh cran download. Looks like Simon's bcdata is up and running again. Seems odd but I am good at breaking my machines so might just be me. Any chance you could confirm that you are all good? Looks like bcdata::bcdc_get_record("78ec5279-4534-49a1-97e8-9d315936f08b")
#> B.C. Data Catalogue Record: School Districts of BC
#> Name: school-districts-of-bc (ID: 78ec5279-4534-49a1-97e8-9d315936f08b)
#> Permalink:
#> https://catalogue.data.gov.bc.ca/dataset/78ec5279-4534-49a1-97e8-9d315936f08b
#> Licence: Open Government Licence - British Columbia
#> Description: The School Districts dataset contains the spatial
#> representation (polygon) of the current extent of the administrative
#> areas defined under section 176(1) of the School Act for the purposes
#> of preservation and promotion of the fundamental principle of local
#> autonomy and control of public education at the public and
#> governmental levels through locally elected school boards.
#> Available Resources (1):
#> 1. WMS getCapabilities request (wms)
#> Access the full 'Resources' data frame using:
#> bcdc_tidy_resources('78ec5279-4534-49a1-97e8-9d315936f08b')
#> Query and filter this data using:
#> bcdc_query_geodata('78ec5279-4534-49a1-97e8-9d315936f08b') bcdata::bcdc_query_geodata("78ec5279-4534-49a1-97e8-9d315936f08b") |>
dplyr::filter(SCHOOL_DISTRICT_NAME %in% c("Greater Victoria", "Prince George","Kamloops/Thompson")) |>
dplyr::select(SCHOOL_DISTRICT_NAME) |>
dplyr::collect()
#> Error in `path_to_connection()`:
#> ! '' does not exist in current working directory
#> ('/private/var/folders/mg/h910y2c54fsc99qj74dyrjph0000gn/T/RtmpdnWOXx/reprex-6b74d586486-tweed-barb'). bcdata::bcdc_get_data("78ec5279-4534-49a1-97e8-9d315936f08b")
#> Error in `path_to_connection()`:
#> ! '' does not exist in current working directory
#> ('/private/var/folders/mg/h910y2c54fsc99qj74dyrjph0000gn/T/RtmpdnWOXx/reprex-6b74d586486-tweed-barb'). Created on 2024-07-04 with reprex v2.1.0 |
@NewGraphEnvironment I'm not on the Mapping Services team so I suggest you connect with them directly. I believe Jira is the preferred route - https://dpdd.atlassian.net/servicedesk/customer/portal/1/group/1/create/30 |
I can reproduce your error @NewGraphEnvironment - I'll look into it |
Tracking the new issue in #339 |
I'm getting an error and several warnings when trying to access spatial data resources (I expect to get a spatial data frame).
This occurs when accessing spatial (but not-non spatial) data, via
bcdc_get_data()
andbcdc_query_geodata()
.Examples which create this error
Example which works fine:
bcdata::bcdc_get_data('d7e6c8c7-052f-4f06-b178-74c02c243ea4')
Reprex
Created on 2023-11-07 with reprex v2.0.2
The text was updated successfully, but these errors were encountered: