Breaking changes coming from BC Data Catalogue update

[stephanieyurchak]

Information from BC Data Catalogue on upcoming changes:

IMPORTANT: If you or your clients use APIs to access, view or edit your datasets and resources, the schema changes will affect your current setup. Please refer to "Breaking Change" below for further guidance and test your applications using our TEST API (toyger.data.gov.bc.ca/api).

What's happening?
The BC Data Catalogue is going through a version upgrade and there is a new version of CKAN (2.9) in our TEST environment. Users of the BC Data Catalogue are invited to participate in end user testing and share your findings with the catalogue team.

Summary of changes:

• Upgrade CKAN version to 2.9
• Upgrade python files from version 2 to version 3
• Schema changes (see below)
  - BREAKING CHANGE - Composite and composite-repeating presets for fields were removed
  + Fields formerly of these types now have a field called repeating_subfields defining the subfields
  + This is functionally the same for fields that were formerly composite_repeating.
  ++ Effected fields: contacts, more_info and dates
  + Fields that were formerly composite now need to be in arrays.
  ++ Effected fields: temporal_extent, preview_info and geographic_extent
  ++ Previous format: {subfield1: value, subfield2: value, ...}
  ++ New format: [{subfield1: value, subfield2: value, ...}]
  ++ Note that these must be arrays of length 1

Please refer to the attached changelog (PDF) for a detailed list of changes included in this version.
BC Data Catalogue Changelog.pdf

What should I do?
If you are a user of the BC Data Catalogue, please consider participating in end user testing to verify the features and use of product in the new version. If you have any feedback regarding testing, please let us know by August 6th using our JSM channel (see details below).

Test Environment Link: https://toyger.data.gov.bc.ca/

How can I find out more about this?
If you are interested in testing, or have any questions, please get in touch with the BC Data Catalogue team and we'd be happy to assist and guide you. Use our JSM channel to submit your inquiries and results from your testing.

JSM: [Provide Feedback - Data Systems & Services - Jira Service Management (atlassian.net)]

[boshek]

Thanks for reporting this @stephanieyurchak

Looks like there no actual data in the staging instance of the catalogue. Could you confirm this with the dev team working on this? You could even reference this issue directly and get them having a conversation about how they are envisioning we test this without data.

[stephanieyurchak]

I have submitted a ticket to their JSM channel and will reach out when I hear back.

[stephanieyurchak]

The data in the test environment has been refreshed and is available now.

[stephanieyurchak]

New testing deadline: August 15th

Additional messaging:
Thank you so much for sharing your findings on the new version of the catalogue in our UAT environment. Our team has refreshed the data, and API keys in UAT environment recently. If you use APIs in our test environment, please update your application(s) with new API keys. You can find it here at https://toyger.data.gov.bc.ca/user/profile.

IMPORTANT: If you or your clients use APIs to access, view or edit your datasets and resources, the schema changes will affect your current setup. Please refer to "Breaking Change" mentioned in the previous email for further guidance and test your applications using our TEST API (toyger.data.gov.bc.ca/api).

[ateucher]

There are several datasets that return a 403 (forbidden) when trying to download the resource in the test environment, but are downloadable from the live catalogue. This is because the URL for the download points to localhost. For example downloading the resource from here works:

https://catalogue.data.gov.bc.ca/dataset/8620ce82-4943-43c4-9932-40730a0255d6 (https://catalogue.data.gov.bc.ca/dataset/8620ce82-4943-43c4-9932-40730a0255d6/resource/29720918-fad5-4619-89c2-bc04e96febf6/download/bc-stats_2022-new-homes-data-final.xlsx)

But from here does not as the url is localhost:

https://toyger.data.gov.bc.ca/dataset/8620ce82-4943-43c4-9932-40730a0255d6 (http://localhost:5000/dataset/8620ce82-4943-43c4-9932-40730a0255d6/resource/29720918-fad5-4619-89c2-bc04e96febf6/download/bc-stats_2022-new-homes-data-final.xlsx).

I assume this will be fixed when moved to prod, but it makes it difficult to test.

[ateucher]

I've opened a ticket for this. Assuming this gets fixed the rest of the tests are passing