MIssing metadata fields when exporting data package

[danfowler]

Without no schema on each resource, these are technically not Tabular Data Packages, even if all the resources are tabular.

This dataset, https://datahub.io/dataset/period-table-4716738971 (imported using "Import Data Package"), for example, exports the following datapackage.json:

{
  "description": "Example Data Package featuring the periodic table.", 
  "license": {
    "type": "ODC-PDDL-1.0", 
    "title": "ODC-PDDL-1.0"
  }, 
  "title": "Periodic Table", 
  "keywords": [
    "atom", 
    "chemistry", 
    "element"
  ], 
  "resources": [
    {
      "url": "https://datahub.io/dataset/aa320a4f-9ae4-4bc7-95c9-f3703bd5ceec/resource/85be2f85-17a0-4447-87df-e54b47477557/download/data.csv", 
      "title": "data", 
      "name": "data", 
      "format": "CSV"
    }
  ], 
  "name": "period-table-4716738971"
}

[danfowler]

Related comment:

https://datahub.io/dataset/rockhampton-regional-council-bus-stops#comment-2824345324

I downloaded the datapackage.json, but it only had highlevel metadata, and did not contain metadata of the columns in the dataset. I also expected that the resource would be downloaded as one package with the metadata.

[EarlButterworth]

My assumption was that, as it was a datapackage that it would deliver on download as a datapackage, exactly as it had been uploaded; i.e. the one ZIP file containing the datapackage.json and the CSV. Instead, all that was delivered was a partial datapackage.json which did not contain the Resource Schema.

The aim is to bring simplicity to data publishing and consumption. Therefore a consistent user experience (UX) should be provided; i.e. what goes in is what comes out.

[danfowler]

Hi @EarlButterworth I'm creating a new issue based on your comment:

#52

Thanks!

[Stephen-Gates]

Agree with above. Testing using v1.0.0

• upload https://github.com/Stephen-Gates/data-package-examples/blob/master/test-date/datapackage.json

• download https://github.com/Stephen-Gates/data-package-examples/blob/master/test-date/datapackage-from-ckan.json

Information lost includes:

package:

• sources

resources:

• profile
• encoding
• mediatype
• licenses
• schema

[amercader]

Implementation

We already discussed licenses in #62 and schemas are now fixed.

For the rest of fields, modify the ckan-datapackage-tools converter to store them as extras (except mediatype, which maps to mimetype on resources). Use these extras on the way out to generate the DP descriptor

Estimate

0.5 day