Expand the data model

[timrobertson100]

As GBIF explores capabilities with a new data model we wish to produce exemplar datasets that demonstrate the output using the IPT.

This is an evolutionary change from the current model and removes the constraints of the star-schema inherent in the DwC-A. It is desirable to minimise the impact of these changes to the user community, and so we are looking to adapt the IPT in a manner that will remain familiar.

It is envisaged that:

1. The table schemas for the new model will be available, in a similar manner to those on rs.gbig.org today. The format of these is yet to be decided, but could be defined using XML (as per today) or by using Frictionless data or Avro schema formats.

2. A user of the IPT will be able to upload spreadsheets, or connect to a database as they do currently

3. During data mapping, the user can select the target table to map data to in a similar manner to the current core and extension. The difference however, is that the table arrangement may not be in a star-format

4. On data publishing the IPT will prepare a Zip file (initially) containing the converted CSV files with header rows, an EML file as it does today, and a meta file that describes the relationships between the tabular data. In the first implementation, we should prepare this meta file in the Frictionless data package format. This may be revised to e.g. the W3C CSV on the web format or even Avro formats as explorations develop.

5. During the archive generation, the IPT will continue to perform key validation checks, including the existing and uniqueness of the necessary IDs, and checking the referential integrity of the relationships.

6. An installation of this branch (v3) of the IPT be available for those working on the data model to test.

[peterdesmet]

Nice! Happy to see this on the roadmap and happy to see so many improvements in the IPT by @mike-podolskiy90.

[mdoering]

It would be great if the IPT would then also allow to generate ColDP which is for most parts very close to frictionless data. There is even a frictionless tabular-data-package generated by the API that contains all possible fields for all possible entities.

Contrary to DwC-A ColDP does not use a semantic mapping of the data files but instead uses column headers and filename conventions to identify the terms/entities.

[CecSve]

5. During the archive generation, the IPT will continue to perform key validation checks, including the existing and uniqueness of the necessary IDs, and checking the referential integrity of the relationships.

This is relevant for a question we received through the portal feedback, so it is great to see it will be incorporated in the new data model.