landscape_analysis.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# DCAT - Landscape analysis "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## What is [DCAT](https://www.w3.org/TR/vocab-dcat-3/)? \n",
    "**DCAT** (Data Catalogue Vocabulary) is a vocabulary for sharing data catalogs on the web. It provides standardized guidelines, rules and convensions on how to structure and describe metadata about datasets, with the goal of increasing interoperability and discoverability of datasets across domains. The vocabulary is machine-readable as it formulates [RDF](https://www.w3.org/RDF/) (Resource Description Framework) classes and properties. \n",
    "\n",
    "DCAT uses terms from other well-established ontologies such as [FOAF](http://xmlns.com/foaf/spec/), [SKOS](https://www.w3.org/2004/02/skos/) and [Dublin Core](https://www.dublincore.org/), and introduces new classes and properties in the [`dcat` namespace](http://www.w3.org/ns/dcat#). \n",
    "\n",
    "\n",
    "Here's a timeline of the development of the vocabulary: \n",
    "- 2010: First development of DCAT by [Vassilios Peristeras](https://www.ihu.gr/ucips/cv/vassilios-peristeras) at the former [Digital Enterprise Research Institute](https://en.wikipedia.org/wiki/Digital_Enterprise_Research_Institute) in Ireland. \n",
    "-2012: DCAT Is taken over by the [W3C](https://www.w3.org/) Government Linked Data Working Group. \n",
    "- 2014: Fully standardized version of DCAT is released. \n",
    "- 2020: Release of DCAT version 2. \n",
    "- 2022: Release of DCAT version 3. \n",
    "\n",
    "The different versions of DCAT are backwards compatible, meaning that older versions remain in conformance with the newer versions. The updates generally included new classes and properties, and some contraints were relaxed. \n",
    "                                                          \n",
    "\n",
    "\n",
    "### Structure \n",
    "DCAT is based around seven main classes. Each class has different properties and different levels of optionality. The classes describe different \n",
    "\n",
    "- [`dcat:Dataset`](https://www.w3.org/TR/vocab-dcat-3/#Class:Dataset): A collection of data, published or curated by a single agent, and available for access or download in one or more representations.\n",
    "- [`dcat:Catalog`]((https://www.w3.org/TR/vocab-dcat-3/#Class:Catalog)): A curated collection of metadata about resources. \n",
    "- [`dcat:Resource`](https://www.w3.org/TR/vocab-dcat-3/#Class:Resource): Resource published or curated by a single agent.\n",
    "- [`dcat:Distribution`](https://www.w3.org/TR/vocab-dcat-3/#Class:Distribution): An accessible form of a dataset such as a downloadable file. \n",
    "- [`dcat:Dataservice`](https://www.w3.org/TR/vocab-dcat-3/#Class:Data_Service): A collection of operations that provide access to a dataset or data processing functions, for example through an interface (API). \n",
    "- [`dcat:DatasetSeries`](https://www.w3.org/TR/vocab-dcat-3/#Class:Dataset_Series): A collection of datasets that are published separately, but share some characteristics that group them. \n",
    "- [`dcat:CatalogRecord`](https://www.w3.org/TR/vocab-dcat-3/#Class:Catalog_Record): A metadata record in the catalog, describing a registration of a single dcat:Resource. This class is optional.   \n",
    "\n",
    "\n",
    "Each class has a number of predicates with different levels of obligation. Some elements are mandatory, such as the title or description of a dataset. Others are recommended or optional.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Overview of the classes](img/dcat_class_diagram.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "vscode": {
     "languageId": "plaintext"
    }
   },
   "source": [
    "See [this file](basic-example.ttl) for an example of a data catalogue described with DCAT. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## EU Recommendations: [DCAT-AP](https://semiceu.github.io/DCAT-AP/releases/3.0.0/) \n",
    "In 2013, the European Union revised some legislation related to the re-use of public data. The revision included the adoption of the \"open by default\" principle, which formulates that data of public bodies should be open and free by default and design, as well as easily accessible via API if possible. As a response to this new standard, European public administrations set up Open Data portals, setting the first steps to a European open data ecosystem. Lack of standardization lead to a fragmented landscape of in which it was difficult to exchange metadata between the over 150 data portals, and which all had to be queried individually. \n",
    "\n",
    "\n",
    "The European Commission recognized the need to improve the fragmented open data environment in Europe. A common metadata language would make it easier to discover and reuse datasets across portals. With this in mind, the commission extended the already existing DCAT by developing additional requirements and recommendations that would adhere to the European context, resulting in **DCAT-AP** (DCAT Application Profile). It's a specification of DCAT that reuses terms but adds more specificity. It identifies more mandatory, recommended and optional elements that should be used in particular situations, as well as recommendations for controlled vocabularies that could be used. DCAT-AP is extendable, meaning that further needs for specifications can be implemented. Although it was developed in the context of Open Data portals in Europe, it can be used for any type of dataset. \n",
    "\n",
    "DCAT-AP does not require a system to implement specific technical environments. The only requirement is that a system can export and import data in RDF. The vocabulary is maintained by the [SEMIC Support Centre](https://interoperable-europe.ec.europa.eu/collection/semic-support-centre). \n",
    "\n",
    "(This section was based on the information found in [this paper](https://doi.org/10.1504/EG.2022.121856).)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![dcat-ap-overview](img/DCAT-APoverview-annotated.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Goals\n",
    "\n",
    "As listed in their [documentation](https://semiceu.github.io/DCAT-AP/releases/3.0.0/), the DCAT-AP aims to facilitate data findability and promote reusability through coherent documentation of datasets. The Application Profile enables this by considering the following aspects: \n",
    "- Understanding the data or service structure\n",
    "- Understanding how to get acccess to it\n",
    "- Legal information about its use\n",
    "- Information about data publishers and other agents\n",
    "- Information about data availability and change policies \n",
    "\n",
    "\n",
    "The main contribution of the Application Profile is to extent DCAT with more contraints or mandatory fields. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Namespaces\n",
    "See here an overview of namespaces that are used in the vocabularies and application profiles discussed in this document. \n",
    "| prefix | namespace IRI | \n",
    "| --- | --- | \n",
    "| adms  | http://www.w3.org/ns/adms# | \n",
    "| dcat |     http://www.w3.org/ns/dcat# | \n",
    "| dcatap | http://data.europa.eu/r5r/ | \n",
    "| dct |     http://purl.org/dc/terms/ |\n",
    "| foaf |     http://xmlns.com/foaf/0.1/ | \n",
    "| dpv | https://w3id.org/dpv# | \n",
    "| healthdcatap |  TBA   | \n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Mandatory properties and DCAT-AP additions\n",
    "The sections below outline mandatory properties for a selection of classes, as well as optional properties that were added specifically for the Application Profile. The information is taken from the [DCAT-AP documentation](https://semiceu.github.io/DCAT-AP/releases/3.0.0/#main-entities). In the original documentation you can find all properties and their associated constraints for all classes. Moreover, it includes an indicator to mark whether a property is:  \n",
    "- `A` reused as-is defined in DCAT, \n",
    "- `E` reused with additional notes or restrictions\n",
    "- `P` DCAT-AP specific. \n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### The `Dataset` class\n",
    "\n",
    "Here's an [overview](https://semiceu.github.io/DCAT-AP/releases/3.0.0/#Dataset) of the most important aspects that the metadata of a dataset should include to conform to the AP. It includes the mandatory properties as well as the additions made by DCAT-AP (indicated with P)\n",
    "\n",
    "| Property | URI | Definition   | Reuse | Optionality |\n",
    "| ---      | --- | ---     | ---   | --- |\n",
    "| [description](https://www.dublincore.org/specifications/dublin-core/dcmi-terms/#description) | dct:description | A free-text account of the Dataset |  E | mandatory | \n",
    "| [title](https://www.dublincore.org/specifications/dublin-core/dcmi-terms/#title) | dct:title | A name given to the Dataset | E | mandatory |\n",
    "| [applicable legislation](https://semiceu.github.io/DCAT-AP/r5r/releases/3.0.0/#applicableLegislation) | dcatap:applicableLegislation | The legislation that mandates the creation or management of the Dataset   | P | optional | \n",
    "| [documentation](http://xmlns.com/foaf/spec/) | foaf:documentation | A page or document about this Dataset  | P | optional | \n",
    "| [has version](https://www.w3.org/ns/dcat#hasVersion) | dcat:hasVersion | A related Dataset that is a version, edition, or adaptation of the described Dataset  | P | optional | \n",
    "| [provenance](https://www.dublincore.org/specifications/dublin-core/dcmi-terms/#provenance) | dct:provenance | A statement about the lineage of a Dataset  | P | optional | \n",
    "| [sample](https://www.w3.org/ns/legacy_adms#sample) | adms:sample | A sample distribution of the dataset | P | optional | \n",
    "| [source](https://www.dublincore.org/specifications/dublin-core/dcmi-terms/#source) | dct:source | A related Dataset from which the described Dataset is derived  | P | optional | \n",
    "| [version notes](https://www.w3.org/ns/legacy_adms#versionNotes) | adms:versionNotes | A description of the differences between this version and a previous version of the Dataset | P | optional | \n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### The `Catalogue` class\n",
    "This class concerns a catalogue or repositor that hosts Datasets or Data Services. A Data Station can be defined as a data service.  \n",
    "\n",
    "| Property  | Definition   | Reuse | Optionality | \n",
    "| ---      | ---        | ---     | --- | \n",
    "| description  | A free-text account of the Catalogue |  E | mandatory | \n",
    "| publisher  | responsible for making the Catalogue available | E | mandatory | \n",
    "| title  | A name given to the Catalogue | E | mandatory | \n",
    "| applicable legislation   | The legislation that mandates the creation or management of the Catalogue   | P | optional | \n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Validation "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Validation / SHACL info DCAT-AP-NL [here](https://docs.geostandaarden.nl/dcat/dcat-ap-nl30/#17C1E0BE)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Who uses DCAT-AP?  \n",
    "- [European data portal](https://data.europa.eu/en): The portal where all EU member states have to upload their open data. \n",
    "- [German data portal](https://www.govdata.de)\n",
    "- [European Commission's Joint Research Centre (JRC)](https://www.digdir.no/)\n",
    "- [CKAN](https://ckan.org/) with the [DCAT harvester](https://github.com/ckan/ckanext-dcat) \n",
    "- [Spanish open data portal](https://datos.gob.es/es/documentacion/recursos-disponibles-para-conocer-mas-sobre-dcat-ap)\n",
    "\n",
    "See more [here](https://ec.europa.eu/isa2/solutions/dcat-application-profile-data-portals-europe_en/). "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## [DCAT-AP-NL](https://geonovum.github.io/DCAT-AP-NL30/)\n",
    "DCAT-AP-NL is the Dutch adaptation of DCAT-AP. It is developed and maintained by [Geonovum](https://www.geonovum.nl/), a Dutch executive committee concerned with developing data standards mainly in the realm of spatial data. A number of collaborators from different field are involved as well, including [health-RI](https://www.health-ri.nl/over-health-ri), [Digitaal erfgoed](https://datasetregister.netwerkdigitaalerfgoed.nl/?lang=en), and [Centraal Bureau voor de Statistiek](https://www.cbs.nl/)\n",
    "\n",
    " The profile adds further specifications, mandatory aspects and recommendations that are catered to Dutch open data catalogues and governmental data portals. It is build upon DCAT-AP, so if metadata complies with DCAT-AP-NL, it automatically also complies with DCAT-AP. The specifications are written in Dutch, but the summary in this document contains their English translations. \n",
    "\n",
    "The Geonovum website [reported](https://www.geonovum.nl/over-geonovum/actueel/nederlands-profiel-voor-europese-metadata-standaard-dcat) in July 2024 that the development of the Dutch application profile has finished, but is still waiting for confirmation from the EU to make it official. This is expected by the end of 2024. \n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Additions\n",
    "Below you find an overview of all the mandatory properties of the `Dataset` class, as well as optional and recommended properties that have been specified by DCAT-AP-NL. There is a column to indicate whether and how the property changed compared to DCAT-AP, and if it's currently present in DataverseNL. \n",
    "\n",
    "| Property | URI | Definition   | Optionality DCAT-AP-NL | Optionality DCAT-AP | In DataverseNL? | \n",
    "| ---      | --- | ---     | ---         |    ---     | --- | \n",
    "| [title](https://geonovum.github.io/DCAT-AP-NL30/#dataset-title)    | dct:title | A name given to the Dataset |  mandatory   | mandatory         |  yes, mandatory | \n",
    "| [description](https://geonovum.github.io/DCAT-AP-NL30/#dataset-description) | dct:description | A free-text account of the Dataset  |  mandatory | mandatory | yes, mandatory | \n",
    "| [access rights](https://geonovum.github.io/DCAT-AP-NL30/#dataset-access-rights) | dct:accessRights | Information about who can access the resource or an indication of its security status   |  mandatory | optional | yes? |  \n",
    "| [contact point](https://geonovum.github.io/DCAT-AP-NL30/#dataset-contact-point) | dcat:contactPoint | Contact information that can be used for correspondence about the Dataset |  mandatory |  optional | yes. mandatory | \n",
    "| [creator](https://geonovum.github.io/DCAT-AP-NL30/#dataset-creator) | dct:creator | An entity responsible for making the resource | mandatory | optional | yes, mandatory | \n",
    "| [identifier](https://geonovum.github.io/DCAT-AP-NL30/#dataset-identifier) | dct:identifier | An unambiguous reference to the resource within a given context | mandatory | optional| yes | \n",
    "| [theme](https://geonovum.github.io/DCAT-AP-NL30/#dataset-theme) | dcat:theme | At least one theme as defined by the [Dataset Theme Vocabulary](https://publications.europa.eu/resource/authority/data-theme) | mandatory | recommended |  no, but there is 'topic classification', optional | \n",
    "| [publisher](https://geonovum.github.io/DCAT-AP-NL30/#dataset-publisher) | dct:publisher | An entity responsible for making the resource available.   | mandatory  | recommended  | yes (distributor?), optional | \n",
    "| [conforms to](https://geonovum.github.io/DCAT-AP-NL30/#dataservice-conforms-to) | dct:conformsTo | An established standard to which the described resource conforms |  recommended | optional | no | \n",
    "| [documentation](https://geonovum.github.io/DCAT-AP-NL30/#dataset-documentation) | foaf:page | A page or document about this Dataset | recommended | optional | no | \n",
    "| [landing page](https://geonovum.github.io/DCAT-AP-NL30/#dataset-landing-page) | dcat:landingPage | A webpage that gives access to the Dataset, as well as more information.  | recommended | optional | yes, optional | \n",
    "| [status](https://geonovum.github.io/DCAT-AP-NL30/#dataset-status) | adms:status | The status of the dataset in the context of its life cycle | optional | NA: added in DCAT-AP-NL |  no  |\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### [DCAT-DONL](https://dcat-ap-donl.readthedocs.io/en/latest/summary.html)\n",
    "The Dutch government developed a standard that is based on DCAT-AP version 2.1 (see image below). You can find an overview of the mandatory, recommended and optional properties for the Dataset class [here](https://dcat-ap-donl.readthedocs.io/en/latest/classes-dataset.html). DCAT-DONL is used in the [data portal of the Dutch government](https://data.overheid.nl/), and was developed to reduce data duplication and further standardization of properties. \n",
    "\n",
    "- Contradictory information: the DCAT-DONL site [claims](https://dcat-ap-donl.readthedocs.io/en/latest/summary.html) that it is based on DCAT-AP-NL, while the image on the [DCAT-AP-NL website](https://docs.geostandaarden.nl/dcat/dcat-ap-nl30/#01CB11A8) says otherwise (see image below). \n",
    "\n",
    "- Possibly outdated information: on the [website of the Dutch data portal](https://data.overheid.nl/en/ondersteuning/open-data/dcat) (see final paragraph), it is stated that DCAT-DONL will in the future be updated to conform to DCAT 2.0 (which released in February 2020), but the image below indicates this already happened.  \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![profilesNL](img/profielenNL.png)\n",
    "[image source](https://docs.geostandaarden.nl/dcat/dcat-ap-nl30/#01CB11A8)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Who is using DCAT-AP-NL? \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## [HealthDCAT-AP](https://healthdcat-ap.github.io/)\n",
    "The HealthDCAT-AP extension was proposed by the [European Health Data Space](https://health.ec.europa.eu/ehealth-digital-health-and-care/european-health-data-space_en) (EHDS) following a [landscape analysis](https://ehds2pilot.eu/wp-content/uploads/2024/04/HealthData@EU-Pilot_MS6.1_FIN.pdf) that established that current metadata catalogues lack vocabulary specific to the health domain. The analysis was performed in response to studies arguing that findability is one of the main barriers to the re-use of health data. The development of a metadata scheme that includes a vocabulary specific to the health domain facilitates secondary use of health data, for instance in research and innovation. \n",
    "\n",
    "The development of the HealthDCAT is still ongoing, and the information that is included in this notebook is based on the latest [unofficial draft](https://healthdcat-ap.github.io/), released in December 2023. The official release was [expected](https://ehds2pilot.eu/upcoming_results/extension-of-dcat-ap-healthdcat-ap/) in early 2024, but has not yet happened. Reports on the status of the scheme have been absent since the start of 2024. The development of HealthDCAT-AP is part of the [HealthData@EU Pilot](https://ehds2pilot.eu/) project. \n",
    "\n",
    "As the HealthDCAT-AP is an extension of the DCAT-AP, it adds elements to the schema while still adhering to the main principles. The mandatory properties retain their status, while the other optionalities may change. \n",
    "\n",
    "There are 3 schema variations:\n",
    "* for Sensitive health Data\n",
    "* for Protected health Data\n",
    "* for Open health Data\n",
    "\n",
    "\n",
    "\n",
    "![HealthDCAT-AP UML Class Diagram (extending DCAT-AP 3.0) for Open health Data](https://healthdcat-ap.github.io/OPEN%20DATA%20HealthDCAT-AP%203.0.0.drawio.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### HealthDCAT additions\n",
    "Here you find an outline of the additions that are made by the HealthDCAT to the Dataset class. Properties with the DCAT Reuse value `E` are reused from DCAT but have additional notes or restrictions in HealthDCAT. In the majority of the cases, the addition was a change of a property from optional or recommended to mandatory. Properties marked with `P` are introduced in the HealthDCAT. \n",
    "\n",
    "\n",
    "| Property                        | URI        | Definition                                                                 | Optionality | DCAT Reuse |\n",
    "|----------------------------------|------------|---------------------------------------------------------------------------|-------------|------------|\n",
    "| [access rights](https://www.dublincore.org/specifications/dublin-core/dcmi-terms/#accessRights)               | dct:       | Dataset is publicly accessible, has access restrictions or is not public  | mandatory   | E          |\n",
    "| [applicable legislation](https://semiceu.github.io/DCAT-AP/r5r/releases/3.0.0/#applicableLegislation)           | dcatap:    | The legislation that mandates the creation or management of the Dataset.   | mandatory   | E          |\n",
    "| [dataset distribution](https://www.w3.org/ns/dcat#distribution)           | dcat:      | An available Distribution for the Dataset                                  | mandatory   | E          |\n",
    "| [geographical coverage](https://www.dublincore.org/specifications/dublin-core/dcmi-terms/#spatial)           | dct:       | A geographic region that is covered by the Dataset                         | mandatory   | E          |\n",
    "| [health data access body](https://healthdcat-ap.github.io/#Dataset)          | healthdcatap: | Health Data Access Body supporting access to data in the Member State     | mandatory   | P          |\n",
    "| [health theme]()                    | healtdcatap: | A category of the Dataset or tag describing the Dataset                    | mandatory   | P          |\n",
    "| [identifier](https://www.dublincore.org/specifications/dublin-core/dcmi-terms/#identifier)                       | dct:       | The main identifier for the Dataset, e.g. the URI or other unique identifier in the context of the Catalogue | mandatory   | P          |\n",
    "| [minimum typical age]()              | healthdcatap: | Minimum typical age of the population within the dataset                   | mandatory   | P          |\n",
    "| provenance                       |            | A statement about the lineage of a Dataset                                 | mandatory   | E          |\n",
    "| publisher                        |            | An entity (organisation) responsible for making the Dataset available     | mandatory   | E          |\n",
    "| publisher note                   |            | A description of the publisher activities                                  | mandatory   | P          |\n",
    "| publisher type                   |            | A type of organisation that makes the Dataset available                   | mandatory   | P          |\n",
    "| purpose                          |            | A free text statement of the purpose of the processing of data or personal data | mandatory   | P          |\n",
    "| theme                            |            | A category of the Dataset                                                  | mandatory   | E          |\n",
    "| type                             |            | A type of the Dataset                                                      | mandatory   | E          |\n",
    "| alternative                      |            | Alternative title of the dataset such as an acronym                       | optional    | P          |\n",
    "| analytics                        |            | An analytics distribution of the dataset                                  | optional    | P          |\n",
    "| code values                      |            | Health classifications and their codes associated with the dataset        | optional    | P          |\n",
    "| coding system                    |            | Coding systems in use (ex: ICD-10-CM, DGRs, SNOMED=CT, ...)               | optional    | P          |\n",
    "| conforms to                      |            | An implementing rule or other specification                               | optional    | E          |\n",
    "| health category                  |            | The health category to which this dataset belongs as described in the Commission Regulation on the European Health Data Space laying down a list of categories of electronic data for secondary use, Art.33 | optional    | P          |\n",
    "| legal basis                      |            | The legal basis used to justify processing of personal data                | optional    | P          |\n",
    "| maximum typical age              |            | Maximum typical age of the population within the dataset                  | optional    | P          |\n",
    "| number of records                |            | Size of the dataset in terms of the number of records                     | optional    | P          |\n",
    "| number of records for unique individuals |        | Number of records for unique individuals                               | optional    | P          |\n",
    "| personal data                    |            | Key elements that represent an individual in the dataset                  | optional    | P          |\n",
    "| population coverage              |            | A definition of the population within the dataset                         | optional    | P          |\n",
    "| quality annotation               |            | Dataset, including rating, quality certificate, feedback that can be associated to the dataset | optional    | P          |\n",
    "| retention period                 |            | A temporal period which the dataset is available for secondary use        | optional    | P          |\n",
    "| sample                           |            | A sample distribution of the dataset                                      | optional    | E          |\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## [Health-RI](https://github.com/Health-RI/health-ri-metadata/tree/master)\n",
    "\n",
    "A Dutch effort to create a data catalogue for health data. It is based on DCAT-AP and HealthDCAT-AP and is developed by [Health-RI](https://www.health-ri.nl/services/metadata-onboarding-NGDC), who were also involved in the development of DCAT-AP-NL. \n",
    "\n",
    "![HRI Core Schema](https://raw.githubusercontent.com/Health-RI/health-ri-metadata/refs/heads/master/Images/1.0_plateau1/HRICoreSchemaReleasePlateau1.jpg)\n",
    "\n",
    "- *How well do Health-RI and HealthDCAT-AP match?* \\\n",
    "Below you see an overview of the mandatory properties of Health-RI, as well as a comparison to their optionality status in the Health-DCAT-AP. \n",
    "- *Do the mandatory Health-RI fields match the Life Science Data Station fields?* \\\n",
    "Not all of the mandatory properties are currently available in the LS Data Station metadata form. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### [Additions & Comparison](https://github.com/Health-RI/health-ri-metadata/tree/master?tab=readme-ov-file#dataset)\n",
    "\n",
    "| property  |URI | description | Health-RI   | Health-DCAT-AP | present in Data Station LS?        |\n",
    "|-------------------------|-----------------------------------------------------------------------------|-------------|------ | -----------|------------------------------------|\n",
    "| access rights           | dcat:contactPoint | Dataset is publicly accessible, has access restrictions or is not public   | mandatory   | mandatory       | yes |                                   \n",
    "| applicable legislation  | dcatap:applicableLegislation | The legislation that mandates the creation or management of the Dataset.    | mandatory   | mandatory       | no                                |\n",
    "| [contact point](https://www.w3.org/TR/vocab-dcat-3/#Property:resource_contact_point)  | dcat:contactPoint           | Relevant contact information for the catalog resource                       | mandatory   | mandatory       | yes, mandatory                     |\n",
    "| [creator](https://www.dublincore.org/specifications/dublin-core/dcmi-terms/#creator)       | dct:creator       | The entity responsible for producing the resource                           | mandatory   | optional        | yes? (Author?), mandatory          |\n",
    "| [description](https://www.dublincore.org/specifications/dublin-core/dcmi-terms/#description)        | dct:description     | A free-text account of the Dataset                                          | mandatory   | mandatory       | yes                                |\n",
    "| geographical coverage | dct:spatial   | A geographic region that is covered by the Dataset                          | mandatory   | mandatory       |      yes, optional                              |\n",
    "| [identifier](https://www.dublincore.org/specifications/dublin-core/dcmi-terms/#identifier)    | dct:identifier          | A unique identifier of the resource being described or catalogued           | mandatory   | mandatory       | yes                                 |\n",
    "| legal basis       | dpv:hasLegalBasis      | The legal basis used to justify processing of personal data                 | mandatory   | optional        |                          no          |\n",
    "| [modified](https://www.dublincore.org/specifications/dublin-core/dcmi-terms/#modified)        | dct:modified        | Most recent date on which the catalog entry was changed, updated or modified| mandatory   | optional        | no                                 |\n",
    "| number of records      | healthdcatap:numberOfRecords | Size of the dataset in terms of the number of records                       | mandatory   | optional        |                          no           |\n",
    "| [publisher](https://www.dublincore.org/specifications/dublin-core/dcmi-terms/#publisher)          | dct:publisher     | The entity responsible for making the resource available                    | mandatory   | mandatory       | yes? (Producer?), optional         |\n",
    "| [theme](https://www.w3.org/TR/vocab-dcat-3/#Property:resource_theme)      | dcat:theme             | A main category of the resource. A resource can have multiple themes        | mandatory   | mandatory       | yes? (Subject?), mandatory         |\n",
    "| [title](https://www.dublincore.org/specifications/dublin-core/dcmi-terms/#title)          | dct:title         | A name given to the Dataset                                                 | mandatory   | mandatory       | yes                                |\n",
    "| [issued](https://www.dublincore.org/specifications/dublin-core/dcmi-terms/terms/issued/)       | dct:issued           | Date of formal issuance (e.g., publication) of the resource                 | mandatory   | optional        | yes, optional                      |\n",
    "\n",
    "\n",
    "For a full overview of the recommended and mandatory properties of all classes, see [this image](img/HRI_metadata_p2.png) [(source)](https://github.com/Health-RI/health-ri-metadata?tab=readme-ov-file#overview-and-diagram). "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Useful links \n",
    "- [Mapping pipeline](https://health-ri.atlassian.net/wiki/spaces/FSD/pages/290291734/Mapping+pipeline): describes the process of metadata mapping \n",
    "- [Health-RI RDF Validator](https://www.itb.ec.europa.eu/shacl/healthri/upload): SHACL shape-based validator to see if your metadata conforms to Health-RI. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "vscode": {
     "languageId": "plaintext"
    }
   },
   "source": [
    "### [Fair Data Point](https://www.fairdatapoint.org/)\n",
    "A FAIR Data Point (FDP) is a metadata service that provides access to metadata in a way that adheres to the FAIR principles. FDP allows owners of digital objects to expose their metadata in a FAIR way, and it allows consumers of digital objects to discover information about them. By default, all uploaded resources ase publicly accessible by anyone. The DCAT model is used as the basis for the metadata in FDP. \n",
    "\n",
    "A FAIR Data Point has three compontents: \n",
    "- A definition of the [Fair Data Point API specification](https://specs.fairdatapoint.org/fdp-specs-v1.2.html)\n",
    "- A service implementing the API specification\n",
    "- A client of the API (a web front end) that can be used to add, edit and query the information in the metadata \n",
    "\n",
    "#### Deployment\n",
    "The deployment of a FDP has a handful of components. See an overview of each component below. The information is taken from the FDP [documentation](https://fairdatapoint.readthedocs.io/en/latest/deployment/local-deployment.html).  \n",
    "\n",
    "FDP deployment components:\n",
    "- Triple Store: the place where the semantic data is stored\n",
    "- MongoDB: the place where other information (e.g. user accounts, roles) is saved\n",
    "- FAIRDataPoint: the core component that handles all logic and operations with the semantic data. It is distributed in Docker image `fairdatapoint`. \n",
    "- FAIRDataPoint-client: provides the user interface for humans. It is distributed in Docker image `fairdatapoint-client`. \n",
    "- Reverse Proxy: handles HTTPS certificates to keep the connection to the FDP secured. \n",
    " \n",
    "![overview of fdp components](img/FDP_components.png)\n",
    "\n",
    "[image source](https://fairdatapoint.readthedocs.io/en/latest/about/components.html#components)\n",
    "\n",
    "Useful links: \n",
    "- [FAIRDataPoint Github](https://github.com/FAIRDataTeam/FAIRDataPoint)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Questions \n",
    "\n",
    "- (see [minutes](https://ehds2pilot.eu/wp-content/uploads/2024/04/HealthData@EU-Pilot_MS6.2_Technical-working-group-on-the-transition-from-existing-metadata-templates-to-HealthDCAT_FIN-1.pdf) )"
   ]
  }
 ],
 "metadata": {
  "language_info": {
   "name": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}