Promote "Metadata in Table Schema" recipe to the specs. #961

pierrecamilleri · 2024-07-05T08:58:06Z

Context and related issue :
fix #899

Migrated pull request from datapackage-v2-draft repo

Still to do on this PR:

Move table schema to /standards/table-schema.md (cf. main)
Move changelog to /overview/changelog.md (cf. main)
Update wording so it aligns with Data Package (my pending PR)
Update some links
Fix invalid json example (it was an unintended change)
Clarify if examples are borrowing title and path from Data Resource (I think they are, included in my PR) or if this is something completely separate.

resources

This reverts commit cc10899.

This reverts commit 8cfb654.

roll · 2024-07-08T07:34:06Z

Thanks a lot for rebasing the PR @pierrecamilleri !

pierrecamilleri · 2024-07-12T13:35:52Z

@peterdesmet @roll

Almost there.

I added "partial" to qualify "Data Resource" in the suggestion of Peter concerning the documentation of top-level "examples" ; because an object with title and path is not a valid Data Resource (name is missing). Or do we want to impose a name property ? What do you think of this wording ?

peterdesmet

@pierrecamilleri thanks for the updates! Regarding:

Or do we want to impose a name property ? What do you think of this wording ?

I think we should simply stick to the Data Resource spec and require a name (over a title). I have made suggestions to the files to reflect this change.

profiles/source/dictionary/common.yaml

profiles/source/dictionary/schema.yaml

peterdesmet · 2024-07-15T15:21:32Z

profiles/target/2.0/datapackage.json

@@ -57,7 +57,7 @@
    "homepage": {
      "propertyOrder": 60,
      "title": "Home Page",
-      "description": "The home on the web that is related to this data package.",
+      "description": "The home on the web that is related to this resource.",


Two notes:

These files should be rebuild based on the changes above.

Is it a good idea to rebuild these files? I think we should have a new directory target/2.1-draft

I am not familiar with the build process, in particular what is the difference between "build/profiles" and "profiles/target" ?

When I run the "build.js", with "VERSION=2.1-draft", it does not create (or for the matter, populate) a target/2.1-draft directory 🤔

@peterdesmet
@pierrecamilleri
I need to set up the branching process for new versions so it's OK if this PR just includes actual changes to the standard and I'll handle the rest

Co-authored-by: Peter Desmet <peter.desmet.work@gmail.com>

pierrecamilleri · 2024-09-05T12:41:24Z

Little ping @peterdesmet, if you can find the time for a review it would be great !

peterdesmet · 2024-09-16T06:33:17Z

Thanks for the heads up! Will have a look this week.

roll · 2024-09-17T14:08:19Z

Amazing @pierrecamilleri!

I'm going to be on it next week as well. First of all, I would suggest us to test a profile patch on #975, and then do this one

name, title, description, homepage, version, created, keywords, contributors, examples

peterdesmet

Nice work @pierrecamilleri I did the following:

List the properties in the same order as on Data Package: name, title, description, homepage, version, created, keywords, contributors, examples
Made some review comments regarding phrasing. I think we can remove the cf. Data Package everywhere, but indicate where it inherits from Data Package where applicable.
Minor update to simplify the CHANGELOG

I've started to use these metadata properties, so I'm looking forward to see this adopted!

peterdesmet · 2024-09-19T15:33:13Z

content/docs/standard/table-schema.md

@@ -266,6 +266,44 @@ If the value of the `foreignKey.reference.resource` property is an empty string
 Data consumer MUST support the `foreignKey.fields` and `foreignKey.reference.fields` properties in a form of a single string e.g. `"fields": "a"` which was a part of the `v1.0` of the specification.
 :::

+#### `name`
+
+A simple name or identifier for the schema (cf. [Data Package](https://datapackage.org/standard/data-package/#name)).


Suggested change

A simple name or identifier for the schema (cf. [Data Package](https://datapackage.org/standard/data-package/#name)).

A simple name or identifier for the schema.

peterdesmet · 2024-09-19T15:34:03Z

content/docs/standard/table-schema.md

+
+#### `description`
+
+A description of the schema (cf. [Data Package](https://datapackage.org/standard/data-package/#description)).


Suggested change

A description of the schema (cf. [Data Package](https://datapackage.org/standard/data-package/#description)).

A description of the schema.

peterdesmet · 2024-09-19T15:34:36Z

content/docs/standard/table-schema.md

+
+#### `created`
+
+The datetime on which the schema was created (cf. [Data Package](https://datapackage.org/standard/data-package/#created)).


Suggested change

The datetime on which the schema was created (cf. [Data Package](https://datapackage.org/standard/data-package/#created)).

The datetime on which the schema was created.

I think this could inherit from Data Package. Would add:

If not specified, the schema inherits from the Data Package if distributed in a Data Package descriptor.

peterdesmet · 2024-09-19T15:35:06Z

content/docs/standard/table-schema.md

+
+#### `version`
+
+A version string identifying the version of the schema (cf. [Data Package](https://datapackage.org/standard/data-package/#version)). If not specified, the schema inherits from the Data Package if distributed in a Data Package descriptor.


Suggested change

A version string identifying the version of the schema (cf. [Data Package](https://datapackage.org/standard/data-package/#version)). If not specified, the schema inherits from the Data Package if distributed in a Data Package descriptor.

A version string identifying the version of the schema. If not specified, the schema inherits from the Data Package if distributed in a Data Package descriptor.

peterdesmet · 2024-09-19T15:36:06Z

content/docs/standard/table-schema.md

+
+#### `keywords`
+
+An array of string keywords to assist users searching for the schema in catalogs.


@pierrecamilleri would you argue this inherits from Data Package or not? If so, please change to:

Suggested change

An array of string keywords to assist users searching for the schema in catalogs.

An array of string keywords to assist users searching for the schema in catalogs. If not specified, the schema inherits from the Data Package if distributed in a Data Package descriptor.

peterdesmet · 2024-09-19T15:36:50Z

content/docs/standard/table-schema.md

+
+#### `contributors`
+
+The people or organizations that contributed to the schema (cf. [Data Package](https://datapackage.org/standard/data-package/#contributors)). If not specified the schema inherits from the Data Package if distributed in a Data Package descriptor.


Suggested change

The people or organizations that contributed to the schema (cf. [Data Package](https://datapackage.org/standard/data-package/#contributors)). If not specified the schema inherits from the Data Package if distributed in a Data Package descriptor.

The people or organizations that contributed to the schema. If not specified the schema inherits from the Data Package if distributed in a Data Package descriptor.

peterdesmet · 2024-09-19T15:40:01Z

profiles/target/2.0/datapackage.json

@@ -348,7 +348,7 @@
          "homepage": {
            "propertyOrder": 70,
            "title": "Home Page",
-            "description": "The home on the web that is related to this data package.",
+            "description": "The home on the web that is related to this resource.",


Suggested change

"description": "The home on the web that is related to this resource.",

"description": "The home on the web that is related to this data package.",

@roll If these files are rebuild, it should say descriptor (general) or Data Package (specific) here.

peterdesmet · 2024-09-19T15:51:51Z

content/docs/standard/table-schema.md

+
+#### `version`
+
+A version string identifying the version of the schema (cf. [Data Package](https://datapackage.org/standard/data-package/#version)). If not specified, the schema inherits from the Data Package if distributed in a Data Package descriptor.


Suggested change

A version string identifying the version of the schema (cf. [Data Package](https://datapackage.org/standard/data-package/#version)). If not specified, the schema inherits from the Data Package if distributed in a Data Package descriptor.

A version string identifying the version of the schema. If not specified, the schema inherits from the Data Package if distributed in a Data Package descriptor.

peterdesmet · 2024-09-19T15:53:36Z

content/docs/standard/table-schema.md

+
+#### `keywords`
+
+An array of string keywords to assist users searching for the schema in catalogs.


Does this inherit from Data Package if not defined? If so, please add:

If not specified, the schema inherits from the Data Package if distributed in a Data Package descriptor.

peterdesmet · 2024-09-19T15:59:59Z

profiles/source/dictionary/schema.yaml

+      "$ref": "#/definitions/version"
+    keywords:
+      "$ref": "#/definitions/keywords"
+    contributors:


Four properties from Data Package are not carried over to Schema. I'm fine with waiting for use cases, but wanted to know your opinion on these @pierrecamilleri:

id: potentially useful. In my use case, we opted for path instead (the URL where the schema is available), which could be useful to add.

licenses: likely a bit tricky to inherit, would not add for now

image: probably seldom useful

sources: not really applicable here

pierrecamilleri · 2024-09-25T15:40:09Z

Thanks for the quality improvements !

I think we can remove the cf. Data Package everywhere,

Cf. Data Package was more to avoid repeating stuff in the documentation, for instance the datapackage.version gives precision about "It SHOULD conform to the Semantic Versioning requirements and SHOULD follow the Data Package Version recipe." that we probably don't want to repeat. So I think these links add some clarity.

WDYT ?

but indicate where it inherits from Data Package where applicable.

Concerning the inheritance, I wonder if we could not rather drop this mention of inheritance, even for version and contributors. As a user of e.g. frictionless-r or frictionless-py, this would create the expectation that if I access the version or contributors property of a schema, it automatically fills a missing field if the schema is part of a package.

I am not quite sure there is a need for this. The metadata is optional and every creator of datapackage can gauge whether it makes sense to repeat this information, and tooling can choose to look at datapackages properties when relevant.

If you think that dropping it would create confusion, OK for adding it to keywords as well.

roll

Great work @pierrecamilleri !

pierrecamilleri · 2024-11-18T14:00:31Z

I merged the next branch to resolve merge conflicts. In the changelog, I included the changes under the v2.1 header (which release PR is still in Draft status), I hope it is planned to release them under this version number. I corrected the linting errors as well.

pierrecamilleri added 19 commits July 5, 2024 10:23

Schema Metadata spec proposal

ebfc90f

🔵 wording and typos

cdbd1c9

taking into account review comments

2631462

Remove path property

83396b7

Remove updated property

d9221c6

inherits "if distribted in a data package descriptor"

ded47a6

resource -> example resource

2161fb0

name "as for Data Package

f7ffacf

Remove sources and build profile

e59b7d9

examples are Data Resources

213ef87

Attempt at removing "schema" property from table schema example

5fdd322

resources

Revert "Attempt at removing "schema" property from table schema example"

9a32b79

This reverts commit cc10899.

Revert "examples are Data Resources"

1088c70

This reverts commit 8cfb654.

Clear-up examples documentation (ref to data files)

229bbf6

fix: created stores only creation date

eddb54c

changelog

063ba61

fix: link

02c8482

fix: unintended change

fac7c7f

fix: link

e640063

pierrecamilleri added 2 commits July 12, 2024 15:29

Update doc with Peters suggestions + fix links

c5ced5b

"partial Data Resource" ?

800593f

peterdesmet added 3 commits July 15, 2024 17:05

Use title case for Data Package

3f80e97

Correct typo + align Data Package

5773c6b

Require name for examples

935cbf5

peterdesmet force-pushed the main branch from 053078f to 935cbf5 Compare July 15, 2024 15:13

peterdesmet requested changes Jul 15, 2024

View reviewed changes

pierrecamilleri and others added 2 commits July 26, 2024 14:19

Update profiles/source/dictionary/common.yaml

ad023bf

Co-authored-by: Peter Desmet <peter.desmet.work@gmail.com>

Update profiles/source/dictionary/schema.yaml

eb1117e

Co-authored-by: Peter Desmet <peter.desmet.work@gmail.com>

pierrecamilleri and others added 3 commits July 26, 2024 14:20

Update profiles/source/dictionary/schema.yaml

dbd4ebd

Co-authored-by: Peter Desmet <peter.desmet.work@gmail.com>

Update profiles/source/dictionary/schema.yaml

79d915e

Co-authored-by: Peter Desmet <peter.desmet.work@gmail.com>

Update : example title to example name

615a635

pierrecamilleri requested a review from peterdesmet July 26, 2024 12:55

peterdesmet added 2 commits September 19, 2024 17:30

Rephrase frictionlessdata#961 changes

a3233f6

Change order cf. Data Package

8288aa7

name, title, description, homepage, version, created, keywords, contributors, examples

peterdesmet approved these changes Sep 19, 2024

View reviewed changes

peterdesmet mentioned this pull request Sep 19, 2024

Automate version updating process #976

Closed

roll mentioned this pull request Oct 22, 2024

Design development model regarding branching/deployments #945

Open

roll changed the base branch from main to next October 28, 2024 10:15

roll approved these changes Oct 28, 2024

View reviewed changes

roll added the Proposal (Requires 6 voting approvals) label Oct 28, 2024

pierrecamilleri added 3 commits November 18, 2024 14:49

Merge remote-tracking branch 'upstream/next' into main

8884e5d

Changelog includes changes for v2.1

94ace89

Linting

de00124

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Promote "Metadata in Table Schema" recipe to the specs. #961

Promote "Metadata in Table Schema" recipe to the specs. #961

pierrecamilleri commented Jul 5, 2024 •

edited

Loading

roll commented Jul 8, 2024

pierrecamilleri commented Jul 12, 2024

peterdesmet left a comment •

edited

Loading

peterdesmet Jul 15, 2024

pierrecamilleri Jul 26, 2024

roll Jul 29, 2024

pierrecamilleri commented Sep 5, 2024

peterdesmet commented Sep 16, 2024

roll commented Sep 17, 2024

peterdesmet left a comment

peterdesmet Sep 19, 2024

peterdesmet Sep 19, 2024

peterdesmet Sep 19, 2024

peterdesmet Sep 19, 2024 •

edited

Loading

roll Oct 28, 2024

peterdesmet Sep 19, 2024

peterdesmet Sep 19, 2024

peterdesmet Sep 19, 2024

peterdesmet Sep 19, 2024

peterdesmet Sep 19, 2024

peterdesmet Sep 19, 2024

peterdesmet Sep 19, 2024

pierrecamilleri commented Sep 25, 2024 •

edited

Loading

roll left a comment

pierrecamilleri commented Nov 18, 2024

	A simple name or identifier for the schema (cf. [Data Package](https://datapackage.org/standard/data-package/#name)).
	A simple name or identifier for the schema.


		#### `description`

		A description of the schema (cf. [Data Package](https://datapackage.org/standard/data-package/#description)).

	A description of the schema (cf. [Data Package](https://datapackage.org/standard/data-package/#description)).
	A description of the schema.


		#### `created`

		The datetime on which the schema was created (cf. [Data Package](https://datapackage.org/standard/data-package/#created)).


		#### `version`

		A version string identifying the version of the schema (cf. [Data Package](https://datapackage.org/standard/data-package/#version)). If not specified, the schema inherits from the Data Package if distributed in a Data Package descriptor.


		#### `keywords`

		An array of string keywords to assist users searching for the schema in catalogs.


		#### `contributors`

		The people or organizations that contributed to the schema (cf. [Data Package](https://datapackage.org/standard/data-package/#contributors)). If not specified the schema inherits from the Data Package if distributed in a Data Package descriptor.

	"description": "The home on the web that is related to this resource.",
	"description": "The home on the web that is related to this data package.",

Promote "Metadata in Table Schema" recipe to the specs. #961

Are you sure you want to change the base?

Promote "Metadata in Table Schema" recipe to the specs. #961

Conversation

pierrecamilleri commented Jul 5, 2024 • edited Loading

roll commented Jul 8, 2024

pierrecamilleri commented Jul 12, 2024

peterdesmet left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pierrecamilleri commented Sep 5, 2024

peterdesmet commented Sep 16, 2024

roll commented Sep 17, 2024

peterdesmet left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

peterdesmet Sep 19, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pierrecamilleri commented Sep 25, 2024 • edited Loading

roll left a comment

Choose a reason for hiding this comment

pierrecamilleri commented Nov 18, 2024

pierrecamilleri commented Jul 5, 2024 •

edited

Loading

peterdesmet left a comment •

edited

Loading

peterdesmet Sep 19, 2024 •

edited

Loading

pierrecamilleri commented Sep 25, 2024 •

edited

Loading