[ENH] Added the specification for using HED libraries in BIDS #1106

VisLab · 2022-05-23T22:43:24Z

This is a preliminary request for change in the BIDS specification to accommodate HED libraries.

The first official HED library schema (SCORE library for clinical neurological annotation) is close to release.

We will prepare a PR for the bids-validator to validate datasets using HED libraries as well as an example datasets for bids-examples when there is agreement on the format.

We would appreciate reviews and comments: @sappelhoff @Remi-Gau @tsalo (the yaml could be improved--please help) @happy5214 @IanCa, @dorahermes @dungscout96 @smakeig, @tpatpa.

codecov · 2022-05-23T22:54:04Z

Codecov Report

Merging #1106 (6101459) into master (b3afa2a) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master    #1106   +/-   ##
=======================================
  Coverage   88.33%   88.33%           
=======================================
  Files           6        6           
  Lines        1020     1020           
=======================================
  Hits          901      901           
  Misses        119      119

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

src/99-appendices/03-hed.md

tpatpa · 2022-05-24T16:55:43Z

Thank you @VisLab, looks good to me!

yarikoptic

was just curious to review , left some questions/feedback

src/99-appendices/03-hed.md

yarikoptic · 2022-05-26T20:45:13Z

src/99-appendices/03-hed.md

+  "HEDVersion": {
+      "base": "8.1.0",
+      "libraries": {
+          "sc": "score_0.0.1",


is this some "official" identity specification combining library and a version? if not, may be worth separating out, even if more verbose:

Suggested change

"sc": "score_0.0.1",

"sc": {"library": "score", "version": "0.0.1"},

or alike to make it explicit, and to allow in the future e.g. to expand with optional extra information (e.g. URL to the HED library which is not in the library of libraries i.e. hed-schema-library) but available from another library of libraries or may be even directly pointed to by that URL?

Some version of what you suggest was the original plan (though URIs were never planned to be supported other than bids: schema paths within the dataset), but it was simplified during development after feedback from the HED community. @VisLab can provide more detail, which is escaping me.

The suggestion is above is one that we have considered (see hed-specification issue#156). The actual format is only proposed at this point in the HED specification, and we would appreciate opinions and are open to changes.

We have gone through a number of versions of this, and originally we thought we would support a file within the bids data set itself and arbitrary URLs. The HED working group consensus was to start with the standard libraries and see how it goes, since the purpose is to have standardized vocabularies to make meaningful comparisons across datasets.

On a related note:
The hed-python tools allow a schema group to be passed into its BIDS data set constructor to override the specification in dataset_description. Right now the hed-javascript public interface (which is what BIDS uses) only passes in the data set and constructs the schema group internally.

I am mentioning this because another possibility is to allow a schema group to be passed in as part of the public interface to the hed-javascript validator (from bids) which would override the internal specification extracted from the dataset_description. This is relevant because we are going to have to do a major version bump to support the libraries because of another interface change and we could add this in as an option and allow BIDS to decide. @sappelhoff. @rwblair?

The specification of which schemas are to be used to be used for validating a dataset are provided in a separate parameter, since hed-javascript cannot handle relative paths within the BIDS dataset directory and needs them parsed to use the absolute path. However, the actual Schema and Schemas types used by the validator to represent schemas are considered implementation details (despite being returned by the validator.buildSchema function) and not part of the stable API. In theory, the object passed as the second argument to validateBidsDataset can be any object conforming to the defined API, regardless of what datasetDescription.json says.

src/99-appendices/03-hed.md

src/schema/objects/metadata.yaml

src/schema/rules/tabular_metadata.yaml

sappelhoff

@VisLab if you want to move on with this PR towards a merge, please let us know and remove the "draft" status.

VisLab · 2022-06-07T13:11:07Z

I would like to hold off for another week to allow more internal discussion.

In working on the example use cases using the score library, it has become apparent that many users will just want to use the score library when they are doing neurological annotation and not use the base schema at all.

We have the option of allowing the following so that users wouldn't have to prefix the score tags:

"HEDVersion": {
      "base": "score_0.0.1",
      "libraries": {
          "bs": "8.0.0"
       }
}

or also:

"HEDVersion":  "score_0.0.1"

@tpatpa @dorahermes comments?

…ication into hed-library

VisLab · 2022-06-09T21:53:57Z

@tpatpa @happy5214 @dorahermes @sappelhoff @dungscout96 @tsalo @yarikoptic
Please review. After discussion.... the proposal has been simplified... for the better.

tpatpa

Thank you @VisLab for a great discussion!
After spending additional time putting together the bids example dataset (still WIP, available here) and working with the HED validation tools I feel the use of a string would be easier altogether.

sappelhoff

Am I correct in assuming that underscores _ are forbidden characters when naming an HED library? Or do you document somewhere how to unambiguously get library name and library version from the suggested format <library-name>_<library-version>.

Relatedly: Are : characters also forbidden at the start of an HED library? Else I think the situation needs to be explicitly clarified when something like this happens: People having a library called :myThing and then want to do "HEDVersion": ["8.0.0", "abbr::myThing"] (note the double :)

Overall I like the proposed format of specifying which HED library is being used 👍 it looks much simpler than before and still encoding the relevant info. We only need to take care that the "separator characters" (_ and :, apparently) are always unambiguous.

One question regarding that: Do you prescribe a versioning scheme for all "official" HED libraries? For example, MUST they always use semantic versioning?

src/99-appendices/03-hed.md

sappelhoff · 2022-06-11T14:52:15Z

src/schema/objects/metadata.yaml

+  - type: string
+  - type: array
+    items:
+      type: string


might be beneficial to think about what validation we can build into the schema here apart from "string". E.g., checking for permissible versioning formats, occurrence of : and _ characters, etc.

That is a good idea if someone knows how to do it that would be great.

Try adding this behemoth regex to src/schema/objects/formats.yaml (the first two groups are for the library nickname and library name, and the rest was adapted from https://semver.org/#is-there-a-suggested-regular-expression-regex-to-check-a-semver-string):

^(?:[a-zA-Z]+:)?(?:[a-zA-Z]+_)?(?:0|[1-9]\d*)\.(?:0|[1-9]\d*)\.(?:0|[1-9]\d*)(?:-(?:(?:0|[1-9]\d*|\d*[a-zA-Z-][0-9a-zA-Z-]*)(?:\.(?:0|[1-9]\d*|\d*[a-zA-Z-][0-9a-zA-Z-]*))*))?(?:\+(?:[0-9a-zA-Z-]+(?:\.[0-9a-zA-Z-]+)*))?$

thanks @happy5214 what's your opinion on adding this regex for validation? Will it cause more pain than it's worth, or should we do it?

I was pinged, but I don't know if the question was directed toward me or if you were just thanking me. The tail end of the regex could be simplified to the following if we were to ban the use of pre-release schema versions:

^(?:[a-zA-Z]+:)?(?:[a-zA-Z]+_)?(?:0|[1-9]\d*)\.(?:0|[1-9]\d*)\.(?:0|[1-9]\d*)$

both to thank you and to get your opinion, because you called it a "behemoth" and I wasn't sure whether that means you advise against adding this. :-)

Meanwhile this discussion progresses in the main thread

Co-authored-by: Stefan Appelhoff <stefan.appelhoff@mailbox.org>

dorahermes · 2022-07-08T18:00:25Z

great! lgtm

- dataset_description.json fixed and waiting for [bids-specification pull request](bids-standard/bids-specification#1106) - units added to _channels.tsv for TUH subjects - Files truncated via [terminal](https://github.com/bids-standard/bids-examples/blob/master/CONTRIBUTING.md#why-do-we-only-host-truncated-data-with-0kb-size)

sappelhoff

It seems to me the only big discussion that is left open is on how to define libraries (#1106 (comment)) -- currently proposed form is <library-name<_>library-version>.

If in the meantime the HED team has converged and made a decision, we can resolve that discussion and proceed with this PR.

We'll also need to adjust the validator ... currently the score example is failing CI because of this:

Example of the HED schema library for SCORE implementation bids-examples#324

src/99-appendices/03-hed.md

sappelhoff · 2022-07-22T06:10:24Z

src/schema/objects/metadata.yaml

+  - type: string
+  - type: array
+    items:
+      type: string


thanks @happy5214 what's your opinion on adding this regex for validation? Will it cause more pain than it's worth, or should we do it?

VisLab · 2022-07-22T13:12:28Z

This version is correct and has been agreed upon by the Hed Working Group. The bids-validator part isn't quite ready yet. I'll try to work on a PR for the BIDS validator side, which just needs a small change.

The modifications needed for the HED side aren't quite there yet, but we can probably write a by-pass temporarily. (We recommend that users check their validation using the Python validator until we get it completed.) @happy5214

As for a link to a library: the SCORE library official 1.0.0 was finished and out for community comment. @tpatpa @dorahermes need to give final approval and then we can release it and provide the stable link. (We know what the link will be but once we put the schema there, tools will start using it.)

It would be nice if this made it into the new release. What time frame do we have for finishing the validation side?

VisLab · 2022-07-22T15:10:04Z

I think we should wait on the validation of the HEDVersion using a regular expression unless this is standard for BIDS and we have to do it. I would rather just pass the HEDVersion to the hed-validator and let it resolve the version. On the first pass, we would only support the straight version strings as shown in the example.

However, longer term we would support BIDS URI's, the PR of which I notice is also moving along.

sappelhoff · 2022-07-22T15:25:53Z

The 1.8 release will probably come mid September. It'd be great to have the new hed features in by then!

happy5214 · 2022-07-22T15:29:18Z

I think we should wait on the validation of the HEDVersion using a regular expression unless this is standard for BIDS and we have to do it. I would rather just pass the HEDVersion to the hed-validator and let it resolve the version. On the first pass, we would only support the straight version strings as shown in the example.

I believe BIDS has the primary responsibility to validate HEDVersion, whether or not we do our own validation, especially if it's as simple as adding a regex to an already-implemented framework. We implement the functionality on our end as a matter of convenience and expedience, but in the end this is a format defined by BIDS, not hed-validator.

VisLab · 2022-08-29T17:56:16Z

We have now released hed-validator 3.8.0 on npm supporting HED library schema. Could we move this PR forward now that it has supporting PRs bids-standard/bids-validator#1496 and bids-standard/bids-examples#332?

I think all three PRs are ready to go. @sappelhoff @Remi-Gau @rwblair

sappelhoff · 2022-08-29T20:24:12Z

I think the only open question is whether or not we want to add the regexp suggested by @happy5214 to the BIDS schema validation.

If you have strict rules about how versions for HED need to look like, then I think it'd be a good idea ... even if the regexp looks a bit unwieldy, as described in #1106 (comment)

cc @VisLab

src/schema/objects/metadata.yaml

src/schema/objects/formats.yaml

sappelhoff · 2022-08-30T11:32:20Z

I have added the hed_version format as suggested by @happy5214. If you disagree, we can discuss removing it again -- but I think it's a good idea to validate HED versions early on, and not only in the HED validator.

VisLab · 2022-08-30T21:16:30Z

Yes the RegEx is fine. I had a chance to test on our set of testcases. Thanks....

sappelhoff · 2022-08-31T08:54:42Z

Thanks @VisLab et al.

🎉

VisLab · 2022-09-01T10:32:06Z

Special thanks to @happy5214

VisLab · 2022-10-11T08:28:58Z

I would prefer to defer adding this regular expression. We do use regular expressions as part of the checking of schema for the hed-validator, but they are not quite this regular expression. Let's put this on a todo list for a future PR.

…

On Mon, Aug 29, 2022 at 3:24 PM Stefan Appelhoff ***@***.***> wrote: I think the only open question is whether or not we want to add the regexp suggested by @happy5214 <https://github.com/happy5214> to the BIDS schema validation. If you have strict rules about how versions for HED need to look like, then I think it'd be a good idea ... even if the regexp looks a bit unwieldy, as described in #1106 (comment) <#1106 (comment)> cc @VisLab <https://github.com/VisLab> — Reply to this email directly, view it on GitHub <#1106 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAJCJOXSK3DPY4Q6UTEYEQTV3UL7PANCNFSM5WXIN25A> . You are receiving this because you were mentioned.Message ID: ***@***.***>

- dataset_description.json fixed and waiting for [bids-specification pull request](bids-standard/bids-specification#1106) - units added to _channels.tsv for TUH subjects - Files truncated via [terminal](https://github.com/bids-standard/bids-examples/blob/master/CONTRIBUTING.md#why-do-we-only-host-truncated-data-with-0kb-size)

VisLab added 3 commits May 23, 2022 17:32

Added the specification for using HED libraries in BIDS

6ce58dc

Added newline at end of hed appendices

976559a

Corrected the metadata.yml spacing

6555ad5

sappelhoff reviewed May 24, 2022

View reviewed changes

src/99-appendices/03-hed.md Outdated Show resolved Hide resolved

Added requirement that user-chosen prefixes be alphanumeric

dc723e4

yarikoptic reviewed May 26, 2022

View reviewed changes

sappelhoff reviewed Jun 7, 2022

View reviewed changes

VisLab mentioned this pull request Jun 9, 2022

Implement support for library schemas hed-standard/hed-javascript#67

Closed

VisLab added 6 commits June 9, 2022 16:16

Merge branch 'master' of https://github.com/bids-standard/bids-specif…

d52c011

…ication into hed-library

Simplified HEDVersion to be array rather than object

8039ec9

Provided example using only SCORE library

ccf749b

Made some typo and editorial corrections

ce26293

Changed base schema to standard schema

a0a7b36

Removed extra colon

37e9b8f

VisLab marked this pull request as ready for review June 9, 2022 21:50

tpatpa approved these changes Jun 9, 2022

View reviewed changes

sappelhoff reviewed Jun 11, 2022

View reviewed changes

VisLab and others added 2 commits June 11, 2022 10:39

Update src/99-appendices/03-hed.md

42708ef

Co-authored-by: Stefan Appelhoff <stefan.appelhoff@mailbox.org>

Update src/99-appendices/03-hed.md

f26bcd1

Co-authored-by: Stefan Appelhoff <stefan.appelhoff@mailbox.org>

sappelhoff added this to the 1.7.1 milestone Jun 22, 2022

dorahermes mentioned this pull request Jul 8, 2022

Example of the HED schema library for SCORE implementation bids-standard/bids-examples#324

Open

sappelhoff modified the milestones: 1.7.1, 1.8.0 Jul 13, 2022

sappelhoff reviewed Jul 22, 2022

View reviewed changes

VisLab mentioned this pull request Aug 4, 2022

Implementation and basic unit tests for HEDVersion with libraries bids-standard/bids-validator#1496

Merged

VisLab mentioned this pull request Aug 28, 2022

Hed library example addition bids-standard/bids-examples#332

Merged

Merge branch 'master' into hed-library

cfea5f7

sappelhoff requested a review from erdalkaraca as a code owner August 29, 2022 20:15

sappelhoff added 2 commits August 29, 2022 22:15

Update src/99-appendices/03-hed.md

5310be6

Update metadata.yaml

a49df64

add new format: hed_version

55d79a6

sappelhoff reviewed Aug 30, 2022

View reviewed changes

src/schema/objects/metadata.yaml Outdated Show resolved Hide resolved

sappelhoff reviewed Aug 30, 2022

View reviewed changes

src/schema/objects/formats.yaml Outdated Show resolved Hide resolved

sappelhoff added 2 commits August 30, 2022 13:28

line break pattern

5c7045d

Update src/schema/objects/metadata.yaml

6101459

sappelhoff approved these changes Aug 30, 2022

View reviewed changes

sappelhoff merged commit da68bd9 into bids-standard:master Aug 31, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[ENH] Added the specification for using HED libraries in BIDS #1106

[ENH] Added the specification for using HED libraries in BIDS #1106

VisLab commented May 23, 2022 •

edited

Loading

codecov bot commented May 23, 2022 •

edited

Loading

tpatpa commented May 24, 2022 •

edited

Loading

yarikoptic left a comment

yarikoptic May 26, 2022

happy5214 May 26, 2022

VisLab May 27, 2022

happy5214 Jun 1, 2022

sappelhoff left a comment

VisLab commented Jun 7, 2022

VisLab commented Jun 9, 2022

tpatpa left a comment •

edited

Loading

sappelhoff left a comment

sappelhoff Jun 11, 2022

VisLab Jun 11, 2022

happy5214 Jun 11, 2022

sappelhoff Jul 22, 2022

happy5214 Jul 22, 2022

sappelhoff Jul 22, 2022

dorahermes commented Jul 8, 2022

sappelhoff left a comment

sappelhoff Jul 22, 2022

VisLab commented Jul 22, 2022

VisLab commented Jul 22, 2022

sappelhoff commented Jul 22, 2022

happy5214 commented Jul 22, 2022

VisLab commented Aug 29, 2022

sappelhoff commented Aug 29, 2022

sappelhoff commented Aug 30, 2022

VisLab commented Aug 30, 2022

sappelhoff commented Aug 31, 2022

VisLab commented Sep 1, 2022

VisLab commented Oct 11, 2022 via email

	"sc": "score_0.0.1",
	"sc": {"library": "score", "version": "0.0.1"},

[ENH] Added the specification for using HED libraries in BIDS #1106

[ENH] Added the specification for using HED libraries in BIDS #1106

Conversation

VisLab commented May 23, 2022 • edited Loading

codecov bot commented May 23, 2022 • edited Loading

Codecov Report

tpatpa commented May 24, 2022 • edited Loading

yarikoptic left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sappelhoff left a comment

Choose a reason for hiding this comment

VisLab commented Jun 7, 2022

VisLab commented Jun 9, 2022

tpatpa left a comment • edited Loading

Choose a reason for hiding this comment

sappelhoff left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dorahermes commented Jul 8, 2022

sappelhoff left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

VisLab commented Jul 22, 2022

VisLab commented Jul 22, 2022

sappelhoff commented Jul 22, 2022

happy5214 commented Jul 22, 2022

VisLab commented Aug 29, 2022

sappelhoff commented Aug 29, 2022

sappelhoff commented Aug 30, 2022

VisLab commented Aug 30, 2022

sappelhoff commented Aug 31, 2022

VisLab commented Sep 1, 2022

VisLab commented Oct 11, 2022 via email

VisLab commented May 23, 2022 •

edited

Loading

codecov bot commented May 23, 2022 •

edited

Loading

tpatpa commented May 24, 2022 •

edited

Loading

tpatpa left a comment •

edited

Loading