Add release notes to published Dandiset

[yarikoptic]

• to relate to git commit or tag -a - it would be useful to everyone to let users annotate their published versions: what has changed and why

• should probably be some freeform TEXT blob of some smallish size

• it would allow to establish a "changelog" for each dandiset (not just some not-so-informative versions)

• probably should not be mandatory but I am ok to make it such: that could help to avoid users to unnecessarily mint versions

• like git commit message it is IMHO ok to have it immutable thus no need to expose some "edit" functionality via API

Ideally web ui, upon Publish should provide a pane with "Publish" and "Cancel" buttons which would have

• version to be published (could be in the title)
• text box with pre-populated grayed out text encouraging a good practice following git commits convention, i.e.

Short description of changes

Optional detailed description or reasoning for changes

which disappears as soon as the user starts typing; or simply two separate TEXT boxes

• somewhere near that edit box it could provide a summary to the user on assets etc changed since previous version (if any; from blank state otherwise - so only "added"), e.g.

Changes since version 0.210326.2211
- assets:
  - added 2 (10TB)
  - deleted 3 (3TB)
  - replaced 4 (+1TB)
- dandiset:
  - metadata changed

(everything easily computable from prior and draft version list of assets), and if that would be expanadable tree showing details (assets paths, diff for dandiset metadata change) -- that would be truly useful!

TODOs:

• parse/add entries from already possibly present in the archive BIDS CHANGES files, like in https://dandiarchive.org/dandiset/000037/0.240209.1623

[yarikoptic]

there was recent user inquiry on that. Pasting from the slack chat

I think the ideal for my workflow might be a UI element in dandi-archive that requests an entry when publishing, and appends it properly in a CHANGELOG file, which would then be viewable in the UI also. We could then also just backfill entries manually in the same format.
Or maybe storing in the DB would be preferable, as long as there's a way to fill older entries (via dandi-cli maybe). Basically any option would be great! And diffs were not at all in scope on my thoughts, but interesting point that they can be viewed in some form via the datalad versions (if someone wants to verify a changelog entry).

related prior response from me was

we support BIDS and BIDS indeed optionally asks for CHANGES file (note, not .md. Should follow https://metacpan.org/release/HAARG/CPAN-Changes-0.400002/source/lib/CPAN/Changes/Spec.pod ).Good that it is optional. But we might want to provide it.
Since dandi-archive is the one which mints the release version, it is doomed to be dandi-archive to accept the changelog-entry and record it in DB. Whatever then can publish (CLI doesn't really IIRC, web UI does) should ask user for description, and if we decide to autopopulate CHANGES (for BIDS or not) we can.

[bendichter]

Another user story here requesting the "release notes" feature:

[satra]

this needs to first be done in dandi-schema.

something like adding this to PublishActivity model.

releaseNotes: str = Field(description="Comment for publishing")

and then the UI needs to have a mandatory element related to this.

[yarikoptic]

FTR: @colleenjg also described this use case in dandi/dandi-cli#1291

[colleenjg]

I was wondering whether there are any updates on including release notes / descriptions to the dandiset versions?

For our dandiset, I've now added our publications to our metadata. I'd like to publish this new version, but I'm keen to do this with release notes. Otherwise, I worry that people might incorrectly think that there are substantive changes to the content of the dandiset, and then be confused about which version to use.

[yarikoptic]

Unfortunately no usable updates yet -- this was kept in the backburner unfortunately... the "back" starts to burn ;)

To expedite your case/release, I am thinking: we are working with BIDS standard to converge on it being usable for animal neurophys etc. BIDS standard reserves CHANGES file to annotate changes/releases. Should follow the CPAN notation notation. So I think, overall, we can adopt convention of having such a file in dandisets following our ad-hoc BIDS-inspired DANDI layout too. Later, when we do finally merge dandi/dandi-schema#185 and add web UI interface to enter etc -- we could extract from already existing CHANGES if any exist.

We do have a "chicken/egg" issue here, and largely because our versions are automatically minted and time versioned, e.g. 0.210831.2033 so it is virtually impossible to determine it ahead of time exactly. We might want to reconsider versioning scheme (e.g. exclude time or make it .0 unless there is already a release for that date, and then add .1 or timepoint to disambiguate). But meanwhile CHANGES could use version without time, e.g. 0.210831, in the record.
Then upload such CHANGES file: trick would be to use

DANDI_DEVEL=1 dandi upload ...options you would use... --allow-any-path CHANGES

and later we would make CHANGES considered automagically.

WDYT @satra ?

[colleenjg]

Thank you for taking the time to propose a detailed workaround! If you both think it's a good option, then I'm on board. I'll just need some additional guidance at the implementation stage.

[satra]

fine with me.

[colleenjg]

I'm leaving a trace of my process here, in case others also want to add a changelog to their dataset.

So, if I understand correctly, the idea is that I upload a CHANGES file to dandi, and then publish the new version of our dandiset on the same day, to make sure the minted DOI matches the version number I included in my file. So, for example, if I were planning to publish today:

CHANGES file
Here is what I would put in the CHANGES file:

Revision history for dandiset #37

0.240207 - 2024-02-07
 - Added link to Dataset descriptor paper.
 - Added link to Analysis paper.
 - Added changelog.

0.230426.0054 - 2023-04-26
 - Initial release.

Adding the CHANGES file to the dandiset
I've only ever followed the steps from Creating Dandisets and Uploading Data, i.e., running validate and organize. Our dataset is 2TB, so I don't want to download the whole thing just to add one file. Can I simply run DANDI_DEVEL=1 dandi upload --allow-any-path CHANGES, or are the additional steps I should take?

A few more details

• To confirm, the file should simply be called CHANGES (not changelog, changelod.md, etc.)
• The file should be placed in the top directory of my dandiset?
• I've included a short title, which is optional in the CPAN changelog notation @yarikoptic linked. Please let me know if you would suggest something different.

[yarikoptic]

Can I simply run DANDI_DEVEL=1 dandi upload --allow-any-path CHANGES, or are the additional steps I should take?

yes, but you must be in the dandiset folder with dandiset.yaml, so can do smth like

dandi download --download dandiset.yaml https://dandiarchive.org/dandiset/000037
cd 000037
cp /somewhere/CHANGES .
DANDI_DEVEL=1 dandi upload --allow-any-path CHANGES

• To confirm, the file should simply be called CHANGES (not changelog, changelod.md, etc.)

yes, according to https://bids-specification.readthedocs.io/en/stable/glossary.html#changes-files

• The file should be placed in the top directory of my dandiset?

yes

• I've included a short title, which is optional in the CPAN changelog notation @yarikoptic linked. Please let me know if you would suggest something different.

so far looks good to me and tons better than nothing ;-)

[colleenjg]

Thank you @yarikoptic !

[colleenjg]

Done! Thank you for your detailed help making this happen @yarikoptic!

[yarikoptic]

Great, meanwhile I added explicit TODO in the original description of the issue to parse out CHANGES whenever we get metadata level support for them to associate with release metadata records.