Propagate input annotations to primary.cwlprov files #1678

RenskeW · 2022-06-07T16:06:07Z

Propagate structured annotations for files and directories (e.g. Schema.org, but also CWL-specific metadata fields) to the provenance graph. In addition, annotations under the cwlprov:prov field should be incorporated in the RDF graph as well.

Step 1: If 'label' is specified in input object, it is added as an annotation to the file entity it corresponds to.

Example:

file: 
  class: File
  path: ./file.tsv
  label: file_label

This would be represented in the cwlprov provenance like this:

id:92fc295f-f6cb-444b-8ffe-0f604b127858 a wf4ever:File,
        wfprov:Artifact,
        prov:Entity ;
    rdfs:label "file_label"^^xsd:string ;
    prov:specializationOf data:86ec86fdbbe70863a78453f69349568f9d1f14a1 ;
    cwlprov:basename "file.tsv"^^xsd:string ;
    cwlprov:nameext ".tsv"^^xsd:string ;
    cwlprov:nameroot "file"^^xsd:string .

See here for the full example RO.

Step 2: Values of doc, format, label, intent are propagated to RDF provenance graph.

EXAMPLE RO: https://github.com/RenskeW/cwlprov-provenance/tree/930cc268da77c5aa3739a2f7f87b94e076b144e6/cwlprov_rdf_examples

Mapping to Schema.org terms:

doc: https://schema.org/description
label: https://schema.org/name
format: https://schema.org/encodingFormat
intent: https://schema.org/featureList

TODO:

Propagate annotations for File
Propagate annotations for Directory
Add support for annotations of type CommentedSeq
Propagate annotations under cwlprov:prov
Propagate values of doc, intent, label, and intent fields to RDF
inputs should be copied into data/ , not be broken symlinks
improve code coverage
Fix double entries in provenance (optionally)

codecov · 2022-06-07T16:24:49Z

Codecov Report

Attention: 16 lines in your changes are missing coverage. Please review.

Comparison is base (bcdbeaf) 83.80% compared to head (dea0b8e) 83.92%.

Files	Patch %	Lines
cwltool/cwlprov/provenance_profile.py	71.69%	11 Missing and 4 partials ⚠️
cwltool/singularity.py	0.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1678      +/-   ##
==========================================
+ Coverage   83.80%   83.92%   +0.11%     
==========================================
  Files          46       46              
  Lines        8221     8266      +45     
  Branches     2182     2201      +19     
==========================================
+ Hits         6890     6937      +47     
+ Misses        854      850       -4     
- Partials      477      479       +2

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

mr-c · 2022-06-08T09:15:37Z

Thank you for this @RenskeW ! can you make format and push the changes?

mr-c · 2022-06-09T08:25:29Z

@RenskeW Thanks!

Checking https://www.commonwl.org/v1.2/CommandLineTool.html#File we see that label is not one of the fields present here in the CWL standards. So we'll need to use a namespaced field name, for which I recommend that you choose one of the schema.org properties like https://schema.org/name as it is linked to rdfs:label and is modeled as equivalent to dcterms:title.

After you make that change, can you add a s:name to one of the input objects used in the cwlprov tests? You'll need to add a $namespaces dictionary to document the s prefix if the corresponding CWL description lacks that. Then you can add a check to one of the cwlprov tests to look for the rdfs:label in the resulting CWLProv RO.

mr-c · 2022-06-09T08:26:09Z

(maybe in CWL v1.3 we will add a label field to File)

lgtm-com · 2022-06-20T10:56:15Z

This pull request introduces 1 alert when merging 99fc196 into c23a9ed - view on LGTM.com

new alerts:

1 for Variable defined multiple times

lgtm-com · 2022-06-22T13:53:37Z

This pull request introduces 1 alert when merging c6e7385 into b21b0c1 - view on LGTM.com

new alerts:

1 for Variable defined multiple times

RenskeW · 2022-06-27T11:22:26Z

cwltool/provenance_profile.py

+
+        def _add_nested_annotations(dataset, e: ProvEntity) -> ProvEntity:
+            for annotation in dataset:
+                if isinstance(dataset[annotation], (str, bool, int, float)): # check if these are all allowed types


isinstance(dataset[annotation], list), isinstance(dataset[annotation], MutableMapping)

MutableSequence = list

cwltool/provenance_profile.py

lgtm-com · 2022-07-04T08:42:16Z

This pull request introduces 2 alerts when merging bf34ca6 into bb41504 - view on LGTM.com

new alerts:

2 for Incomplete URL substring sanitization

RenskeW · 2022-07-05T08:20:53Z

cwltool/provenance_profile.py

+            #         # how do we identify the correct file to write to? self.workflow_run_uri?
+            #         # 
+            #         pass
+


First step in propagating metadata under cwlprov:prov to provenance as well.

lgtm-com · 2022-07-05T08:27:28Z

This pull request introduces 2 alerts when merging 70d184f into bb41504 - view on LGTM.com

new alerts:

2 for Incomplete URL substring sanitization

cwltool/provenance_profile.py

cwl-bot · 2023-03-25T07:38:26Z

This pull request has been mentioned on Common Workflow Language Discourse. There might be relevant details there:

https://cwl.discourse.group/t/equivalent-to-file-metadata-for-local-environment/782/2

cwltool/cwlprov/provenance_profile.py

TODO: fix intent list add/amend tests

RenskeW · 2024-01-22T12:32:06Z

@mr-c here is a hand-annotated example of the new RDF provenance graph, including the CWL metadata fields:

https://github.com/RenskeW/cwlprov-provenance/blob/d0b69c24e57b6db9e7d061f9512ba35e842c9cc5/cwlprov_rdf_examples/scenario1/ro/metadata/provenance/primary.cwlprov.ttl

There is a separate example, containing propagation of annotation of input parameter values (e.g. an input File), to CWLprov RDF, here:

https://github.com/RenskeW/cwlprov-provenance/tree/930cc268da77c5aa3739a2f7f87b94e076b144e6/prov_data_annotations/example1

Propagate 'label' to primary.cwlprov file

b954975

RenskeW requested a review from mr-c June 7, 2022 16:06

Run make format

93f091a

Propagate schema.org annotations to provenance

99fc196

Added format annotations to provenance + started Directory annotations

c6e7385

update provenance_profile

7a509b6

RenskeW commented Jun 27, 2022

View reviewed changes

RenskeW added 2 commits July 3, 2022 22:53

added support for file arrays + https://schema.org

79f8b74

Directory annotations propagated to provenance

bf34ca6

RenskeW commented Jul 4, 2022

View reviewed changes

cwltool/provenance_profile.py Outdated Show resolved Hide resolved

github-advanced-security bot found potential problems Jul 4, 2022

View reviewed changes

cwltool/provenance_profile.py Fixed Show fixed Hide fixed

cwltool/provenance_profile.py Fixed Show fixed Hide fixed

RenskeW commented Jul 4, 2022

View reviewed changes

cwltool/provenance_profile.py Outdated Show resolved Hide resolved

RenskeW added 2 commits July 5, 2022 10:10

Added support for CommentedSeq

1598959

Transferred annotations from data: to id: entity

70d184f

RenskeW commented Jul 5, 2022

View reviewed changes

mr-c and others added 2 commits December 14, 2022 11:50

Merge branch 'main' into input_annotations

a230044

formatting

6909002

github-advanced-security bot found potential problems Dec 14, 2022

View reviewed changes

cwltool/provenance_profile.py Fixed Show fixed Hide fixed

cwltool/provenance_profile.py Fixed Show fixed Hide fixed

mr-c force-pushed the input_annotations branch from e5c9176 to 00cf177 Compare December 14, 2022 16:31

basic test & fix typing

e2009ac

mr-c force-pushed the input_annotations branch from 00cf177 to e2009ac Compare December 15, 2022 09:13

github-advanced-security bot found potential problems Dec 15, 2022

View reviewed changes

cwltool/provenance_profile.py Fixed Show fixed Hide fixed

cwltool/provenance_profile.py Fixed Show fixed Hide fixed

start of a more advanced test

5466f95

mr-c force-pushed the input_annotations branch from 09193f7 to 5466f95 Compare December 15, 2022 17:24

Merge remote-tracking branch 'origin/main' into input_annotations

faddc3a

mr-c force-pushed the input_annotations branch from 95def37 to faddc3a Compare November 13, 2023 11:56

github-advanced-security bot found potential problems Nov 13, 2023

View reviewed changes

cwltool/cwlprov/provenance_profile.py Fixed Show fixed Hide fixed

cwltool/cwlprov/provenance_profile.py Fixed Show fixed Hide fixed

mr-c and others added 4 commits November 13, 2023 13:16

Renske's surname is not a typo

de090f9

Merge branch 'main' into input_annotations

45ddd8d

Merge branch 'main' into input_annotations

c13e26a

Merge remote-tracking branch 'origin/main' into input_annotations

8edabf8

mr-c force-pushed the input_annotations branch 4 times, most recently from 25b3ddb to fad5e8e Compare December 18, 2023 13:34

github-advanced-security bot found potential problems Dec 18, 2023

View reviewed changes

cwltool/cwlprov/provenance_profile.py Dismissed Show dismissed Hide dismissed

cwltool/cwlprov/provenance_profile.py Dismissed Show dismissed Hide dismissed

mr-c force-pushed the input_annotations branch from fad5e8e to 9e10585 Compare December 18, 2023 14:46

store label & doc fields as prospective provenance

dea0b8e

TODO: fix intent list add/amend tests

mr-c force-pushed the input_annotations branch from 9e10585 to dea0b8e Compare December 18, 2023 15:44

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Propagate input annotations to primary.cwlprov files #1678

Propagate input annotations to primary.cwlprov files #1678

RenskeW commented Jun 7, 2022 •

edited by mr-c

Loading

codecov bot commented Jun 7, 2022 •

edited

Loading

mr-c commented Jun 8, 2022

mr-c commented Jun 9, 2022

mr-c commented Jun 9, 2022

lgtm-com bot commented Jun 20, 2022

lgtm-com bot commented Jun 22, 2022

RenskeW Jun 27, 2022

RenskeW Jun 27, 2022

lgtm-com bot commented Jul 4, 2022

RenskeW Jul 5, 2022

lgtm-com bot commented Jul 5, 2022

cwl-bot commented Mar 25, 2023

RenskeW commented Jan 22, 2024

Propagate input annotations to primary.cwlprov files #1678

Are you sure you want to change the base?

Propagate input annotations to primary.cwlprov files #1678

Conversation

RenskeW commented Jun 7, 2022 • edited by mr-c Loading

codecov bot commented Jun 7, 2022 • edited Loading

Codecov Report

mr-c commented Jun 8, 2022

mr-c commented Jun 9, 2022

mr-c commented Jun 9, 2022

lgtm-com bot commented Jun 20, 2022

lgtm-com bot commented Jun 22, 2022

RenskeW Jun 27, 2022

Choose a reason for hiding this comment

RenskeW Jun 27, 2022

Choose a reason for hiding this comment

lgtm-com bot commented Jul 4, 2022

RenskeW Jul 5, 2022

Choose a reason for hiding this comment

lgtm-com bot commented Jul 5, 2022

cwl-bot commented Mar 25, 2023

RenskeW commented Jan 22, 2024

RenskeW commented Jun 7, 2022 •

edited by mr-c

Loading

codecov bot commented Jun 7, 2022 •

edited

Loading