GKS product schema folder organization conventions

[larrybabb]

Below is a perspective that is pretty close to how we now use folders/subfolders etc... except where I point it out in parens. This approach is mostly organizing by general categories of class groups with 'profiles' being the most vague.

gks-common
    schema
        gks-common
            core-im
                gks-core-im-source.yaml ( suggested to be both core-im and data-types )
            domain-entity-profiles ( name is currently domain-entities, but really these are subtypes/profiles )
                gks-domain-entity-profiles-source.yaml.  (one file with all the subtypes of domain-entities that gks needs to author and support?)

vrs
    schema
        vrs
            vrs-source.yaml

cat-vrs
    schema
        cat-vrs
            core
                cat-vrs-core-source.yaml
            profiles
                cat-vrs-profiles-source.yaml. (one file with ALL the forthcoming cat-vrs profiles?)

va-spec
    schema
        va       ( shorthand for `profiles` only which is the non-core-im part of va. this does not exist currently )
            profiles. (this sits up one level)
                <several *-source.yaml files for various groups of profiles (e.g. caf, vp, var-study, ave ...)>

I think we really should consider grouping the yaml files based on how they will be used in functional groups rather than by high-level categories (like 'profile'). The core or core-im term works pretty well when the yaml file contains a group of classes that work together foundation-ally and is not necessary or pragmatic to break up. However, the profiles folder is a bit short-sighted IMO in that we will likely have many types of profiles some of which will work in a similar functional area together. For example the AVE profiles (3 of them) can be in a single AVE-source.yaml or an AVE-profiles folder (or just an AVE folder). same goes for CAF and VP. For Cat-VRS the profiles will also likely be broken up based on the varied types of categorical variants that seem to be possible.

[larrybabb]

The reason this is useful to sort out now is so that we have consistency across the gks products. And since our products are built into a hierarchical dependency (DAG based) it makes sense to have some conventions that are followed to make users and implementors that want to further extend these profiles and classes easier to follow.

[larrybabb]

In the end, there are 2 questions, the first is pretty straightforward.

Q1. Can I merge the gks-common data-types yaml in with the gks-common core-im yaml thus changing gks-common to have only 2 sets of class files core-im (which will include the data-types) and domain-entities instead of the 3 that exist now (core-im, data-types and domain-entities). It has become somewhat apparent to me that one would never likely use data-types without core-im and vice versa (at least not in a gks world). Maybe someday if we break out data-types for general use across ga4gh it would be good, but that will likely be a much bigger and broader change regardless.

Q2. (this one is a bit involved) Cat-Vrs and VA-Spec have profiles subfolders under there schema folders. While this is a very clear and useful way to see where the profiled classes live it is a bit shortsighted IMO because this will grow and become cumbersome. So we will either create multiple *-source.yaml files under profiles which is different than what we do in every other schema folder or we should consider creating schema subfolders named after a give profile or set of profiles that functionally belong together. For example, in cat-vrs we may have profiles based on variant type "allele" or "genotype" or "copynumber" profiles (or something like that). For va-spec, we already could break it up into assay-var-effect, cohort-allele-freq and var-path to start to allow ourselves a way to grow organically without having to reshuffle the deck later. Thoughts?

[mbrush]

A few quick thoughts here Larry . . .

1. I don’t have strong feelings about your proposal to group data types into the core-im file. Or to rename/restructure the domain entity file/folder. But I would like to talk through this on a leads call before making any final changes.

2. I think we will want to think about and clearly specify how simply subtyping a class is or is not different form 'profiling' - in particular as it relates to specific domain entity classes being 'profiles' in the same sense as something like the Variant Pathogenicity statement Profile. In my mind these are not really the same thing, but to be fair I haven't fully fleshed this out in my mind.

3. Re: grouping standard profiles defined by MAVE into a separate folder because they represent a functional unit as used by a particular implementor - I have concerns here because all standard profiles in the va-spec repo are meant to be generally useful models that can be used independently, or in combination with standard profiles for any other type of Statement/Study Result. Yes, MAVE uses the three profiles they helped define together, but others may not want to use them together. I don’t think we should force them to be used together by bundling them in a single file, or organizing them in a single repo, just because one implementer uses them together. That said, I may not be entirely understanding your proposal, so please set me straight if this is the case.

[larrybabb]

Here is the final 1.0 Trial Use decision that has been made in regards to schema folder organization...

gks-core
    schema
        gks-core
            gks-core-im-source.yaml 
vrs
    schema
        vrs
            vrs-source.yaml

cat-vrs
    schema
        cat-vrs
            cat-vrs-core-source.yaml
            recipes-source.yaml
va-spec
    schema
        base
            va-spec-source.yaml
            domain-entities-source.yaml
            cohort-allele-frequency-study-result-profile-source.yaml
            experimental-functional-impact-study-result-profile-source.yaml
        #-- community guideline profile folders
        aac-2017
            diagnostic-study-statement-profile-source.yaml
            prognostic-study-statement-profile-source.yaml
            therapeutic-response-study-statement-profile-source.yaml
        acmg-2015
            pathogenicity-functional-impact-evidence-line-profile-source.yaml
            pathogenicity-statement-profile-source.yaml
        ccv-2022
            oncogenicity-functional-impact-evidence-line-profile-source.yaml
            oncogenicity-study-statement-profile-source.yaml

This approach was chosen primarily for the consistency in depth and contents of each repos schema folders. This makes the tooling across all repos easier to apply and maintain. As you get into the higher-order dependencies in va-spec the need for additional sibling folders within the schema folder to support groupings of profiles and classes for community standards. Implementation specific representations beyond va-spec would be applied in separate repos managed and maintained by the implementers themselves.

We will be recommending guidance for implementers to easily extend this approach when we have the time and experience to do so.