FDC3 Software Conformance Framework #583
Comments
|
From a FINOS standpoint, we'd be supportive to build a conformance program if the FDC3 project maintainers are supportive of this proposal for an open source conformance testing framework - this is in line with what was discussed several times at the Governing Board meeting over the last 2/3 quarters, as a reflection of the growing importance of this standard and to make sure out of the box interoperability for products claiming compatibility with the FDC3 standard. Please let us know! |
|
Hi Colin, The FDC3 maintainers have previously been consulted on support for a conformance framework/compliance testing, and responded positively (and for my part I can confirm that I am strongly in favour). At that time we also highlighted some of the ambiguities in both the API specification and compliance documentation which make the development of such formal testing Of particular interest are two of the remaining issues, which I'm about to start work on resolving:
It is my hope and intention that resolving these issues will make it much easier for both 'platform providers' (those implementing a Desktop Agent) to produce a compliant implementation and for a conformance testing framework to be implemented - hence, I would be happy to involve any team working on conformance testing in the generation or review any PRs associated with these issues. At present, I believe this can be achieved without changing compliance requirements for any existing features, although we may need to revisit some of the accepted 2.0 proposals to confirm if any should be optional. As part of that work, I also intend to raise an issue to add support for feature flags to the N.B. Throughout this response, I am assuming that your proposed work will focus on compliance for 'Platform Providers' (i.e. Desktop Agent implementations) - compliance for Application providers is currently much looser, with the majority of requirements being recommended rather than required (i.e. they are labelled with SHOULD, rather than MUST). There may I look forward to discussing this work further. |
|
With support of @finos/fdc3-maintainers, the work on building a software conformance framework has started (see first iteration). I suggest we keep this issue open (@ColinEberhardt i've assigned to you) until the first milestone is completed and framework is functional. |
kriswest
changed the title
|
Hey @ColinEberhardt @kriswest - given tests are now going to be in a separate repo, should this issue be closed now as well as the related #615 ? |
|
Are they going to stay in a separate repo, or be contributed forwards in the fdc3 repo (toolbox folder) when complete? |
|
FDC3 is already a multi-repo project within FINOS so I was planning on keeping it as a separate repo, unless there's a good reason not to...? |
I thought hosting that's consistent with FDC3 Workbench and FDC3 Explained in the toolbox folder would be good and saves on a separate hosting/deployment setup/subdomain. Also, it might be easier to maintain alongside changes to the standard if it's in the same repo (i.e. you can tweak the API types and the relevant conformance test in the same PR) - once it reaches parity with the standard that is. It'll probably need duplicating for each version to host multiple copies at the same time and should probably only go in when it's accepted as complete for at least the first version anyway - so development on a separate repo has its advantages for the time being. |
|
Hi @kriswest I am not at all opposed to it folding into this repo when the time is right. However, at the moment, when we are looking to iterate on it a bit more rapidly, it made sense to do it in a separate repo. |
|
I might be missing something, but I don't think it makes sense to fold everything in the same repo. While ultimately the FDC3 maintainers should do what they feel best for the FDC3 project itself, I feel strongly the conformance framework should a separate repo because:
And while point 2 can be worked around, I think point 1 is really an important one - yes, I would surely trust @ColinEberhardt not to touch anything beyond his subfolder, but really "trust" is not a great scalable way to enforce ACLs. For an example, we're modeling against, check out the Kubernetes conformance tests in a dedicated repo. If you really feel you want to show everything in the FDC3 repo (and I argue tho that our "consistency" aggregator is the microsite, not the repo), I think you should really consider git submodules, where we can still have separate repos with separate permissions and lifecycles, but pull them together through what effectively are symlinks. In fact I'd argue the toolbox and FDC3 explained should also be submodules - so they could have a life of their own while still being included in the main FDC3 repo as needed. |
|
I might be missing something, but I don't think it makes sense to fold
everything in the same repo. While ultimately the FDC3 maintainers should
do what they feel best for the FDC3 project itself, I feel strongly the
conformance framework should a separate repo because:
1. This is a different set of maintainers ***@***.***
<https://github.com/ColinEberhardt> is maintaining this) - Github doesn't
offer "sub repo" permissioning, so effectively either we add
@ColinEberhardt <https://github.com/ColinEberhardt> as FDC3 maintainer
for the whole repo or he will not have write access.
I think it very odd for a set of conformance tests for a standard to be
under different governance to the standard its testing conformance with.
Ultimately, it needs to be accepted as correct to have standing and that
acceptance should be under the usual review and governance of the standard.
It would also make sense for the wider community to participate in
maintenance.
You don't need write access to raise a PR, only to merge it.
I think using subfolders for this purpose is just wrong and counter to any
open source project I know.
I don't think mono-repos are all that uncommon in opensource projects.
2. The conformance framework will likely follow a different release
lifecycle - ie. it will come AFTER the standard itself is released. Unless
you are suggesting the FDC3 maintainers team plans to update the
conformance framework at every release BEFORE releasing the standard.
Eventually (i.e. its once its caught up-to-date) it would absolutely make
sense to update the conformance tests at the same time as the standard
draft - in fact it would help clarify the exact meaning and intent of
changes and help with implementation of revisions to the standard -
absolutely what we should be aiming for! We do already have tests in the
FDC3 repo for the NPM module code and that is already updated in-sync with
the standard. They currently have similar coverage to the conformance tests
- although the conformance tests need to grow significantly beyond them.
For my experience basing a separate component in a "sub folder" vs a
proper git repo (the atom of collaboration) creates only more
complexity/additional confibguration in build and release, as build tools
generally expect you to be at the based of the repo
Which build tools? Build tools are generally set up to run from a base
folder, with little interaction with the actual repository. For example, in
the FDC3 repo you build the NPM module/linting/typescript type generation
from the root, FDC3 Workbench from <root>\toolbox\fdc3-workbench and the
website from <root>\website. They happily have different linting setups,
dependencies, builds etc.. The website build is set up to rather simply run
the other builds, do some work on the schemas and assemble the whole thing
for hosting on fdc3.finos.org.
And while point 2 can be worked around, I think point 1 is really an
important one - yes, I would surely trust @ColinEberhardt
<https://github.com/ColinEberhardt> not to touch anything beyond his
subfolder, but really "trust" is not a great scalable way to enforce ACLs.
I think you're worrying too much about whether they can merge whenever they
want - and IMHO leaning the wrong way. Like all PRs, from anyone other than
a maintainer, they can fork the repository and work there totally
unhindered, raising a PR to merge the result back into FDC3. This was the
original approach as the PR was raised from
https://github.com/ScottLogic/FDC3. They don't require reviews to make
changes to their fork, only to integrate it into FDC3. A delay in reviewing
does not block anything other than integration into the FINOS FDC3 repo -
or work they specifically need to be advised on how to
complete/interpretation of the FDC3 spec (which is hopefully significantly
helped by the formal spec project and the new compliance info round up on
this PR: #753).
For an example, we're modeling against, check out the Kubernetes
conformance tests <https://github.com/cncf/k8s-conformance> in a
dedicated repo.
That is a very different shaped community with many thousands of
contributors, and Kubernetes itself living in a totally different github
organisation to that conformance framework. What fits them is not likely to
fit us the same way... Also Kubernetes is a software framework rather than
a standard... their conformance framework is perhaps acting more like the
standard and ensuring that the implementation isn't accidentally migrating
off spec in releases. Our situation is different in that we are a standard,
with implementations produced by third parties that need testing for
conformance with it - hence you might argue that keeping the standard and
confirmance testing together is More (not less) like what kubernetes/CNCF
is doing. I'm no expert on that community however and am happy to be
corrected on any of that by someone closer to it.
If you really feel you want to show everything in the FDC3 repo (and I
argue tho that our "consistency" aggregator is the microsite
<https://fdc3.finos.org/>, not the repo), I think you should really
consider git submodules <https://gist.github.com/gitaarik/8735255>, where
we can still have separate repos with separate permissions and lifecycles,
but pull them together through what effectively are symlinks.
The microsite is of course part of the repo and actually integrates teh
builds of the sub-projects. There are a variety of mechanisms available for
us to integrate - but what I'm concerned about is governance. I appreciate
the work you put into helping this community to succeed (and this project
is or will be, I hope, an important part of that). But I also don't think
the community actually knows how this project came about, wasn't on how it
would be implemented or governed, or on changes to that governance. For it
to help, it needs to have standing in the community, needs to be trusted
and needs to be maintained - likely with the help and consent of those
implementing the standard as platform providers. At the bare minimum you
need to involve those who will be tested by the framework and those who
will benefit from the testing (the app providers) in its governance.
I'm not that bothered whether its a subfolder or a submodule (although In
our experience submodules can be an unholy nightmare that always seem to
result in moving to a mono repo) - but I do want to see it closely tied to
the standard and at a particular revision with a formal acceptance process
by the FDC3 community.
Finally, I want to add that I really want this project to succeed and
appreciate the work you, FINOS and ScottLogic have done to bring it about.
For it to have the effect I think we're all looking for I think it needs to
be brought into the core FDC3 and adopted officially at key milestones. I
look forward to the day that it is released on the same day that a standard
version is adopted AND I look forward to contributing to making that happen
;-)
K
…--
Kris West
Principal Engineer
[image: Cosaic]
+1 (800) 821-8147
***@***.***
Cosaic.io: <https://cosaic.io/> ChartIQ <http://cosaic.io/ChartIQ> +
Finsemble <http://cosaic.io/Finsemble>
|
|
@mindthegab @robmoffat I want to second @kriswest here. I think a discussion should happen between FDC3 maintainers and proposed conformance project maintainers to go through the pros and cons and define the relationship better. |
|
Regarding code ownership in a single repo, this is supported by GitHub today: https://docs.github.com/en/repositories/managing-your-repositorys-settings-and-features/customizing-your-repository/about-code-owners |
I thought this already happened (at least with @kriswest and @robmoffat ) and there was an agreement to open a PR to @finos/fdc3-maintainers to bless "releases" of the conformance framework, as @finos/conformance-fdc3-maintainers need approval from @finos/fdc3-maintainers to ensure a certain release of the framework indeed correctly proves compliance to a certain version of FDC3. Basically, regardless of the underlying repo, everyone agrees the authority of blessing the conformance framework functional validity lies with @finos/fdc3-maintainers. But ultimately guys, it's your project and you should decide what's best. I shared my, maybe outdated, views, but please do feel free to organize the project as you prefer. |
|
@mindthegab I haven't had any such discussions. Lets pick up this discussion on the fdc3-maintainers list. |
|
Closing this issue as the first iteration of the conformance framework was implemented, DA implementations tested and badges issued! |
Hello to all FDC3 users and maintainers, I’d like to propose the creation of an FDC3 Software Conformance framework, and am eager to get your feedback.
FDC3 has become a mature and valuable open standard for applications running on the financial desktop. The standard is still under active development, with a number of candidate features identified for version 2.0. As the standard expands in scope, and the number of implementations increase, the consumers of these implementations would benefit from a formal indication of compliance - i.e. that a product, application or service faithfully implements a specific FDC3 version.
Furthermore, the move to a more formal approach to compliance would help resolve any ambiguities in the API specification, or the current outlined approach to compliance.
Our goal is to create an automated test suite that ensures all aspects of the API specification are correctly implemented. This suite may be provided as a web-application, or it may be an automation suite appropriate for use within CI pipelines.
We’ll work with the FINOS / Linux Foundation teams and the FDC3 community to determine how this test suite, plus any necessary associated terms for conformance, are governed. For reference, a very mature example in the Linux Foundation is the Kubernetes conformance test suite, owned by the Community, which backs the CNCF conformance program, owned by the CNCF.
I’m interested in your thoughts and feedback on the above? I’m sure there are technical concerns to consider, and that this group will no doubt have lots of ideas around how these may be addressed (and I’d love to enter into discussion). But in the first instance, can I ask you to respond with an indication whether you are supportive of this concept?
The text was updated successfully, but these errors were encountered: