"Evidence based decision process" for Releasing new functionalities

[dorin100]

Context

With the latest hard forks (Alonzo and Babbage), the Cardano blockchain became a "mission critical system" with many businesses operating on top of it (stake pools, dApps, exchanges, etc). In this context, the Cardano Node releases we are doing have a direct impact on the value of these businesses and also in the Cardano adoption.

Working on such a critical system, our release decision is extremely sensitive and we need to make sure that we (i) don't affect the existing users & businesses and that (ii) we are increasing the flow of value.

In order to decrease the friction during the Quality Control phase (on the Validation & Verification operations) but also in order to speed up the releases, it would be useful to have a "evidence-based decision process" for developing, testing, and releasing new functionalities.

During the Lisbon meetup, together with Jordan, Vitor, Carlos, and others we defined and adopted a user-facing-functionality template to be used between the Cardano System Test and CLI/API teams. It includes a generic Definition of Done (with all the steps that need to happen during the SDLC) but also the Acceptance Criteria - this way the team (dev, product, system test, project, delivery) is all the time in sync and the probability of delivering good quality functionalities is increased (the whole teams works together and shares the responsibility of the deliverables).

On the Cardano System Test team, we are tracking Quality Control activities in a single place in order to increase clarity and visibility on the required actions versus the remaining ones. Here is an example for tracking the quality control & release activities for a feature delivered together with the Plutus Core squad.

Problem

Now, in the context of other teams developing user-facing functionalities (like UTXO-HD, P2P, new Tracing System, etc), I feel it would be useful to have a similar process defined and used for all user-facing functionalities at Cardano (Core) level.

To start this discussion, my proposal is to create a checklist with all the activities & owners/responsible for (i) Quality Control and (ii) Release decisions.

Proposal

The checklist could be a generic Definition of Done

and the Quality Control activities could be tracked through a decision table like the below one:

Requirements	Owner	Status	Link to Evidence
unit tests	owner	in progress/blocked/done/not required	link to test report
property tests	owner	in progress/blocked/done/not required	link to test report
integration tests	owner	in progress/blocked/done/not required	link to test report
other component tests	owner	in progress/blocked/done/not required	link to test report
security audit	owner	in progress/blocked/done/not required	link to test report
legal audit	owner	in progress/blocked/done/not required	link to test report
performance	owner	in progress/blocked/done/not required	link to test report
system tests	owner	in progress/blocked/done/not required	link to test report
uat (user acceptance test)	owner	in progress/blocked/done/not required	link to test report

LE:

• there might be a difference between the evidence expected at different levels but in order to make the expectations clear for anybody, it would be valuable to have it defined together.
• on the top level (release) it might be useful to define the most comprehensive list of requirements, that would cover the most complex functionalities, and in the case of less complex functionalities mark the unnecessary steps as "Not Required".

[michaelpj]

Happy to have this discussion here, but I think this is probably:

1. A bit too Cardano Node focussed, and
2. A bit too prescriptive
   for it to fit into the CEH as it stands. I think probably it belongs in a cardano-node release process doc for now.

[michaelpj]

I just mean that if e.g. mithril signs up to the CEH they may well want to do something quite different. So this is really about the Cardano Node and its dependencies. Which is fine and good!

[dorin100]

Let's see; hopefully, we are mature enough to generate a procedure of evidence to be used in the Quality Control and Release phases, generic enough to be used in any Cardano/IOG project.

[dorin100]

(the proposal that I made might be generic enough already)

[ghost]

While I agree the proposal might and should be generic enough, I would like to remind that the CEH is meant as guidelines and policies for All open-source projects within the Cardano ecosystem and takes the stance of being as much as possible independent of IOG. In the (near) future, Cardano core will be run under the auspices of a so-called MBO, with development effort being paid for by various companies and individuals and not solely by IOG.
It's unclear to me at this point what would be the role of IOG's QA team, given that the whole releasing process of the node will lay outside the hands of IOG.

[dorin100]

It's unclear to me at this point what would be the role of IOG's QA team, given that the whole releasing process of the node will lay outside the hands of IOG.

Same here, this is why I am not looking at the role of a specific team but at the list of evidence that would be expected/required at different levels. If we get this, it would not matter who provides/review the evidence.

[dorin100]

There were some concerns raised about a link from Confluence.

I don't have any other better place for it right now. I added it in the context of this discussion, in order to help the IOG audience (that has access to Confluence); In order to not affect the history, I will remove it after we will have a decision on this discussion.

[ghost]

This is really not helping much external people (eg. from CF) understand the details of the proposal and discussion. Could you please move the content of those pages to something like a gist or hackmd.io?

[dorin100]

Ok. I moved the DoD in a gist and I updated the link in the main message. Thank you for the suggestion @abailly-iohk

[dorin100]

Now, that we have this fixed, hopefully, we can move to the next steps like defining what/if we need at the component/team/squad level and what at the product level. Plus define the decision tables/checklists (with all the required/expected evidence) and what could (not) be considered evidence.

[dorin100]

I am looking for agreement at least from:

• @dnadales
• @coot
• @kosyrevSerge
• @JaredCorduan
• @disassembler
• @abailly-iohk

Optional:

• @michaelpj
• @zliu41
• @danielmain
• @HeinrichApfelmus

[JaredCorduan]

@dorin100 I think better visibility is great, but I do have some questions. I'll mostly be wearing my "ledger team hat", just so that I can make it concrete, but of course I love that this is a team effort!

• unit tests - ledger always has many of these, we could certainly point to them. of course, there are many reports from CI, often multiple a day, so it's not clear to me what exactly to link to. How many unit tests should there be? Of course this is a ridiculous question, but I ask just to point out that checking this box isn't actually that meaningful. What if instead we just linked to the tests themselves?
• property tests - all the issues above for unit tests apply here too. in fact, if I had infinite time, I would probably like to find a way to turn all the unit tests into property tests. I'm not sure we need to separate these two.
• other component tests - I'm not entirely sure what you mean, could you give an example of who/what was done for Babbage on this? which team is responsible for these?
• security audit - we would need to talk with our auditors to see how to publish it, but this would be excellent.
• legal audit - is this optional? how many of these have we done in the past?
• performance - yep, making @kosyrevSerge 's test publicly available seems awesome to me.
• system tests - would these be pretty much the tests Martin writes?
• uat - do we have examples of these? who would be responsible for these?

[dorin100]

@JaredCorduan thank you for taking the time to look into this. Let's look first more into the scope of this discussion - that is more on defining together a set of checks/evidence that needs to take place before considering a new functionality ready to be released.

I will try to respond to your questions/concerns in the above context(scope):

• unit tests - the scope of the checklist (DoD) is to make sure there are such tests and there is a responsible/owner for them (in your case that would be probably you and the Ledger team). How many of them? Good question - you as the owner of the task would know when you have enough (that you have the assurance that the risk is low) - can we say that No tests are not acceptable!?. Ideally, we (as a group, but also "they" as business owners on Cardano) would be interested in some evidence/assurance that everything is fine - this might be done through links to tests (or even to a short explanation on what/when is run and what is covered). This is meaningful if you think from the Project/Product perspective; Some teams might not be as rigorous as others.
• property tests - the same as above; I think it would be in everybody's interest if there would be a very short summary explaining the context of the assurance/evidence (I think phrases like "it is on our CI" are not so helpful in this context).
• other component tests - (optional) this depends on the feature; in some cases, there might be multiple teams testing different parts of functionality (a good example is the secp)
• security audit - this line is to help us get the awareness that there might be (or not) a security audit required for the new functionality; This line can have a link to the security audit results or "Not Necessary" --> that way the owner of the new functionality will be reminded to ask him(her)self if such an audit is required and ask for it or mark it as not required; Also when it is not required, the information is useful in order to give visibility to anybody else that this was not required
• other audit types - same as above (it is at least a reminder)
• performance - this is not about making the tools or the results public yet but about making sure we took this step into consideration during development/release and when this step was required and executed, there is a report with the results linked to this exact line
• system tests - same as above; it helps the team decide what needs to be automated at this step and it will provide evidence of the work being done (and results as expected) (yes, these are the tests from cardano-node-tests in the context of the Cardano Core; they might be different tests in the bigger context of Cardano --> the idea is to demonstrate (provide evidence) that the new functionality is covered by automated tests at different levels (this is valid for any component/new-functionality --> for example, we already agreed with the Plutus Core team that for the new user-facing-functionalities they will have from now on, they will also be responsible for writing system tests in cardano-node-test)
• uat - (optional) for some functionalities it might make sense to not release it before we involve the community in the test and decision process (like we did with PlutusV2 or Secp, or as we plan to do with P2P); we can decide if this is useful and based on that decide/find a responsible;

So, to conclude, the scope of this discussion is to define together what is the list of steps/checks/evidence that a new (user-facing) functionality should pass in order to be releasable through the release process.

[JaredCorduan]

That all makes a lot of sense to me, thank you @dorin100 . Knowing that much of it is optional helps clarify a lot. I love the idea of the public trail.

can we say that No tests are not acceptable!?.

absolutely. One test is also not acceptable. But there is a sorites paradox lurking here, and I think it could perhaps sometimes make things look a lot more factual and "scientific" than they actually are. Checking that box isn't actually all that meaningful, though assigning responsibility is (and maybe that's all the checkbox is for). 🤔

[dorin100]

Checking that box isn't actually all that meaningful, though assigning responsibility is (and maybe that's all the checkbox is for).

100% - the checkbox is for increasing ownership, visibility, responsibility, and accountability.

[disassembler]

I think for this to be in the handbook, we need to make this super generic. That's difficult because each tribe is going to have a different model. Ignoring what I just said for the moment, I'm going to analyze this from the perspective of Cardano Node, I think this would be something like Squad leads coming together to align on missing gaps. For example, the following questions would need to be answered:

1. Do we have enough unit/property test coverage and where are the results of these tests (e.g. for p2p release, link to CI of ouroboros-network)?
2. Do we need additional integration tests for this feature? And if so, should they occur within the node test suite or externally in cardano-node-tests?
3. Is there a need a for an external or internal security audit for the features being released (much more likely necessary at hard forks for example)?
4. What is the acceptable performance degradation for this release? (maybe it makes more sense to make this decision after we know if there is a degradation and analyze if we should accept the degradation or not rather than stating up front what we expect is allowable)
5. What are we asking the community to test for this release? (e.g. in the case of p2p would be to upgrade preview nodes to 1.35.5 and remove development network features from their configuration file since p2p network protocols are default enabled now)

If this can be made generic enough to apply to all Cardano tribes within IOG, I'd be fine having it in the handbook, but otherwise it probably makes more sense to be more local to the tribe.

[dorin100]

Exactly what I was looking for @disassembler !

Your example is very real and useful; if you ignore the name of the teams, it would be generic enough to be applied at least in Cardano (and Plutus from the discussions I had with them). Others may evaluate it later based on our experience too.

[dorin100]

@coot what do you think about this? Is it something we can apply for the upcoming P2P functionality?

[coot]

I agree with others that we should try it first in the cardano-node, then find how it needs to be improved, then we will be able to distil the essence for CEH.

The notion of coverage & completeness of tests is a bit vague and we should be careful when describing any metrics publicly to not claim too much. Even with very large coverage, there might be some untested interaction which will result in a bug, we had a few of these. That's something general enough to be described by CEH.

Currently the split between unit / property / e2e tests would be difficult. At least GitHub does not provide links to parts of its CI tests, although Hydra does. Still I prefer the former over the latter as it's cleaner (although there are some possible workarounds). I would not distinguish between unit tests & property tests at all, they have the same function just use a different tactic to achieve the goal. A custom report, which has been discussed elsewhere, might be better but this will take time to develop.

Another question we should cover is what architectures have official support, and how these are tested. This varies between teams: the networking team runs its tests on Windows, Linux & MacOS; other teams run their tests on Linux, MacOS and on Wine rather than Windows. Currently we don't test the cross compiled binaries distributed to Daedalus users on Windows, they are only tested on Wine. We also don't have an official support for Aarch64 (RPI4) for example, although some of us run nodes in such environments. I think this information ought to be included in some form.

Some of the audits are not publishable. This has to be agreed upfront with the auditor; e.g. Galois code review was not publishable, although we got a green light on tweeting about its results.

There's a question where do we publish this report?
For sure it would be nice to include this in the release notes, but there are also other cases where parts of such a report will be useful. For example if a release of a component is delayed because of lets say performance regression discovered in the performance cluster, how should we go about it?
My preference would be to include a report why a given feature is still behind a flag in the release notes, but only if that feature was already integrated with cardano-node. Each team should also communicate such problems in biweekly reports with a suitable link.

Who is the audience for such a report?
There are different audiences. There are technical people, SPOs, 3rd party companies, non technical people. We should be explicit in what audience we target. It seems to be that we are thinking about SPOs, 3rd party companies & possible contributors.

What such report should contain?
If we just stash links to CI this will not be that useful. This is easily accessible now, and anyway since we use CI publishing report that test passes is not that useful in my opinion. This is for two reasons:

• the tests are run automatically (they must pass for a PR to be merged),
• the audience is technical, usually knows how to navigate through GitHub.

CEH should suggest the same for all its projects, and it should also cover what does it mean to have good testing for a project that is part the MBO. Perhaps it should suggest a good organisation of tests, splitting them in different components (as we did for protocol-tests here).

It's another matter for the other things included in the suggested table, e.g. the performance review
is vital for SPOs and also us to drive decision whether a feature can be published or not. The same with system tests.

Is it something we can apply for the upcoming P2P functionality?

The P2P is a normal release and as such we are going to apply our (changing) release process.

[zliu41]

My 2c: an overarching guideline in the CEH could be very helpful. I'm not sure, though, whether you are suggesting that we set up a decision table for each package (which means Plutus has its own table for each feature in each release, same with ledger etc.), or a decision table per node release candidate (which is what your example is).

In the former case, would it be better to let each squad decide the process for themselves, and document it as part of their release process? I'd imagine the ideal process could vary from package to package (e.g., I wouldn't imagine using the same process for Plutus Core and Plutus Tools - some Plutus Tools packages are far less mission critical compared to the node itself).

In the latter case, this should belong to the node release process rather than the CEH. This would be my preference because it's nice and simple - as soon as everything is checked in the table, we are ready to tag the rc as the official release!

[dorin100]

First of all, imo, we want to avoid getting the release decision based only on limited evidence. Ideally, we should provide and scrutinize the evidence internally and next ask the community to do the same before making an important change to the existing tools - Cardano Node, DB-Sync, Wallet, Plutus, etc.

We are doing the development from bottom to top (from the component level to the final functionality to be provided to end users). When looking at the system from the perspective of the value it creates, this will always be from the end user's perspective.

In the context of a mission-critical system, like Cardano is, I am initially looking at defining together an evidence-based decision process at least for the final step that would impact the real users - and that probably is the release step. Firstly, this is the Cardano Node release but the same is true for the downstream components too - DB-Sync, Wallet, Rosseta, (even Plutus) etc that are used by users to develop businesses (exchanges, dApps, wallets, etc) and increase the flow of value.

In the context of finding possible issues as soon as possible, the same process can also be cascaded down (shift left) at a single component level (where the team works on some functionalities and those would need to be demoed with evidence inside the team - like the constructive discussions we have during the code reviews but 1 step higher).

We can look at the decisions table as a tool we can use to:

1. define all the required evidence to release a new functionality (Node, DB-Sync, Plutus, etc but also, in the context of ChaP - this could be Ledger, Consensus, Plutus Core, Plutus Tools, etc)
2. based on the required evidence, find/define/identify clear owners for each step/piece of evidence
3. for each team (through the squad leads) to provide the evidence from squad's perspective about their deliverables
4. trigger constructive discussions about possible gaps (required vs provided evidence)
5. define clear owners for each piece of evidence
6. provide visibility and transparency on our decision processes to anybody interested (users, business owners, etc)

In conclusion, I am looking at defining together a generic evidence-based decision process for releasing new functionalities and responding, also together, to questions like:

• How such a process could look like?
  • my initial proposal is to use a generic Definition of Done and a decision table per release (at the component and at package level)
• What would be the required evidence?
  • we need to define this together at the component and package level
• What could be considered evidence?
  • we need to define this together at the component and package level
  • IMO the below could not be considered evidence:
    • responses like "the evidence is in our code (or repo, or CI, etc)"
    • no unit/property/integration/system/e2e tests
    • lack of a required evidence decision table
• Who would be the owner/accountable for the evidence?
  • we need to define this together at the component and package level

I am not looking yet at where these guidelines should be stored but to make sure we are putting together all the required steps in order for our teams to take the best possible decision. I suggest looking later together if it would make more sense to have 1 guideline per package or to have 1 generic one - or even have both.

[michaelpj]

I am looking at defining together a generic evidence-based decision process for releasing new functionalities

I think I mostly disagree on tactics. I think a process change like this is much more likely to work if we demonstrate it working effectively on a single component, and then grow its usage from there. Hence why I keep suggesting that we focus on the release process for the node; make it very clear, make it work, and then export the working process to other components.

[dorin100]

@michaelpj IMO, notions like "what could (not) be considered evidence", "who should provide (what) evidence at different levels", or "what would be the required evidence at different levels (pr/team/squad/package/product/etc)?" could be quite generic (using more input from more people will help us make sure we don't miss anything important).

[michaelpj]

I'm not saying they're not generic, I'm saying I think it would be more effective to start with a specific instance of them and generalize when/if that process is working.

[dorin100]

We definitely can do that but in the meantime, in order to use it anywhere we need to define it and this is why I tagged some people from the Cardano ecosystem in this discussion. As soon as we agree on a process we can start using it --> I was looking to apply it asap and the next Node release could be a very good point (if we are able to define the process together in the meantime).

[ghost]

Discussion linked to #16

[ghost]

My 2 cts:

• Having a more or less complete list of sources of evidence to draw from is great and I support adding that to the CEH as a generic policy to setup for any project, something stating that each "delivered increment of work" should provide a list of evidences, possibly drawn from @dorin100's list, of the "quality" of the work
• Having checklists is great and it is well-known that checklists are essential to any mission critical process. I support the idea of adding to the CEH a section stating that each project should define checklist(s) for each issue/feature/work item possibly drawing items from Dorin's list plus some more (check for example what's done in Mithril:
• From my experience, waiting for the "release" to add (and check) those trail of evidences leads to increasingly long release cycles. Instead of checks happening at the end of some trail of work, as a "gate" before release, it would seem much more useful to gather evidences during the work, eg. as new tests and new code are written.

[dorin100]

From my experience, waiting for the "release" to add (and check) those trail of evidences leads to increasingly long release cycles.

100%. There are at least 2 ways of defining these checklists (and decision tables) - Top-Down or Bottom-Up. I was looking initially to describe the complete list of required evidence and owners at the Top level (release - using the Top-Down model), and then cascading (shifting left) it down, toghether, to the team/squad/component level.

Considering the criticality of the system, I think it is valuable to gather and provide/create evidence on all levels.