What should this gh-action do

[epijim]

Primary aim is to show that a gh-action can, on each release of a version (I would assume that most packages use releases, rather than just tracking master/main. We then add this to our open sourced R packages to provide some transperancy on how the installation of the package is on a common system (e.g. rocker:4.0.1 on ubuntu).

Short term I'd be happy to focus on quite a generic document, that simply is an rmd prints the output of:

• sessionInfo(), capabilities() and .libPaths()
• what container the gh-action is using?
• get's some core metadata from the Description like maintainer, description, imports, depends, suggests and version (and this could be crude calls to read.dcf for now
• covr::package_coverage()
• covtracer::test_trace_df()
• rcmdcheck::rcmdcheck()

And then allows the user to easily edit the key validation topics (but we can have some example content in the example template). With headings like those we use internally:

## Validation Plan
## Terms and abbreviations
## User requirements specification
## System description
## Solution archeticture
## Test Plan

And then adding it is super easy. e.g. use just needs to add this code (plus the template).

---
name: R Package Validation report

on:
  release:
    types: [published]

jobs:
  r-pkg-validation:
    runs-on: ubuntu-latest
    container: rocker/verse:4.1.0 # also versions packages installed via install.packages()
    steps:
      - uses: actions/checkout@v2
      - name: Build report
        uses: insightsengineering/r-pkg-validation@main
        with:
          # R package root path, in case your R package is within a subdirectory of the repo
          path: "."
          # Template location
          template_path: ".github/validation_template.rmd"
          # Output type
          output_type: "pdf"

And we get this up quickly (maybe @dinakar29's team can look into it?) - but we make sure not to spend too much time on it, as potentially valtools could be refactored to automate the documentation to test linkage, and when that happens we could refactor the backend to lean more on their setup.

@dgkf - super keen to get your thoughts on this.

[epijim]

@dgkf - I assume we'd want it to not depend on autovalidate, and particularly autovalidateutils`? (at least not when we are at the MVP stage).

[dgkf]

Agreed - covtracer is meant to be the majority of the business logic without organizational opinions attached. I think it's the right tool for the job here.

[cicdguy]

I've added a basic structure to this action: #2

[epijim]

thanks Dinakar!

[dgkf]

The scope looks great to me. I think this is in line with where we want to take validation eventually, so it will be great to use this to champion CI jobs/logs as an alternative to full fledged validation reports - where the only thing that is documented is the opinions on the applicability and definitions of CI failure modes.

Just to put a finer point on how much of our validation requirements would be covered under this umbrella:

Validation Plan

• ☑️ "validation plan as code"
• 🔲 addendum opinions doc characterizing why this approach is sufficient

Terms and abbreviations

• 🔲 not in scope, but trivial compared to the rest

User requirements specification

• ☑️ defined as documented exported objects, eg the package's user contract

System description

• ☑️ Comes from sessionInfo() and supporting descriptive functions, although applicability to where the software is eventually used needs to come from an opinions doc

System architecture

• ☑️ testing architecture as code
• ☑️ lifecycle management covered by git+git service (*Hub, *Lab)
  • ☑️ traceable author via git blame
  • ☑️ traceable reviewer via PR policy if needed (risk based)
• 🔲 separate usage environment from validation environment, so no validation of usage environment needed
• 🔲 applicability of testing architecture in relation to usage architecture from opinions doc

Test Plan

• ☑️ testing plan as code, including:
  • ☑️ Functional Testing - testing of generic R package expectations, could be considered fully covered by R CMD check
  • ☑️ User Requirements Testing - testing of advertised R package behaviors, considering exported documentations as a form of requirements specification.
  • ☑️ Traceability Matrix - handled by covtracer with minimal processing
  • ☑️ Any other coverage-related requirements as a heuristic benchmark for minimal quality

The only lingering pieces are how we can ensure reproducibility of the testing and confidence that the execution of the tests meets our expectations for quality assurance of the installed version of the package when it's used. That's definitely a non-trivial piece, but this will hopefully leap frog us straight past mundane discussions about reproducible documentation of software to focus on reproducible use of software.

[epijim]

Thanks Doug - this looks great. For

The only lingering pieces are how we can ensure reproducibility of the testing and confidence that the execution of the tests meets our expectations for quality assurance of the installed version of the package when it's used. That's definitely a non-trivial piece, but this will hopefully leap frog us straight past mundane discussions about reproducible documentation of software to focus on reproducible use of software.

Totally agree - although I see that as out of scope for 'phase 1'. I'll start another thread for that.

For the opinions doc you mention, is that package specific? Or a general statement? The reason I suggested to give very easy access to the template (or maybe the option to inject an .md) was that I still think ideally each package should have at least a statement on how validation was approached (as an optional parameter). E.g. for something like rtables,I'd agree most of what you needed should be captured in the function level documentation. But when the operations on the data get abstract and full of 'expert judgements', like in a statistical package, I'd still like to see (again as optional) a statement at least on the approach taken to ensure what's implemented (e.g. 'we worked with academic x, or colleagues x and y to review decisions made and defaults taken. As well as statements if they did something like submitted the package for review - either to colleagues, as a paper, or maybe if the ROpenSci Statistical Software Review process).

[dgkf]

By opinions, I'm talking primarily about general stances on the process - the exact criteria that qualify or disqualify a package as validated. They might include:

• What characteristics of a testing environment must be consistent with the usage environment for validated use
• What constitutes an acceptable requirement (is existing R package documentation acceptable?)
• Is a coverage threshold an important criteria for exclusion?
• What are acceptable remediation steps if a package fails?
• If we want to include risk measures in the report, how do they influence the validation process?

Things like that - more of the organizational stances than the technical components. This is where the process gets touchy since I'm sure there are a wide variety of opinions that people hold about what is good enough.

[epijim]

In terms of what this gh-action is, vs what it's not, I'd stress I don't see V1 of this gh-action representing validation in terms of a company definition based on their assessment of how to minimise risk and maximise quality. But I do see it as all valid components in helping to define validation:

Phase 1 Hopefully live and in place across Nest, Admiral, and any other packages people want to add this to - but taking information we usually build anyway (e.g. build badges), but moving it into verbose reports aligned under the validation topics they address (what @dgkf's excellent comment dives into).

Phase 2 Formalise it into something that begins to represent a shared definition of validation between companies, including to have conversations about what is needed to make the statement it's validated - then each company individually needs to think about how a shared definition of R package validation intergrates into the company specific SCE validation process which covers a lot more than R packages.

Phase 3 I think this work would flow into the working group kicked off between CRAN, R-Core, R consortium and the FDA, as well as other analogous efforts like Accumulus Synergy.