"Governable" deployment patterns w/ CALM for GenAI - Cross-Project Collaboration Hypothesis

[karlmoll]

Feature Request

Description of Problem:

I am looking to evaluate the potential for CALM to provide code definitions of reference architectures, implement controls, and/or define observability tooling described by Governance documents (e.g. AI Readiness Governance Framework ).

Feedback we have received: "everyone knows what the threats, risks, controls - but not how to implement".

Ideally would augment existing work done by CCC/CFI to map services described in AIR GF Reference Architecture
https://github.com/finos/common-cloud-controls/milestone/6
finos/common-cloud-controls#314

There is also work needed by the AIR SIG to make GF (and regulations) machine readable.

Potential Solutions:

This is a rough draft of the vision so far:

• Make every part of the governance framework machine readable (eventually, also do the same to regulations - similar to LCR)
• Map services/architecture of the AIR GF to controls and compliant infrastructure (partially in progress, AIR needs to provide machine readable architecture, etc)
• CALM takes the reference architecture and builds controls, deployment patterns, and metrics/logging/data observability
• Morphir manages governance/regulation/common business logic "as code" so that applications across the tech stack can be updated on changes
  • I believe morphir can also be used to manage to deployment of controls, architecture of code, etc
  • Morphir provides the ability to monitor/visualize business logic rules being executed in production which provides powerful observability capabilities
• Waltz provides data observability for RAG queries, training data, user queries, etc
• Models fine tuning using proposed open source benchmarks from LLM Exploration
  • Fine tuning efforts based on regulatory interpretation
• There might be additional projects that can be looped in which I don't have as much exposure (certainly OpenRegTech, probably Backstage)
• The next step would be adding new tools for Model Governance

Benefits:

• Solves "we know what the threats/risks/controls are but we don't know how to implement" problem heard during workshop
• Allows seamless, updatable, auditable deployment compliant with governance and regulations (even as they change)
• Give industry a way to demonstrate use cases to regulators and other supervisors/governance concerns where they can showcase use cases on certified architecture/infrastructure/business logic so they can get sign-off before deploying in production