Merge branch 'main' into f/6588-dataclass-defaults-fix

microsoft · Nov 14, 2024 · 4f095f6 · 4f095f6
2 parents f209e29 + 999fa17
commit 4f095f6
Show file tree

Hide file tree

Showing 55 changed files with 208 additions and 608 deletions.
diff --git a/.github/ISSUE_TEMPLATE/bug_report.md b/.github/ISSUE_TEMPLATE/bug_report.md
@@ -1,8 +1,8 @@
 ---
 name: Bug report
-about: Create a bug report
+about: Report a bug in CCF
 title: ""
-labels: bug
+type: bug
 assignees: ""
 ---
 

diff --git a/.github/ISSUE_TEMPLATE/feature_request.md b/.github/ISSUE_TEMPLATE/feature_request.md
@@ -1,8 +1,8 @@
 ---
 name: Feature request
-about: Suggest an idea for this project
+about: Request that a feature is implemented in CCF
 title: ""
-labels: enhancement
+type: feature
 assignees: ""
 ---
 

diff --git a/.github/ISSUE_TEMPLATE/task.md b/.github/ISSUE_TEMPLATE/task.md
@@ -0,0 +1,18 @@
+---
+name: Project task
+about: Create a task/TODO
+title: ""
+type: task
+assignees: ""
+---
+
+**NOTE**: This type of issue is reserved for the project team. If you are not a member of the team, please use either the bug or feature templates.
+
+**What is the motivation for this task**
+A clear and concise description of the reason why the task must be completed.
+
+**Dependencies**
+A clear and concise description of what this task depends on, and when work on it can start.
+
+**Describe impact if task is not completed**
+A clear and concise description of the consequences if the task is not completed.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -12,6 +12,7 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.
 ### Added
 
 - Added a `ccf::any_cert_auth_policy` (C++), or `any_cert` (JS/TS), implementing TLS client certificate authentication, but without checking for the presence of the certificate in the governance user or member tables. This enables applications wanting to do so to perform user management in application space, using application tables (#6608).
+- Added OpenAPI support for `std::unordered_set`.
 
 ## [6.0.0-dev5]
 

diff --git a/README.md b/README.md
@@ -1,15 +1,16 @@
 # The Confidential Consortium Framework [![CI](https://github.com/microsoft/CCF/actions/workflows/ci.yml/badge.svg)](https://github.com/microsoft/CCF/actions/workflows/ci.yml)
 
-<img alt="ccf" align="right" src="doc/_static/ccf.svg" width="200">
+The Confidential Consortium Framework (CCF) is an open-source framework for building a new category of secure, highly available, and performant applications that focus on multi-party compute and data.</p>
 
-The Confidential Consortium Framework (CCF) is an open-source framework for building a new category of secure, highly available,
-and performant applications that focus on multi-party compute and data.
+<p align="center">
+<img alt="ccf" src="doc/_static/ccf_overview.png" width="700">
+</p>
 
 ## Get Started with CCF
 
-- Read the [CCF overview](https://microsoft.github.io/CCF/main/overview/index.html) and get familiar with [CCF's core concepts](https://microsoft.github.io/CCF/main/overview/what_is_ccf.html)
+- Read the [CCF overview](https://microsoft.github.io/CCF/main/overview/index.html) and becime familiar with the [core concepts](https://microsoft.github.io/CCF/main/overview/what_is_ccf.html)
 - [Install](https://microsoft.github.io/CCF/main/build_apps/install_bin.html) CCF on Linux
-- Get familiar with the CCF core developer API with the [template CCF app](https://github.com/microsoft/ccf-app-template)
+- Become familiar with the CCF core developer API with the [template CCF app](https://github.com/microsoft/ccf-app-template)
 - Quickly build and run [sample CCF apps](https://github.com/microsoft/ccf-app-samples)
 - [Build new CCF applications](https://microsoft.github.io/CCF/main/build_apps/index.html) in TypeScript/JavaScript or C++
 
@@ -23,7 +24,6 @@ and performant applications that focus on multi-party compute and data.
 
 - Browse the [documentation](https://microsoft.github.io/CCF/)
 - Read the [Research Papers](https://microsoft.github.io/CCF/main/research)
-- Learn more about [Azure Confidential Computing](https://azure.microsoft.com/solutions/confidential-compute/) offerings like Azure DC-series (which support Intel SGX TEE) and the [Open Enclave](https://github.com/openenclave/openenclave) SDK
 
 ## Third-party components
 

diff --git a/doc/_static/ccf_overview.png b/doc/_static/ccf_overview.png
diff --git a/doc/architecture/threading.rst b/doc/architecture/threading.rst
@@ -27,7 +27,4 @@ Programming Model
 
 To ensure session consistency all commands that originate from the same connection are executed on the same thread.
 It is strongly advised that during the execution of a command the application does not mutate any global state outside of the key-value store.
-Any inter-command communication must be performed via the key-value store, to ensure that CCF can rollback commands or change the primary as required.
-
-If an application has global state that exists outside the key-value store, CCF offers several concurrency control primitives (via Open Enclave) to protect memory that could be accessed concurrently by multiple threads.
-It is recommended that these primitives are used rather than other primitives, such as mutexes, which may result in an OCALL.
+Any inter-command communication must be performed via the key-value store, to ensure that CCF can rollback commands or change the primary as required.
diff --git a/doc/architecture/tls_internals.rst b/doc/architecture/tls_internals.rst
@@ -4,11 +4,9 @@ TLS Internals
 Overview
 ~~~~~~~~
 
-In CCF, the :term:`TLS` layer is implemented using OpenSSL 1.1.1 from :term:`Open Enclave` (OE). However, the original implementation was using OE's MbedTLS library, which the current implementation replaced. During the transition period, the OpenSSL implementation had to emulate the previous MbedTLS one and the remaining code isn't particularly suited to OpenSSL.
+In CCF, the :term:`TLS` layer is implemented using OpenSSL 1.1.1 or 3.1. However, the original implementation was using OE's MbedTLS library, which the current implementation replaced. During the transition period, the OpenSSL implementation had to emulate the previous MbedTLS one and the remaining code isn't particularly suited to OpenSSL.
 
-In addition, we'll soon be adding :term:`QUIC` support to CCF, which has its own usage of TLS (supporting OpenSSL, but from its own tree).
-
-This document is an attempt to describe how that works, so that we can plan a suited implementation for both TCP-based and QUIC-based endpoints using TLS.
+This document is an attempt to describe how that works, to faciliate further changes.
 
 Enclave Connections
 ~~~~~~~~~~~~~~~~~~~
@@ -63,13 +61,9 @@ The main reasons why we moved to OpenSSL are:
 
 - We already use OpenSSL for our crypto library ('src/crypto').
 - We wanted TLS 1.3 support and MbedTLS doesn't have it.
-- We want to support QUIC, that doesn't work with MbedTLS.
-
-By now we have removed any traces of MbedTLS, but we are still using OE's OpenSSL library, which doesn't have QUIC support.
-
-To implement the last point above, we need to build QUIC with its own OpenSSL and use that as our library for the remaining crypto, but OE has its own modifications to OpenSSL as well, and we can't have both.
+- We wanted to support QUIC, which doesn't work with MbedTLS.
 
-So, for now, the only way to have QUIC support is in 'virtual' mode, retaining OE's OpenSSL for 'sgx' mode. Once we have other types of enclaves that don't require Open Enclave we can use QUIC's version, too.
+By now we have removed any traces of MbedTLS, but we are still using a version of OpenSSL which doesn't have QUIC support.
 
 MbedTLS vs OpenSSL
 ~~~~~~~~~~~~~~~~~~

diff --git a/doc/audit/builtin_maps.rst b/doc/audit/builtin_maps.rst
@@ -122,7 +122,7 @@ Identity, status and attestations (endorsed quotes) of the nodes hosting the net
 ``nodes.code_ids``
 ~~~~~~~~~~~~~~~~~~
 
-Versions of the code allowed to join the current network on :doc:`SGX <../operations/platforms/sgx>`.
+DEPRECATED. Previously contained versions of the code allowed to join the current network on SGX hardware.
 
 **Key** MRENCLAVE, represented as a base64 hex-encoded string (length: 64).
 

diff --git a/doc/audit/python_library.rst b/doc/audit/python_library.rst
@@ -118,23 +118,23 @@ The ``ledger_code.py`` command line utility can be used to display the history o
 .. code-block:: bash
 
     $ ledger_code.py -s /path/to/ledger/dir
-    2.1 Introduced : [('OE_SGX_v1', '3258fb')]
-    2.103 Introduced : [('OE_SGX_v1', '048b8c')]
-    3.175 Removed : [('OE_SGX_v1', '3258fb')]
+    2.1 Introduced : [('AMD_SEV_SNP_v1', '3258fb')]
+    2.103 Introduced : [('AMD_SEV_SNP_v1', '048b8c')]
+    3.175 Removed : [('AMD_SEV_SNP_v1', '3258fb')]
 
     $ ledger_code.py -vs /path/to/ledger/dir
-    2.1: {('OE_SGX_v1', '3258fb'): ['9b013f']}
-    2.3: {('OE_SGX_v1', '3258fb'): ['9b013f', 'beb114']}
-    2.5: {('OE_SGX_v1', '3258fb'): ['9b013f', 'beb114', '6ff2f4']}
-    2.103: {('OE_SGX_v1', '3258fb'): ['9b013f', 'beb114', '6ff2f4'], ('OE_SGX_v1', '048b8c'): ['3ebd58']}
-    2.119: {('OE_SGX_v1', '3258fb'): ['9b013f', 'beb114', '6ff2f4'], ('OE_SGX_v1', '048b8c'): ['3ebd58', '53cb2a']}
-    2.146: {('OE_SGX_v1', '3258fb'): ['9b013f', 'beb114', '6ff2f4'], ('OE_SGX_v1', '048b8c'): ['3ebd58', '53cb2a', '1d1396']}
-    2.152: {('OE_SGX_v1', '3258fb'): ['beb114', '6ff2f4'], ('OE_SGX_v1', '048b8c'): ['3ebd58', '53cb2a', '1d1396']}
-    3.156: {('OE_SGX_v1', '3258fb'): ['beb114', '6ff2f4'], ('OE_SGX_v1', '048b8c'): ['3ebd58', '53cb2a', '1d1396']}
-    3.165: {('OE_SGX_v1', '3258fb'): ['6ff2f4'], ('OE_SGX_v1', '048b8c'): ['3ebd58', '53cb2a', '1d1396']}
-    3.168: {('OE_SGX_v1', '3258fb'): ['6ff2f4'], ('OE_SGX_v1', '048b8c'): ['3ebd58', '53cb2a', '1d1396']}
-    3.175: {('OE_SGX_v1', '048b8c'): ['3ebd58', '53cb2a', '1d1396']}
-    4.179: {('OE_SGX_v1', '048b8c'): ['3ebd58', '53cb2a', '1d1396']}
+    2.1: {('AMD_SEV_SNP_v1', '3258fb'): ['9b013f']}
+    2.3: {('AMD_SEV_SNP_v1', '3258fb'): ['9b013f', 'beb114']}
+    2.5: {('AMD_SEV_SNP_v1', '3258fb'): ['9b013f', 'beb114', '6ff2f4']}
+    2.103: {('AMD_SEV_SNP_v1', '3258fb'): ['9b013f', 'beb114', '6ff2f4'], ('AMD_SEV_SNP_v1', '048b8c'): ['3ebd58']}
+    2.119: {('AMD_SEV_SNP_v1', '3258fb'): ['9b013f', 'beb114', '6ff2f4'], ('AMD_SEV_SNP_v1', '048b8c'): ['3ebd58', '53cb2a']}
+    2.146: {('AMD_SEV_SNP_v1', '3258fb'): ['9b013f', 'beb114', '6ff2f4'], ('AMD_SEV_SNP_v1', '048b8c'): ['3ebd58', '53cb2a', '1d1396']}
+    2.152: {('AMD_SEV_SNP_v1', '3258fb'): ['beb114', '6ff2f4'], ('AMD_SEV_SNP_v1', '048b8c'): ['3ebd58', '53cb2a', '1d1396']}
+    3.156: {('AMD_SEV_SNP_v1', '3258fb'): ['beb114', '6ff2f4'], ('AMD_SEV_SNP_v1', '048b8c'): ['3ebd58', '53cb2a', '1d1396']}
+    3.165: {('AMD_SEV_SNP_v1', '3258fb'): ['6ff2f4'], ('AMD_SEV_SNP_v1', '048b8c'): ['3ebd58', '53cb2a', '1d1396']}
+    3.168: {('AMD_SEV_SNP_v1', '3258fb'): ['6ff2f4'], ('AMD_SEV_SNP_v1', '048b8c'): ['3ebd58', '53cb2a', '1d1396']}
+    3.175: {('AMD_SEV_SNP_v1', '048b8c'): ['3ebd58', '53cb2a', '1d1396']}
+    4.179: {('AMD_SEV_SNP_v1', '048b8c'): ['3ebd58', '53cb2a', '1d1396']}
 
 As with ``read_ledger.py``, non-committed ledger files are ignored, unless the ``--uncommitted`` command line argument is specified.
 

diff --git a/doc/build_apps/install_bin.rst b/doc/build_apps/install_bin.rst
@@ -18,7 +18,7 @@ The dependencies required to build and run CCF apps can be conveniently installe
 
 .. tab:: Virtual
 
-    .. warning:: The `virtual` version of CCF can also be run on hardware that does not support SGX/SNP. Virtual mode does not provide any security guarantees and should be used for development purposes only.
+    .. warning:: The `virtual` version of CCF can also be run on hardware that does not support SEV-SNP. Virtual mode does not provide any security guarantees and should be used for development purposes only.
 
     .. code-block:: bash
 
@@ -34,37 +34,7 @@ Installation from .deb
 
 Alternatively, CCF releases are available on the `GitHub repository release page <https://github.com/microsoft/CCF/releases>`_.
 
-The CCF Debian package (``ccf_<platform>_<version>_amd64.deb``) contains the libraries and utilities to start a CCF service and build CCF applications. CCF can be installed as follows, for the ``SGX``, ``SNP`` and ``Virtual`` platforms:
-
-.. tab:: SGX
-
-    .. code-block:: bash
-
-        # Set CCF_VERSION to most recent LTS release
-        $ export CCF_VERSION=$(curl -ILs -o /dev/null -w %{url_effective} https://github.com/microsoft/CCF/releases/latest | sed 's/^.*ccf-//')
-        # Alternatively, set this manually, e.g.:
-        # export CCF_VERSION=4.0.0
-        $ wget https://github.com/microsoft/CCF/releases/download/ccf-${CCF_VERSION}/ccf_sgx_${CCF_VERSION}_amd64.deb
-        $ sudo apt install ./ccf_sgx_${CCF_VERSION}_amd64.deb
-
-    .. tip:: A separate `unsafe` package (``ccf_sgx_unsafe_<version>_amd64.deb``) with extremely verbose logging is also provided for troubleshooting purposes. Its version always ends in ``unsafe`` to make it easily distinguishable. The extent of the logging in these packages mean that they cannot be relied upon to offer confidentiality and integrity guarantees. They should never be used for production purposes.
-
-    Assuming that CCF was installed under ``/opt``, the following commands can be run to verify that CCF was installed successfully:
-
-    .. code-block:: bash
-
-        $ /opt/ccf_sgx/bin/cchost --version
-        CCF host: ccf-<version>
-        Platform: SGX
-
-        $ /opt/ccf_sgx/bin/sandbox.sh
-        No package/app specified. Defaulting to installed JS logging app
-        Setting up Python environment...
-        Python environment successfully setup
-        [16:10:16.552] Starting 1 CCF node...
-        [16:10:23.349] Started CCF network with the following nodes:
-        [16:10:23.350]   Node [0] = https://127.0.0.1:8000
-        ...
+The CCF Debian package (``ccf_<platform>_<version>_amd64.deb``) contains the libraries and utilities to start a CCF service and build CCF applications. CCF can be installed as follows, for the ``SNP`` and ``Virtual`` platforms:
 
 .. tab:: SNP
 
@@ -142,12 +112,6 @@ Uninstall
 
 To remove an installation of CCF, run:
 
-.. tab:: SGX
-
-    .. code-block:: bash
-
-        $ sudo apt remove ccf_sgx
-
 .. tab:: SNP
 
     .. code-block:: bash
@@ -163,28 +127,4 @@ To remove an installation of CCF, run:
 From Source
 -----------
 
-To build and install CCF from source, please see :doc:`/contribute/build_ccf`.
-
-In Azure
---------
-
-CCF can be installed on an Azure Virtual Machine by running a single script:
-
-.. tab:: SGX
-
-    .. code-block:: bash
-
-        $ /opt/ccf_sgx/getting_started/azure_vm/install_ccf_on_azure_vm.sh
-
-.. tab:: SNP
-
-    .. code-block:: bash
-
-        $ /opt/ccf_snp/getting_started/azure_vm/install_ccf_on_azure_vm.sh
-
-.. tab:: Virtual
-
-    .. code-block:: bash
-
-        $ /opt/ccf_virtual/getting_started/azure_vm/install_ccf_on_azure_vm.sh
-
+To build and install CCF from source, please see :doc:`/contribute/build_ccf`.
diff --git a/doc/build_apps/js_app_bundle.rst b/doc/build_apps/js_app_bundle.rst
@@ -373,7 +373,7 @@ Note that this removes the sandboxing protections described above. If the conten
 
 This behaviour is controlled in ``app.json``, with the ``"interpreter_reuse"`` property on each endpoint. The default behaviour, taken when the field is omitted, is to avoid any interpreter reuse, providing strict sandboxing safety. To reuse an interpreter, set ``"interpreter_reuse"`` to an object of the form ``{"key": "foo"}``, where ``foo`` is an arbitrary, app-defined string. Interpreters will be shared between endpoints where this string matches. For instance:
 
-.. code-block:: json
+.. code-block:: yaml
 
     {
       "endpoints": {

diff --git a/doc/build_apps/logging.rst b/doc/build_apps/logging.rst
@@ -26,9 +26,7 @@ These logging functions do several things:
 - Variable substitution. See `libfmt <https://fmt.dev/latest/>`_ for more details of the formatting syntax used in C++
 - Declare the severity of the entry. CCF defines 5 levels (``trace``, ``debug``, ``info``, ``fail``, and ``fatal``), and production nodes will generally ignore entries below ``info``
 - Prefix formatted metadata. The produced log line will include a timestamp, the name and line number where the line was produced, and an ``[app]`` tag
-- Write without an ECALL. The final write must be handled by the host, so writing directly from the enclave would require an expensive ECALL. Instead these macros will queue writes to a ringbuffer for the host to process, so diagnostic logging should not cause significant performance drops
-
-The most-verbose lines which are emitted may be restricted at build-time, if required for security on that platform. On SGX, the ``CCF_APP_TRACE`` and ``CCF_APP_DEBUG`` macros are disabled at compile-time, to avoid leaking confidential information during execution. To enable these, define ``VERBOSE_LOGGING`` before including ``logger.h`` in your application code. The unsafe build variants of CCF include verbose logging of the framework itself, but this can be used independently of verbose logging in the application code.
+- Queue writes to a ringbuffer for the host to process, so diagnostic logging should not cause significant performance drops
 
 .. note:: The app's logging entries will be interleaved (line-by-line) with the framework's logging messages. Filter for entries containing ``[app]`` to extract only application log lines.
 

diff --git a/doc/build_apps/run_app.rst b/doc/build_apps/run_app.rst
@@ -45,7 +45,7 @@ For example, deploying the generic JS application:
 
 .. note::
 
-    - ``sandbox.sh`` defaults to using CCF's `virtual` mode, which does not require or make use of SGX. To load debug or release enclaves and make use of SGX, ``--enclave-type`` must be set to the right value, for example: ``sandbox.sh --enclave-type release -p ./libjs.enclave.so.signed``
+    - ``sandbox.sh`` automatically uses the platform which either the local build or install of CCF targeted, i.e. either Virtual or SNP.
     - The ``--verbose`` argument can be used to display all commands issued by operators and members to start the network.
 
 The command output shows the addresses of the CCF nodes where commands may be submitted (ie, via ``curl https://127.0.0.1:8000/...``).

diff --git a/doc/contribute/build_ccf.rst b/doc/contribute/build_ccf.rst
@@ -9,25 +9,28 @@ First, checkout the CCF repository:
 
     $ git clone git@github.com:microsoft/CCF.git
 
-To build CCF from source on a SGX-enabled machine, run the following:
+To build CCF from source, run the following:
 
-.. code-block:: bash
+.. tab:: SNP
 
-    $ cd CCF
-    $ mkdir build
-    $ cd build
-    $ cmake -GNinja ..
-    $ ninja
+    .. code-block:: bash
 
-Alternatively, on a non-SGX machine, you can build a `virtual` instance of CCF:
+        $ cd CCF
+        $ mkdir build
+        $ cd build
+        $ cmake -GNinja -DCOMPILE_TARGET=snp .. 
+        $ ninja
 
-.. code-block:: bash
 
-    $ cd CCF
-    $ mkdir build
-    $ cd build
-    $ cmake -GNinja -DCOMPILE_TARGET=virtual ..
-    $ ninja
+.. tab:: Virtual
+
+    .. code-block:: bash
+
+        $ cd CCF
+        $ mkdir build
+        $ cd build
+        $ cmake -GNinja -DCOMPILE_TARGET=virtual ..
+        $ ninja
 
 .. note:::
 
@@ -50,7 +53,7 @@ The most common build switches include:
 
 * **BUILD_TESTS**: Boolean. Build all tests for CCF. Default to ON.
 * **SAN**: Boolean. Build unit tests with Address and Undefined behaviour sanitizers enabled. Default to OFF.
-* **COMPILE_TARGET**: String. Target compilation platform. Defaults to ``sgx``. Supported values are ``sgx``, ``snp``, or ``virtual``.
+* **COMPILE_TARGET**: String. Target compilation platform. Defaults to ``snp``. Supported values are ``snp``, or ``virtual``.
 
 Run Tests
 ---------

diff --git a/doc/contribute/build_images.rst b/doc/contribute/build_images.rst
@@ -6,7 +6,7 @@ They can be pulled by unauthenticated users:
 
 .. code-block:: bash
 
-    $ docker pull ghcr.io/microsoft/ccf/ci/(default|sgx):build-*
+    $ docker pull ghcr.io/microsoft/ccf/ci/default:build-*
 
 Pushing a git tag of the form ``build/*`` will trigger the :ccf_repo:`.github/workflows/ci-containers-ghcr.yml` workflow that builds and pushes a new ``ghcr.io/microsoft/ccf/ci/(default|sgx):build-*`` image.