Lazy loading of Clients

[elenaf9]

• Feature Name: snapshot_state
• Start Date: 2021-12-06

Summary

In this RFC we propose a way of dynamically loading and writing client states from / to snapshot files in a lazy manner. We discuss how we can reduce the amount of locked memory and keep the states of unused clients encrypted at rest. Finally, we propose a mechanism for storing snapshot keys so that snapshot files can be written without requiring the end-user to re-enter their password.

Motivation

Terminology:

• Snapshot File: Compressed and symmetrically encrypted file in which the vaults and store of all clients of a Stronghold are written.
• Snapshot State: Collection of all client states that are persisted in a snapshot file.
  • HashMap<ClientId, (HashMap<VaultId, PKey<Provider>>, DbView<Provider>, Store)>
• (Snapshot-)Key: [u8;32] bytestring that is used for symmetric encryption of the Snapshot File
• Client: Owner of a collection of secret and non-secret data stored in Stronghold.
• Client State: Collection of vaults with records, a keystore for storing vault-keys, and a non-secret store.
  • (DbView<Provider>, KeyStore, Store)
• Vault-Key: Key for a vault that is used for symmetric encryption of all records in this vault. Protected in memory with the libsodium memory API.

When discussing the dynamic loading and persisting of client states during runtime, it inevitably creates a conflict of convenience vs security.
On one hand it is essential that the stored secrets are always protected and any risk of exposing them is reduced to a minimum. Only the state of the currently used clients should be loaded at a point in time, any unauthorized access should be prevented, and as soon as a client is stopped its data should be persisted and cleared from memory. In addition, it should be possible to frequently backup a clients state during runtime even while the client is active, to be able to recover it in case of an unexpected error.
On the other hand, loading and persisting a client's state from / to a snapshot file always requires the snapshot key, which poses a large inconvenience if that would force the end-user to enter their password every single time. Depending on the volume and velocity of the stored data and used clients, it may happen ever other second. Therefore a policy is required that allows, after the user has initially authorized, to load and persist client states without an externally entered key. Furthermore, the operations on the snapshot file should should be reduced, to avoid the tear and wear on persistent memory drives.
From this, we can derive the following requirements:

• Access to any secrets stored in a snapshot requires that the user has proven their authority
• Loading a client's state should not need the snapshot key each time, if the user previously has proven their authority
• Backing up a client's state should not need the snapshot key each time, if the user previously has proven their authority
• A snapshot key is never exposed in memory
• A snapshot key is not written into a snapshot file
• Lazy loading of clients: Only load clients when they are needed
• Client states are cleared from memory when they are not needed anymore
• States of stopped clients should be persisted even if the snapshot is not written immediately
• Client states from a read snapshot should be accessible at a later time even if they are no immediately loaded

Guide-level explanation

Read snapshot and load clients during runtime

For reading a snapshot, the requirement is kept that the user has to provide the key. Reading the snapshot file populates the snapshot struct in the background with the snapshot state, and optionally already loads the states of some clients. The interface then provides a method for individually loading states into clients, without requiring a key. If the client does not exist yet it is spawned.

impl Stronghold {
  /// Load a snapshot state.
  pub async fn read_snapshot<T: Zeroize + AsRef<Vec<u8>>>(
    &mut self,
    keydata: T,
    filename: Option<String>,
    path: Option<PathBuf>,
    write_key: Option<Vec<u8>>, // optionally write the key into a dedicated vault to the specified record-path
    load_clients: Vec<ClientId> // list of clients that should be loaded directly
  );
  /// Load state from snapshot into the client. Spawn the client if it does not exist yet.
  pub async fn load_client(&mut self, client_path: Vec<u8>);
}

If a snapshot state is already present and a new snapshot is loaded, the old state is dropped. Analogously for the clients an old client state is dropped if a new state is loaded from the snapshot to the client.
Note: If a snapshot state is dropped because a new state is loaded, the client states are not automatically dropped as well. It is the user's responsibility to clear those states and/ or load the new state into each active client.

Persist clients during runtime and write snapshot files

During runtime, client states can be written to the state of the snapshot in memory, which allows to kill clients without the need to write a snapshot file.
Note: Persisting a state in the snapshot only stores it (encrypted) in memory, it does not automatically write it to the file.
Writing the snapshot state then to file can be done either directly with a key, or by entering the record-path that was used in read_snapshot to store the key. Optionally, only selected clients can be written to the file instead of all, which additionally provides a way of exporting clients to a file that could be shared with other Strongholds.

impl Stronghold {
  /// Write state of all clients to snapshot. This automatically persists the state of active clients in the snapshot state.
  /// Optionally only selected clients can be persisted and written to file.
  pub async fn write_snapshot<T: Zeroize + AsRef<Vec<u8>>>(
    &mut self,
    key: T,
    filename: Option<String>,
    path: Option<PathBuf>,
    select_clients: Option<Vec<ClientId>>
  );
  
  /// [`Stronghold::write_snapshot`] with a key stored at the specified record-path.
  pub async fn write_snapshot_with_stored_key(
    &mut self,
    key_path: Vec<u8>,
    filename: Option<String>,
    path: Option<PathBuf>,
    select_clients: Option<Vec<ClientId>>
  );
  /// Write the state of a client to the snapshot state, optionally drop the underlying client struct.
  pub async fn persist_client(&mut self, client_path: Vec<u8>, kill: bool);
}

Clear states

The user may clear the states of clients or the snapshot in memory.
If the state of a client is cleared, all changes since the last Stronghold::persist_client are lost.
If the state of the snapshot is cleared, it removes all client states from the snapshot state in memory. It does not clear any state that was loaded into a client. Hence, the effect is that the states of all non-loaded clients are dropped.
Optionally, only selected client states may be dropped.

impl Stronghold {
  /// Clear a client's state and optionally drop the whole client from memory, without persisting it in the snapshot.
  /// **Note:** this does not influence any state from this client that was already persisted in the snapshot state beforehand.
  pub async fn kill_client(&mut self, client_path: Vec<u8>, kill: bool);
  /// Drop the state of all/ selected clients from the snapshot state.
  /// This does not clear any states in active client.
  pub async fn clear_snapshot_state(&mut self, clients_paths: Option<Vec<u8>>);
}

Reference-level explanation

The fundamental restriction is that reading a snapshot file always requires a key from the user, hence there is no way of loading a state into memory without knowledge of the key. Any following loading of data always relies on the state being present in the snapshot in memory, which enforced this authorization throughout the whole session.
After decrypting a snapshot state Hashmap<ClientId,ClientState> from a file, it is stored in the snapshot struct by serializing, compressing and encrypting the state of each client separately with an ephemeral key that is newly generated: HashMap<ClientId, EncryptedClientState = Vec<u8>>. By encrypting the client state again, it reduces the risk from having each Vault-Key in (protected) memory, to just a single, ephemeral key per client. Futhermore, compared to just re-encrypting the whole snapshot state again, the separate encryption of each client's state eases the dynamic loading and persisting of individual clients during runtime.
The snapshot type owns a keystore that contains the ephemeral keys for each client, protected with the libsodium memory protection.
When the user loads a client, its encrypted client state is cloned from this hashmap, decrypted with the ephemeral key, and loaded into the client types. At the same time the encrypted copy is kept in the snapshot in memory, to prevent data loss in case that clients would unexpected panic. Whether or not a single client panicking is possible depends on the underlying concurrency architecture of Stronghold. See RFC/0001 transactional memory for more info.
When a client type is stopped or manually persisted, its state is stored in the snapshot by re-encrypting it with a new ephemeral key and (re-)inserting it to the hashmap. The old state is overwritten by this, and the key in the keystore is updated with the new key.
The user may load the client state again at any later point.
If a client is unexpectedly dropped, the state is not automatically persisted. It is the user's responsibility to manage the states and we don't want to introduce any hidden/ "magic" behaviour.
Apart from the keystore, the snapshot type owns a singe vault with a random vault-id, which is used to store snapshot keys. The key for this vault is also stored in the keystore.
If the user provides a path for storing the key when reading a snapshot, the key is inserted to this vault at the specified path.
When writing a snapshot, the user may use a stored key instead of entering a key themselves, in which case the key is extracted from the vault and used to encrypt the snapshot file. This does not remove the key from the vault.
Writing a snapshot copies all/selected clients states from the hashmap, decrypts them, and re-encrypts them with the snapshot key. It then compresses the encrypted data and writes it to the snapshot file.
Neither the keystore, nor the vault of the snapshot struct are ever written to a snapshot file.

struct Snapshot {
  // Keys used to encrypt each client state
  keystore: Keystore,
  // Vault with stored snapshot keys
  key_vault: Vault<Provider>,
  // Loaded snapshot state with each client state separately encrypted.
  state: HashMap<ClientId, (Vec<u8>, Store)>,
}

Drawbacks

1. The snapshot state is kept in memory without any automation to clear it.
   Especially if we also add a feature to load and merge multiple snapshot states, it would be easy to loose track of what states are loaded and which of those are still needed. Even with additional layers of encryption, it is naturally less secure to have states laying around in memory rather than clearing them.
2. The new convenience of writing a snapshot may cause the user to do it more often than necessary.
   Users may write multiple times in a second to the vault and decide that they want to prevent any data loss by directly writing to the snapshot after each operation. This is now very easy without any effort from the user's side, they only need to know the path of the key.
   If the key itself would have to be provided on the other hand, the user would be confronted with fetching the key themselves, and possibly also be more aware of the whole encryption process that is triggered with this operation. Possibly this would encourage them to rethink their need of writing the snapshot.
3. An overhead is introduces when when reading a snapshot file, as first the snapshot state is decrypted, including the locking of memory for all vault keys, before they are separately re-encrypted again.

Rationale and alternatives

Storing the snapshot key

The decision to support a mechanism for managing the snapshot key within Stronghold was made because the only alternative would have been to always require the user to provide it. While this may be convenient for Stronghold as library because it would have handed off the responsibility to the user, it would have paved the way for large security holes. Realistically, it is very likely that an application will need to frequently write a snapshot to backup the state. It can not expect the end-user to enter their password every single time this is done, which means that the snapshot key will need to be stored somewhere. If now e.g. some unwary user would decide to just keep it in a environment variable, any process on the computer could access both, the file stored on the device and the key, and use it to steal all secrets. In that case, all security measures within Stronghold would be completely irrelevant. Stronghold is not only responsible for keeping its internals secure, but also for guiding the user into a secure integration of Stronghold, which includes the key management.

Keeping the state of unused clients in memory

Ideally, we would always only have the secrets in memory that we really need, e.g. only always the current client, and drop the snapshot state after a client is loaded. However, this would mean that every time a clients is spawned the file would have to be re-read, and every time a client is stopped it would have to be re-written. Futhermore, it would create some problems when writing the snapshot, because even though we might not have loaded certain clients, we still want to write the state of all clients back to the file. Therefore on each write, we first would have to read the old state again, then update the active clients, and then write back to the file.
Generally, relying that much on reading/ writing the snapshot would:

1. create a large overhead when loading / stopping a client, because reading / writing a snapshot always involves all clients, even if only a single client is needed
2. increase the risk of errors, as reading and writing a snapshot involves multiple steps of locking / unlocking memory for each vault in each client, encrypt data, compressing it and use I/O, instead of just lazy loading the state out of memory and do a single encryption / decryption
3. require the user's key more often, which is something we should to avoid
4. loose data if the snapshot file was moved after first read. The user might expect that it was fully loaded, when in reality only the state of active clients are in memory. In this situation, we could either write a new snapshot, but then have conflicting files with incomplete states, or not be able to write to a file at all and loose all changes.
   The above issues can be prevented by caching the snapshot state. Furthermore, with Stronghold::clear_snapshot_state the user still has the option to manually clear state from memory if they e.g. only use a single client and do not plan to modify the clients state.

Alternatives for how the snapshot state is cached

1. Store it in the decrypted format, which is a hashmap of all un-encrypted client states (current Solution on Stronghold dev).
Decrypt the client states and keep them in a hashmap. When a client is loaded, their state can be taken from this map without any overhead.

Pro

• no overhead when loading / storing client states

Contra

• duplicate amount of locked memory, as the key for every vault of every client are protected with the GuardedVec type
• more memory occupied
• unnecessary risk of exposing secrets, especially if a client is not active

2. Store it fully encrypted how it is read/ written from/to the snapshot file.
Store a copy of the encrypted bytes so that the file does not need to be re-read again. Loading / writing clients would still require the same steps as writing from the file, apart from the I/O operations.

Pro

• no overhead when reading/ writing snapshot files if the user's password is used as encryption key

Contra

• Large overhead when loading/ stopping a single client: requires to decrypt the whole state, lock memory for every client on deserialization, etc.
• Requires the user key each time a client is loaded
• Minimal gain towards just writing it to the file, which would have the above discussed drawbacks

Different encryption for the snapshot state in snapshot struct vs snapshot file

Instead of writing the already encrypted client states from the snapshot struct to the file, they are all decrypted and deserialized, and then re-serialized and re-encrypted in a batch. While this may create an overhead and unnecessarily locks memory during deserialization and re-serialization, this serves two purposes:

1. preserve backward compatibility towards current implementation, as otherwise it would change the snapshot format
2. the encryption key for each client state does not need to be kept in memory and can be ephemeral on every session

Unresolved questions

Future possibilities

• Allow to load and merged multiple snapshot files with a configurable policy