Merge branch 'master' into master

blog/features/composite-query.md

@@ -0,0 +1,145 @@
+---
+title: Building a partitioned key-value store with composite queries on the Internet Computer
+description: "Today, we’ll dive into composite queries and walk through the process of building a sample app: a partitioned key-value store. Each partition is represented by a single canister. We'll leverage the power of the Internet Computer’s composite queries to efficiently retrieve data from those partition canisters."
+tags: [New features]
+image: /img/blog/dev-update-blog-composite-query.png
+---
+
+# Building a partitioned key-value store with composite queries on the Internet Computer
+
+Welcome to another developer blog post! Today, we’ll dive into composite queries and walk through the process of building a sample dapp: a partitioned key-value store. Each partition is represented by a single canister. We'll leverage the power of the Internet Computer’s composite queries to efficiently retrieve data from those partition canisters.
+
+In essence, the partitioned key-value store is structured as a single frontend with multiple backends. Each backend represents one partition of the key-value store.
+
+![Partitioned key-value store](/img/blog/dev-update-blog-composite-query.png)
+
+## Frontend code
+The frontend code does the following for a put and get call:
+
+ * Determines the ID of the canister that holds the partition with the given key.
+ * Makes a call into the `get` or `put` function of that canister and parsing of the result.
+
+The following code shows a simplified version of the frontend code. Note the line `#[query(composite = true)]` which is used to leverage the new composite query feature:
+
+```rust
+#[query(composite = true)]
+async fn frontend_get(key: u128) -> Option<u128> {
+    let canister_id = get_partition_for_key(key);
+    match call(canister_id, "get", (key, ), ).await {
+        Ok(r) => {
+            let (res,): (Option<u128>,) = r;
+            res
+        },
+        Err(_) => None,
+    }
+}
+```
+
+For completeness, the `put` code cannot benefit from composite query calls, as adding values to the key value store modifies the canister’s state and therefore needs to be implemented as an `update` call.
+
+```rust
+#[update]
+async fn put(key: u128, value: u128) -> Option<u128> {
+    let canister_id = get_partition_for_key(key);
+    match call(canister_id, "put", (key, value), ).await {
+        Ok(r) => {
+            let (res,): (Option<u128>,) = r;
+            res
+        },
+        Err(_) => None,
+    }
+}
+```
+
+## Backend code
+The backend simply stores the key value pairs in a `BTreeMap` in stable memory:
+
+```rust
+#[update]
+fn put(key: u128, value: u128) -> Option<u128> {
+    STORE.with(|store| store.borrow_mut().insert(key, value))
+}
+
+#[query]
+fn get(key: u128) -> Option<u128> {
+    STORE.with(|store| store.borrow().get(&key))
+}
+```
+
+And that’s it!
+
+The complete code can be found [here](https://github.com/dfinity/examples/tree/master/rust/composite_query).
+
+An alternative implementation for Motoko can be found [here](https://github.com/dfinity/examples/tree/master/motoko/composite_query).
+
+## Using composite queries
+To start, let's set up our development environment. Make sure you have [dfx](https://internetcomputer.org/docs/current/developer-docs/setup/install/) installed on your computer. You will need at least version 0.15.0 of dfx for composite query support. Open your terminal and follow these commands:
+
+```bash
+DFX_VERSION=0.15.0-beta.1 sh -ci "$(curl -fsSL https://internetcomputer.org/install.sh)"
+```
+
+Then clone the IC sample apps as follows:
+
+```bash
+git clone https://github.com/dfinity/examples.git
+```
+
+## Deploy the example canister
+We first need to start a local IC instance via dfx and then create and build our frontend canister:
+
+```bash
+cd rust/composite_query/src
+dfx start
+dfx canister create kv_frontend
+dfx build kv_frontend
+```
+
+During compilation of the fronted canister, the backend canister's wasm code will be compiled and inlined in the frontend canister's wasm code.
+Finally, let’s install the frontend canister:
+
+```bash
+dfx canister install kv_frontend
+```
+
+Excellent! We have our partitioned key-value store set up and ready to go. Now, let's explore its capabilities.
+
+## Interacting with the canister
+To add a key-value pair via the frontend canister, run the following command in your terminal:
+
+```bash
+$ dfx canister call kv_frontend put '(1, 1337)'
+(null)
+```
+
+Keep in mind that the first call to put might be slow to respond because the data partition canisters have to be created first.
+Now, let's retrieve the value associated with a key using the power of composite queries:
+
+```bash
+$ dfx canister call kv_frontend get '(1)'
+(opt (42 : nat))
+```
+
+As you can see, we can effortlessly fetch the value using composite queries with very low latency.
+Let’s now compare the performance of composite query calls with those of an equivalent implementation that leverages calls from update functions: for that, we use the `get_update` method, which contains the exact same logic, but is implemented based on update calls:
+
+```bash
+$ dfx canister call kv_frontend get_update '(1)'
+(opt (1_337 : nat))
+```
+
+We can observe that with update calls we receive the very same result, but the call is at least one order of magnitude slower compared to composite query calls.
+Furthermore, we can orchestrate two query calls: first into the frontend canister and then into the data partition canister. This has similar latency as the composite query call, but requires extra logic on the client side.
+```bash
+$ dfx canister call kv_frontend lookup '(1)'
+(1 : nat, "dmalx-m4aaa-aaaaa-qaanq-cai")
+$ dfx canister call dmalx-m4aaa-aaaaa-qaanq-cai get '(1: nat)' --query
+(1_337 : nat)
+```
+
+In summary, by using composite queries, we achieve low latency while keeping the client side simple. This is especially useful for dapps that are scaling vertically by partitioning data across multiple canisters.
+
+Congratulations! You have successfully built a key-value store using Rust and leveraged the powerful composite query feature of the Internet Computer. This allows for efficient retrieval of data from your canisters.
+
+We hope you found this blog post helpful. Happy coding with the composite query feature!
+Many thanks to the DFINITY for contributing to the composite query feature: Adam Spofford, Claudio Russo, Martin Raszyk, Robin Künzler, Roel Storms, Stefan Kaestle, Ulan Degenbaev, Yan Chen

docs/developer-docs/integrations/composite-query/composite-query.md

@@ -0,0 +1,136 @@
+# Composite queries
+## Overview
+The Internet Computer (IC) supports two types of messages: updates and queries. An update message is executed on all nodes and persists canister state changes. A query message discards state changes and typically executes on a single node. It is possible to execute a query message as an update. In such a case, the query still discards the state changes, but the execution happens on all nodes and the result of execution goes through consensus. This “query-as-update” execution mode is also known as replicated query.
+
+An update can call other updates and queries. However a query cannot make any calls, which can hinder development of scalable decentralized applications (dapps), especially those that shard data across multiple canisters.
+
+Composite queries solve this problem. A composite query is a new type of query that you can add to your canister using the following annotations:
+
+ * Candid: `composite_query`
+ * Azle: `$query`; in combination with `async`
+ * Motoko: `composite query`
+ * Rust: `#[query(composite = true)]`
+
+Users and the client-side JavaScript code can invoke a composite query endpoint of a canister using the same query URL as for existing regular queries. In contrast to regular queries, a composite query can call other composite and regular queries. Due to limitations of the current implementation, composite queries have two restrictions:
+
+ * A composite query cannot call canisters on another subnet.
+ * A composite query cannot be executed as an update. As a result, updates cannot call composite queries.
+
+These restrictions will be hopefully lifted in future implementations.
+
+Composite queries are enabled in the following releases:
+
+| Platform / Language        | Version |
+| -------------------------- | ------- |
+| Internet computer mainnet  | Release [7742d96ddd30aa6b607c9d2d4093a7b714f5b25b](https://nns.ic0.app/proposal/?u=qoctq-giaaa-aaaaa-aaaea-cai&proposal=123311)     |
+| Candid                     | [2023-06-30 (Rust 0.9.0)](https://github.com/dfinity/candid/blob/master/Changelog.md#2023-06-30-rust-090)     |
+| Motoko                     | [0.9.4](https://github.com/dfinity/motoko/releases/tag/0.9.4), revision: [2d9902f](https://github.com/dfinity/motoko/commit/2d9902fb75bb04e377c28913c311aa2be373e159)    |
+| Rust                       | [0.6.8](https://github.com/dfinity/cdk-rs/blob/219ae179b9c5ef0ebfff20b926a90f6624ebe704/src/ic-cdk/CHANGELOG.md#068---2022-11-28)    |
+| Azle                       | [0.11.0](https://github.com/demergent-labs/azle/releases/tag/0.11.0)    |
+
+
+## Code example
+As an example, consider a partitioned key-value store, where a single frontend does the following for a `put` and `get` call:
+
+- First, determines the ID of the data partition canister that holds the value with the given key.
+- Then, makes a call into the `get` or `put` function of that canister and parses the result.
+
+Below is a simplified example of the frontend code. Take note of the line `#[query(composite = true)]` which is used to leverage the new composite query feature:
+
+```rust
+#[query(composite = true)]
+async fn frontend_get(key: u128) -> Option<u128> {
+	let canister_id = get_partition_for_key(key);
+	match call(canister_id, "get", (key, ), ).await {
+    	Ok(r) => {
+        	let (res,): (Option<u128>,) = r;
+        	res
+    	},
+    	Err(_) => None,
+	}
+}
+```
+
+The backend simply stores the key value pairs in a `BTreeMap` in stable memory:
+
+```rust
+#[query]
+fn get(key: u128) -> Option<u128> {
+	STORE.with(|store| store.borrow().get(&key))
+}
+```
+
+## Using composite queries
+### Prerequisites
+ - [x] [Download and install the IC SDK.](https://internetcomputer.org/docs/current/developer-docs/setup/install/)
+ - [x] [Download and install git.](https://git-scm.com/downloads)
+
+### Setting up the canisters
+### Step 1: Open a terminal window and clone the DFINITY examples repo with the command:
+
+```bash
+git clone https://github.com/dfinity/examples.git
+```
+
+### Step 2: Navigate into the `rust/composite_query/src` directory, start a local replica, and build the frontend canister with the commands:
+
+```bash
+cd rust/composite_query/src
+dfx start
+dfx canister create kv_frontend
+dfx build kv_frontend
+During compilation of the fronted canister, the backend canister's wasm code will be compiled and inlined in the frontend canister's wasm code.
+
+### Step 3: Then, install the frontend canister with the command:
+dfx canister install kv_frontend
+```
+
+### Interacting with the canisters
+### Step 1: To add a key-value pair via the frontend canister, run the following command:
+
+```bash
+dfx canister call kv_frontend put '(1, 1337)'
+```
+
+:::note
+The first call to put might be slow to respond because the data partition canisters have to be created first.
+:::
+
+The output should resemble the following indicating that no value has previously been registered for this key:
+```(null)```
+
+### Step 2: Retrieve the value associated with a key using composite queries with the command:
+
+```bash
+dfx canister call kv_frontend get '(1)'
+```
+
+The output should resemble the following:
+```(opt (1337 : nat))```
+
+This workflow displays the ability to fetch the value using composite queries with very low latency.
+
+### Comparing composite queries to calls from update functions
+Let’s now compare the performance of composite query calls with those of an equivalent implementation that leverages calls from update functions. To do this, we will use the `get_update` method, which contains the exact same logic, but is implemented based on update calls. Run the following command in your terminal window:
+
+```bash
+dfx canister call kv_frontend get_update '(1)'
+```
+
+The output will resemble the following:
+```(opt (1_337 : nat))```
+
+We can observe that with update calls we receive the very same result, but the call is at least one order of magnitude slower compared to composite query calls.
+
+:::note
+The examples repository also contains an equivalent [Motoko example](https://github.com/dfinity/examples/tree/master/motoko/composite_query).
+:::
+
+## Resources
+The following example canisters demonstrate how to use composite queries:
+
+ * [Azle example](https://github.com/demergent-labs/azle/tree/main/examples/composite_queries)
+ * [Motoko example](https://github.com/dfinity/examples/tree/master/motoko/composite_query)
+ * [Rust example](https://github.com/dfinity/examples/tree/master/rust/composite_query)
+
+Feedback and suggestions can be contributed on the forum here: [https://forum.dfinity.org/t/proposal-composite-queries/15979](https://forum.dfinity.org/t/proposal-composite-queries/15979)

sidebars.js

@@ -514,6 +514,11 @@ const sidebars = {
                   "developer-docs/integrations/vetkeys/using-vetkeys",
               ],
             },
+            {
+              type: "doc",
+              label: "Composite Queries",
+              id: "developer-docs/integrations/composite-query/composite-query",
+            },
           ],
     },
     {