Update project level docs (paritytech#1734)

* updated project level document

* updated high level overview document

* GRANDPA finality relay sequence diagram

* Parachains Finality Relay Sequence Diagram

* Messages Relay Sequence Diagram

* Complex Relayer Sequence Diagram

* small fix

* Polkadot <> Kusama bridge flowchart

* remove obsolete files

* started polkadot-kusama-bridge-overview.md

* continue polkadot-kusama-bridge-overview.md

* couple more sections in polkadot-kusama-bridge-overview.md

* continue polkadot-kusama-bridge-overview.md

* renew deployments readme

* fixed review suggestions

* Update docs/high-level-overview.md

Co-authored-by: Adrian Catangiu <adrian@parity.io>

* removed obsolete section

* Update docs/high-level-overview.md

Co-authored-by: Adrian Catangiu <adrian@parity.io>

* Update docs/high-level-overview.md

Co-authored-by: Adrian Catangiu <adrian@parity.io>

* Update docs/high-level-overview.md

Co-authored-by: Adrian Catangiu <adrian@parity.io>

* Update docs/high-level-overview.md

Co-authored-by: Adrian Catangiu <adrian@parity.io>

* typo

* Update docs/polkadot-kusama-bridge-overview.md

Co-authored-by: Adrian Catangiu <adrian@parity.io>

* Update docs/high-level-overview.md

Co-authored-by: Adrian Catangiu <adrian@parity.io>

* Update docs/high-level-overview.md

Co-authored-by: Adrian Catangiu <adrian@parity.io>

* Update docs/high-level-overview.md

Co-authored-by: Adrian Catangiu <adrian@parity.io>

* Update docs/high-level-overview.md

Co-authored-by: Adrian Catangiu <adrian@parity.io>

* Update docs/polkadot-kusama-bridge-overview.md

Co-authored-by: Adrian Catangiu <adrian@parity.io>

* Update docs/polkadot-kusama-bridge-overview.md

Co-authored-by: Adrian Catangiu <adrian@parity.io>

* Update docs/polkadot-kusama-bridge-overview.md

Co-authored-by: Adrian Catangiu <adrian@parity.io>

* Update docs/polkadot-kusama-bridge-overview.md

Co-authored-by: Adrian Catangiu <adrian@parity.io>

Co-authored-by: Adrian Catangiu <adrian@parity.io>

README.md

@@ -10,11 +10,6 @@ Substrate chains.
 
 🚧 The bridges are currently under construction - a hardhat is recommended beyond this point 🚧
 
-**IMPORTANT**: this documentation is outdated and it is mostly related to the previous version of our
-bridge. Right there's an ongoing work to make our bridge work with XCM messages. Old bridge is still
-available at [encoded-calls-messaging](https://github.com/paritytech/parity-bridges-common/releases/tag/encoded-calls-messaging)
-tag.
-
 ## Contents
 
 - [Installation](#installation)
@@ -97,7 +92,7 @@ description of the bridge interaction.
 
 ## Project Layout
 
-Here's an overview of how the project is laid out. The main bits are the `node`, which is the actual
+Here's an overview of how the project is laid out. The main bits are the `bin`, which is the actual
 "blockchain", the `modules` which are used to build the blockchain's logic (a.k.a the runtime) and
 the `relays` which are used to pass messages between chains.
 
@@ -106,16 +101,16 @@ the `relays` which are used to pass messages between chains.
 │  └── ...
 ├── deployments     // Useful tools for deploying test networks
 │  └──  ...
-├── diagrams        // Pretty pictures of the project architecture
-│  └──  ...
 ├── modules         // Substrate Runtime Modules (a.k.a Pallets)
+│  ├── beefy        // On-Chain BEEFY Light Client (in progress)
 │  ├── grandpa      // On-Chain GRANDPA Light Client
 │  ├── messages     // Cross Chain Message Passing
-│  ├── dispatch     // Target Chain Message Execution
+│  ├── parachains   // On-Chain Parachains Light Client
+│  ├── relayers     // Relayer rewards registry
 │  └──  ...
 ├── primitives      // Code shared between modules, runtimes, and relays
 │  └──  ...
-├── relays          // Application for sending headers and messages between chains
+├── relays          // Application for sending finality proofs and messages between chains
 │  └──  ...
 └── scripts         // Useful development and maintenance scripts
 ```
@@ -127,8 +122,11 @@ on each side of the bridge (source and target chain).
 
 There are 2 ways to run the bridge, described below:
 
-- building & running from source
-- running a Docker Compose setup (recommended).
+- building & running from source: with this option, you'll be able to run the bridge between two standalone
+chains that are running GRANDPA finality gadget to achieve finality;
+
+- running a Docker Compose setup: this is a recommended option, where you'll see bridges with parachains,
+complex relays and more.
 
 ### Using the Source
 
@@ -204,7 +202,33 @@ You will also see the message lane relayers listening for new messages.
 
 To send a message see the ["How to send a message" section](#how-to-send-a-message).
 
-### Full Network Docker Compose Setup
+### How to send a message
+
+In this section we'll show you how to quickly send a bridge message. The message is just an encoded XCM
+`Trap(43)` message.
+
+```bash
+# In `parity-bridges-common` folder
+./scripts/send-message-from-millau-rialto.sh
+```
+
+After sending a message you will see the following logs showing a message was successfully sent:
+
+```
+INFO bridge Sending message to Rialto. Size: 5.
+TRACE bridge Sent transaction to Millau node: 0x5e68...
+```
+
+And at the Rialto node logs you'll something like this:
+
+```
+... runtime::bridge-dispatch: Going to execute message ([0, 0, 0, 0], 1) (...), Trap(43)])    
+... runtime::bridge-dispatch: Incoming message ([0, 0, 0, 0], 1) dispatched with result: Incomplete(2000000000, Trap(43))    
+``` 
+
+It means that the message has been delivered and successfully dispatched.
+
+## Full Network Docker Compose Setup
 
 For a more sophisticated deployment which includes bidirectional header sync, message passing,
 monitoring dashboards, etc. see the [Deployments README](./deployments/README.md).
@@ -220,24 +244,6 @@ docker run -p 30333:30333 -p 9933:9933 -p 9944:9944 \
   --rpc-cors=all --unsafe-rpc-external --unsafe-ws-external
 ```
 
-### How to send a message
-
-In this section we'll show you how to quickly send a bridge message, if you want to
-interact with and test the bridge see more details in [send message](./docs/send-message.md)
-
-```bash
-# In `parity-bridges-common` folder
-./scripts/send-message-from-millau-rialto.sh remark
-```
-
-After sending a message you will see the following logs showing a message was successfully sent:
-
-```
-INFO bridge Sending message to Rialto. Size: 286. Dispatch weight: 1038000. Fee: 275,002,568
-INFO bridge Signed Millau Call: 0x7904...
-TRACE bridge Sent transaction to Millau node: 0x5e68...
-```
-
 ## Community
 
 Main hangout for the community is [Element](https://element.io/) (formerly Riot). Element is a chat

deployments/README.md

@@ -1,11 +1,13 @@
 # Bridge Deployments
 
 ## Requirements
+
 Make sure to install `docker` and `docker-compose` to be able to run and test bridge deployments. If
 for whatever reason you can't or don't want to use Docker, you can find some scripts for running the
-bridge [here](https://github.com/svyatonik/parity-bridges-common.test).
+bridge [here](./local-scripts/).
 
 ## Networks
+
 One of the building blocks we use for our deployments are _networks_. A network is a collection of
 homogenous blockchain nodes. We have Docker Compose files for each network that we want to bridge.
 Each of the compose files found in the `./networks` folder is able to independently spin up a
@@ -18,6 +20,7 @@ docker-compose -f ./networks/rialto.yml up
 After running this command we would have a network of several nodes producing blocks.
 
 ## Bridges
+
 A _bridge_ is a way for several _networks_ to connect to one another. Bridge deployments have their
 own Docker Compose files which can be found in the `./bridges` folder. These Compose files typically
 contain bridge relayers, which are services external to blockchain nodes, and other components such
@@ -43,10 +46,11 @@ and Grafana. We cover these in more details in the [Monitoring](#monitoring) sec
 the monitoring Compose file is _not_ optional, and must be included for bridge deployments.
 
 ### Running and Updating Deployments
+
 We currently support three bridge deployments
 1. Rialto Substrate to Millau Substrate
 2. Rialto Parachain Substrate to Millau Substrate
-2. Westend Substrate to Millau Substrate
+2. Westend Substrate to Millau Substrate (only finality bridge)
 
 These bridges can be deployed using our [`./run.sh`](./run.sh) script.
 
@@ -71,11 +75,12 @@ You can also bring down a deployment using the script with the `stop` argument.
 ```
 
 ### Adding Deployments
+
 We need two main things when adding a new deployment. First, the new network which we want to
 bridge. A compose file for the network should be added in the `/networks/` folder. Secondly we'll
 need a new bridge Compose file in `./bridges/`. This should configure the bridge relayer nodes
 correctly for the two networks, and add any additional components needed for the deployment. If you
-want you can also add support in the `./run` script for the new deployment. While recommended it's
+want you can also add support in the `./run.sh` script for the new deployment. While recommended it's
 not strictly required.
 
 ## General Notes
@@ -111,8 +116,8 @@ is not recommended, because this may lead to nonces conflict.
 
 Following accounts are used when `rialto-millau` bridge is running:
 
-- Millau's `Rialto.HeadersAndMessagesRelay` signs complex headers+messages relay transactions on Millau chain;
-- Rialto's `Millau.HeadersAndMessagesRelay` signs complex headers+messages relay transactions on Rialto chain;
+- Millau's `Rialto.HeadersAndMessagesRelay1` signs complex headers+messages relay transactions on Millau chain;
+- Rialto's `Millau.HeadersAndMessagesRelay1` signs complex headers+messages relay transactions on Rialto chain;
 - Millau's `Rialto.MessagesSender` signs Millau transactions which contain messages for Rialto;
 - Rialto's `Millau.MessagesSender` signs Rialto transactions which contain messages for Millau;
 - Millau's `Rialto.OutboundMessagesRelay.Lane00000001` signs relay transactions with message delivery confirmations (lane 00000001) from Rialto to Millau;
@@ -136,18 +141,22 @@ Following accounts are used when `rialto-parachain-millau` bridge is running:
 - RialtoParachain's `Millau.HeadersAndMessagesRelay` signs complex headers+messages relay transactions on RialtoParachain chain.
 
 ### Docker Usage
+
 When the network is running you can query logs from individual nodes using:
 
 ```bash
 docker logs rialto_millau-node-charlie_1 -f
 ```
 
+You may use the [dump-logs.sh](../scripts/dump-logs.sh) to dump logs of most of running containers.
+
 To kill all leftover containers and start the network from scratch next time:
 ```bash
 docker ps -a --format "{{.ID}}" | xargs docker rm # This removes all containers!
 ```
 
 ### Docker Compose Usage
+
 If you're not familiar with how to use `docker-compose` here are some useful commands you'll need
 when interacting with the bridge deployments:
 
@@ -172,10 +181,12 @@ docker-compose -f docker-compose.yml -f docker-compose.override.yml config > doc
 ```
 
 ## Docker and Git Deployment
+
 It is also possible to avoid using images from the Docker Hub and instead build
 containers from Git. There are two ways to build the images this way.
 
 ### Git Repo
+
 If you have cloned the bridges repo you can build local Docker images by running the following
 command at the top level of the repo:
 
@@ -189,16 +200,17 @@ This will build a local image of a particular component with a tag of
 You can configure the build using Docker
 [build arguments](https://docs.docker.com/engine/reference/commandline/build/#set-build-time-variables---build-arg).
 Here are the arguments currently supported:
-  - `BRIDGE_REPO`: Git repository of the bridge node and relay code
-  - `BRIDGE_HASH`: Commit hash within that repo (can also be a branch or tag)
-  - `ETHEREUM_REPO`: Git repository of the OpenEthereum client
-  - `ETHEREUM_HASH`: Commit hash within that repo (can also be a branch or tag)
   - `PROJECT`: Project to build withing bridges repo. Can be one of:
     - `rialto-bridge-node`
     - `millau-bridge-node`
+    - `rialto-parachain-collator`
     - `substrate-relay`
 
+You may use the [build-containers.sh](../scripts/build-containers.sh) script to build all available
+containers.
+
 ### GitHub Actions
+
 We have a nightly job which runs and publishes Docker images for the different nodes and relayers to
 the [ParityTech Docker Hub](https://hub.docker.com/u/paritytech) organization. These images are used
 for our ephemeral (temporary) test networks. Additionally, any time a tag in the form of `v*` is
@@ -209,6 +221,7 @@ With images built using either method, all you have to do to use them in a deplo
 `image` field in the existing Docker Compose files to point to the tag of the image you want to use.
 
 ### Monitoring
+
 [Prometheus](https://prometheus.io/) is used by the bridge relay to monitor information such as system
 resource use, and block data (e.g the best blocks it knows about). In order to visualize this data
 a [Grafana](https://grafana.com/) dashboard can be used.
@@ -223,6 +236,7 @@ dashboard can be accessed at `http://localhost:9090`. The Grafana dashboard can
 `http://localhost:3000`. Note that the default log-in credentials for Grafana are `admin:admin`.
 
 ### Environment Variables
+
 Here is an example `.env` file which is used for production deployments and network updates. For
 security reasons it is not kept as part of version control. When deploying a network this
 file should be correctly populated and kept in the appropriate [`bridges`](`./bridges`) deployment
@@ -247,13 +261,8 @@ UI_EXPECTED_ETHEREUM_NETWORK_ID=105
 
 ### UI
 
-Use [wss://rialto.bridges.test-installations.parity.io/](https://polkadot.js.org/apps/)
-as a custom endpoint for [https://polkadot.js.org/apps/](https://polkadot.js.org/apps/).
-
-### Polkadot.js UI
-
-To teach the UI decode our custom types used in the pallet, go to: `Settings -> Developer`
-and import the [`./types.json`](./types.json)
+Use [wss://wss.rialto.brucke.link](https://polkadot.js.org/apps/) as a custom endpoint for
+[https://polkadot.js.org/apps/](https://polkadot.js.org/apps/).
 
 ## Scripts
 

docs/complex-relay.html

@@ -0,0 +1,85 @@
+<!DOCTYPE html>
+<html>
+	<head>
+		<meta charset="utf-8">
+		<meta name="viewport" content="width=device-width">
+		<title>Complex Relay</title>
+	</head>
+	<body>
+		<h1>Complex Relay</h1>
+		<p>
+			Both Source Chain and Target Chains have Bridge Messages pallets deployed. They also have required
+			finality pallets deployed - we don't care about finality type here - they can be either Bridge GRANDPA,
+			or Bridge Parachains finality pallets, or any combination of those.<br/>
+		</p>
+		<p>
+			There are 4-6 relayer subprocesses inside the Complex Relayer. They include two message relayers,
+			serving the lane in both directions and 2-4 Complex Relayers (depending on the finality type of Source
+			and Target Chains).<br/>
+		</p>
+		<p>
+			The following diagram shows the way the complex relayer serves the lane in single direction. Everything
+			below may be applied to the opposite direction if you'll swap the Source and Target Chains.
+		</p>
+		<div class="mermaid">
+			sequenceDiagram
+			participant Source Chain
+			participant Complex Relayer
+			participant Target Chain
+
+			Note right of Source Chain: Finalized: 480, Target Finalized: 50, Sent Messages: 42, Confirmed Messages: 42
+			Note left of Target Chain: Finalized: 60, Source Finalized: 420, Received Messages: 42
+
+			Source Chain ->> Source Chain: someone Sends Message 43
+			Source Chain ->> Source Chain: Import and Finalize Block 481
+
+			Source Chain ->> Complex Relayer: notes new outbound message 43 at Source Chain Block 481
+			Note right of Complex Relayer: can't deliver message 43, Source Chain Block 481 is not relayed
+			Complex Relayer ->> Complex Relayer: asks on-demand Finality Relayer to relay Source Chain Block 481
+
+			Source Chain ->> Complex Relayer: Read Finality Proof of Block 481
+			Complex Relayer ->> Target Chain: Submit Finality Proof of Block 481
+			Target Chain ->> Target Chain: Import and Finalize Block 61
+			Note left of Target Chain: Finalized: 61, Source Finalized: 481, Received Messages: 42
+
+			Source Chain ->> Complex Relayer: Read Proof of Message 43 at Block 481
+			Complex Relayer ->> Target Chain: Submit Proof of Message 43 at Block 481
+			Target Chain ->> Target Chain: Import and Finalize Block 62
+			Note left of Target Chain: Finalized: 62, Source Finalized: 481, Received Messages: { rewarded: 42, messages-relayer-account: [43] }
+
+			Target Chain ->> Complex Relayer: notes new unrewarded relayer at Target Chain Block 62
+			Note right of Complex Relayer: can't relay delivery confirmations because Target Chain Block 62 is not relayed
+			Complex Relayer ->> Complex Relayer: asks on-demand Finality Relayer to relay Target Chain Block 62
+
+			Target Chain ->> Complex Relayer: Read Finality Proof of Block 62
+			Complex Relayer ->> Source Chain: Submit Finality Proof of Block 62
+			Source Chain ->> Source Chain: Import and Finalize Block 482
+			Note right of Source Chain: Finalized: 482, Target Finalized: 62, Confirmed Messages: 42
+
+			Target Chain ->> Complex Relayer: Read Proof of Message 43 Delivery at Block 62
+			Complex Relayer ->> Source Chain: Submit Proof of Message 43 Delivery at Block 612
+			Source Chain ->> Source Chain: rewards messages-relayer-account for delivering message [43]
+			Source Chain ->> Source Chain: prune delivered message 43 from runtime storage
+			Note right of Source Chain: Finalized: 482, Target Finalized: 61, Confirmed Messages: 43
+
+			Source Chain ->> Source Chain: someone Sends Message 44
+			Source Chain ->> Source Chain: Import and Finalize Block 483
+
+			Source Chain ->> Complex Relayer: notes new outbound message 44 at Source Chain Block 483 and new confirmed message 43
+			Note right of Complex Relayer: can't deliver message 44, Source Chain Block 483 is not relayed
+			Complex Relayer ->> Complex Relayer: asks on-demand Finality Relayer to relay Source Chain Block 483
+
+			Source Chain ->> Complex Relayer: Read Finality Proof of Block 483
+			Complex Relayer ->> Target Chain: Submit Finality Proof of Block 483
+			Target Chain ->> Target Chain: Import and Finalize Block 63
+			Note left of Target Chain: Finalized: 63, Source Finalized: 483, Received Messages: { rewarded: 42, messages-relayer-account: [43] }
+
+			Source Chain ->> Complex Relayer: Read Proof of Message 44 and Proof of Message 43 reward at Block 483
+			Complex Relayer ->> Target Chain: Submit Proof of Message 44 and Proof of Message 43 reward at Block 483
+			Target Chain ->> Target Chain: Import and Finalize Block 64
+			Note left of Target Chain: Finalized: 64, Source Finalized: 483, Received Messages: { rewarded: 43, messages-relayer-account: [44] }-->
+		</div>
+		<script src="https://cdn.jsdelivr.net/npm/mermaid@8.8.4/dist/mermaid.min.js"></script>
+		<script>mermaid.initialize({startOnLoad: true})</script>
+	</body>
+</html>