Merge pull request #59 from Convex-Dev/develop

Updates for latest docs

docs/cad/003_encoding/README.md

@@ -627,34 +627,20 @@ Implementations which require more than 63 fields MAY adopt their own scheme to
 - Have the Record specify one field which contains a vector of additional fields
 - Use the first field (index 0) to specify the interpretation of following fields (which may contain arbitrary values as sub-structures)
 
-### `0xB0` - `0xB1` Byte Flags (Boolean)
-
-The possible Boolean values are `true` and `false`, which are coded as 1-byte Byte Flags.
-
-```
-Encoded as:
-0xB0 <=> false
-0xB1 <=> true
-```
-
-The two Boolean Values `true` or `false` have the Encodings `0xb1` and `0xb0` respectively. 
-
-Note: These Tags are chosen to aid human readability, such that the first hexadecimal digit `b` suggests "binary" or "boolean", and the second hexadecimal digit represents the bit value.  
-
-### `0xB2`-`0xBF` Byte Flags (Extensible)
+### `0xB0`-`0xBF` Byte Flags (Extensible)
 
 Byte flags are one byte encodings (similar to Booleans) available for application specific use.
 
 ```
 `0xBn`
 
 Where
-- n = a hex value from 2-15 
+- n = a hex value from 0-15 
 ```
 
 Applications MAY use byte flags as a efficient single byte value, i.e. the complete value encoding is always exactly one byte. For example, the encoding `0xb2` might represent an "unknown" value in ternary logic.
 
-Values `0xb0` and `0xb1` are already reserved for the two boolean values, though an application MAY repurpose these as single byte values (along with `0x00` and `0x10`) providing these values are not needed for some other purpose.
+Values `0xb0` and `0xb1` are typically used for the two boolean values, though an application MAY repurpose these as single byte values (along with `0x00` and `0x10`) providing these values are not needed for some other purpose.
 
 Fun Idea: A 1-byte Lisp where `0x10` is an opening paren, `0x00` is a closing paren and `0xb0 - 0xbf` are the allowable tokens.
 
@@ -721,6 +707,10 @@ Examples:
 - an application might define `0xE5` as an extension where the value references a known JSON schema.
 - another application might define `0xE0` as an enum where the values are the possible states of a finite state machine
 
+### `0xF0` - `0xFE` Reserved
+
+These values are reserved for possible future CAD3 extensions. at present they are illegal
+
 ### `0xFF` Illegal
 
 The `0xFF` tag is always illegal as a tag byte in any encoding.

docs/cad/015_peercomms/README.md

@@ -14,34 +14,47 @@ Objectives:
 - Asynchronous model supported by default
 - Flexibility to adapt to different transport protocols
 - Consistency with lattice data principles
+- Usage of CAD3 data for message payloads
 
 ## Messages
 
 Peers communicate via messages. A message is an atomic, asynchronous piece of data passed from a sender to a receiver.
 
-A Message consists of:
+Each message has a CAD3 payload.
+
+### Message Encoding
+
+The Message encoding format is designed to efficiently encode a CAD3 payload. Since CAD3 structures can contain multiple branches, each with their own encoding, the key idea is to send the top level encoding first, then follow this with any required child branches.
+
+A Message is normally encoded as a Blob of bytes consisting of the following:
 
-- A Message Type tag (one byte)
-- Encoded message payload (according to Cell encoding rules)
+- Encoded top level message payload (according to CAD3 Cell encoding rules)
 - Optional: One or more additional branch cell encodings
     - A VLC encoded length 
     - Branch cell encoding (according to Cell encoding rules)
 
-Encoded message data depends on the message type.
-
-Each individual cell encoding MUST fit within a fixed size buffer (currently 8192 bytes). However, by including the additional branch cell encodings, it is possible to include branch cells referenced by the payload. In this way:
+Each individual cell encoding MUST fit within a fixed size buffer (currently 16383 bytes). However, by including the additional branch cell encodings, it is possible to include branch cells referenced by the payload. In this way:
 - Large data structures can be passed in a single message
-- Branch cells can be omitted, in which case the message is regarded as **partial**. Partial messages are appropriate for values such as lattice deltas where the recipient is expected to already be in possession of the omitted branches.
+- Branch cells can be omitted, in which case the message is regarded as **partial**. Partial messages are appropriate for values such as lattice deltas where the recipient is expected to already be in possession of the omitted branches. A partial message is valid, however the receiver may not be able to access the full payload immediately.
 
-The overall size of the message is not part of the message itself, but will be provided by the transport mechanism e.g.:
+The overall size of the message is not part of the message itself, but will typically be provided by the transport mechanism e.g.:
 - For binary protocol messages, the message length precedes the message
 - For HTTP messages of type `application/cvx-raw` the message length is specified in the HTTP `Content-Length` header.
 - For messages passed as an octet stream, the message length is naturally delineated by the end of the stream
 
 ### Message Types
 
+The type of the message can be inferred from the payload. Any CAD3 value may form a valid Message, so the interpretation of these values is part of the peer protocol (i.e. application specific). 
+
+Currently recognised message types follow:
+
 #### BELIEF
 
+```
+CAD3 Payload:
+Belief
+```
+
 This message specifies a belief from an other peer that is being shared as part of the CPoS consensus algorithm.
 
 Receiving peers SHOULD validate this belief message, and if valid perform a belief merge with their current belief.
@@ -50,12 +63,13 @@ Receiving peers MAY ignore beliefs if they are experiencing high demand and need
 
 Receiving peers SHOULD ignore and/or minimise processing for beliefs that have already been received and merged. This is safe because belief merges are idempotent.
 
-#### DATA
-
-This message type provides one or more encoded cells of data. Usually, this will be a component part of a longer message or a response to a `MISSING_DATA` message.
-
 #### QUERY
 
+```
+CAD3 Payload:
+[:QR msg-id form address?]
+```
+
 This message represents a request for a peer to compute the results of a query (considered as a read-only transaction).
 
 Peers SHOULD make a best effort attempt to respond to queries from authorised clients.
@@ -64,6 +78,11 @@ Peers MAY reject queries if they are experiencing high demand. In such cases pee
 
 #### TRANSACT
 
+```
+CAD3 Payload:
+[:TX msg-id signed-transaction]
+```
+
 This message represents a request for a Peer to process a transaction by incorporating it into a subsequent Block that the Peer proposes to the Network.
 
 Peers MUST reject transactions that are not correctly signed. Failure to do so may result in slashing.
@@ -72,27 +91,42 @@ Peers MUST reject transactions that have a previously used sequence number.
 
 #### RESULT
 
+```
+CAD3 Payload:
+Result
+```
+
 This message represents the result of another message request (usually a transaction or query).
 
-A Result message MUST reference the message ID of the original message.
+A Result message MUST reference the message ID of the original message. Results that do not correspond to a pending  outgoing request ID should normally be ignored.
 
 #### STATUS
 
+```
+CAD3 Payload:
+[:SR msg-id]
+```
+
 This message represents a request for a Peer to provide information about its current status, including data about latest consensus state.
 
 Peers SHOULD respond to the status request immediately if able. 
 
 Peers MAY cache their most recent status response for efficiency reasons.
 
-#### MISSING_DATA
+#### DATA_REQUEST
+
+```
+CAD3 Payload:
+[:DR msg-id hash0 hash1 .....]
+```
 
 This message represents a request for missing data. Usually, this is sent by a peer when it is attempting to process a partial message but some data is missing locally, and it needs to acquire the missing data before proceeding.
 
-The Missing Data request must include the Value ID (hash of encoding) for the missing data. Note: It is guaranteed that if a peer has received a partial message, it must be able to determine the hashes of any directly missing data (since they will be encoded as refs in the partial message).
+The Missing Data request must include the Value IDs (hash of encoding) for the missing data branches. Note: It is guaranteed that if a peer has received a partial message, it must be able to determine the hashes of any directly missing data (since they will be encoded as branch refs in the partial message).
 
 Peer that send this message MAY suspend processing of a message pending receipt of missing data from the original sender. If the original sender is unable to satisfy this request in a timely manner, the suspended message SHOULD be discarded.
 
-Receiving Peers SHOULD respond by sending a `DATA` message containing the missing data specified. Failure to do so may result in a previous message being ignored.
+Receiving Peers SHOULD respond by sending a `RESULT` message containing the missing data specified. 
 
 ### Trust
 
@@ -108,6 +142,10 @@ Peers MAY accept messages from any source, but if they do, they SHOULD prioritis
 
 The standard mechanism for message passing is TCP connections established between peers.
 
+Messages are sent as:
+- a VLQ encoded message length N
+- N bytes representing the Message encoding
+
 ### UDP Connections
 
 UDP will be explored as a potential future transport protocol for efficiency and performance reasons.

docs/cad/017_peerops/README.md

@@ -4,9 +4,9 @@
 
 Operating a peer is an important responsibility in the Convex Network. 
 
-Anyone can run a peer, and they are responsible for maintaining the Consensus of the Network. 
+Anyone can run a peer, and they are responsible for maintaining the Consensus of the Network. They must place a minimum stake of 1000 Convex Coins.
 
-Most users of the Convex Network do not need to run a peer - they connect to peers via clients software that submits transactions and queries information on their behalf. 
+Most users of the Convex Network do not need to run a peer - they connect to peers via client software that submits transactions and queries information on their behalf. The peer API is open to client requests by default: Peer operators who wish to restrict this may need to set up additional configuration or infrastructure.
 
 This document primarily contains recommendations for peer operators
 
@@ -15,32 +15,54 @@ This document primarily contains recommendations for peer operators
 Running a Peer requires:
 
 - An Internet-connected Server
-- At least 100 MBits/sec continuous bandwidth
+- At least 100 MBits/sec continuous network bi-directional bandwidth
 - A modern processor with at least 8 dedicated Cores
-- At least 8 GB RAM (32 Gb Recommended)
-- At least 2 TB fast Storage (NVMe Recommended)
+- At least 8 GB RAM (32 GB Recommended)
+- At least 1 TB fast Storage (NVMe Recommended)
 - A secure modern operating system (Linux recommended) with good support for memory mapped files
 - Java 21 or above
 
 The network should be configured with:
-- a publicly accessible IP address (IPv4 or IPv6)
+- a publicly accessible IP address (IPv4 or IPv6). Dual stack networking support is required in the OS.
 - firewall access to the server via TCP on a chosen port (the Convex protocol default `18888` is recommended)
-- a trusted DNS entry (e.g. `peer.mycompany.com`)
+- a trusted DNS entry (e.g. `peer.mycompany.com`) is recommended
+- HTTPS certificates recommended for the HTTPS REST API
 
 The DNS entry is optional, but it will help significantly with discoverability / user access to your peer.
 
 ## Configuration
 
+### Accounts
+
+In order to operate a peer you will need a Peer Controller account. This can be any account on the Convex network, e.g. `#1678` with at least 1000 Convex Coins.
+
 ### Peer Config
 
 Peers can be configured at launch in various ways.
 
-Peers SHOULD configure the number of concurrent outgoing Peer connections according to their available bandwidth. 20 recommended for Peers with fast connections.
+#### Outgoing connections
+
+Peers MAY configure the number of concurrent outgoing Peer connections according to their available bandwidth. 20 (the default) recommended for Peers with sufficient outgoing bandwidth. There are trade-offs here:
+- With more outgoing connections, your transactions will reach consensus faster
+- You must weight this up against bandwidth costs
+- If the number is too low your published blocks may get lost if the destinations do not relay them.
+
+TODO: describe mechanism to set connection count controls
 
 
 
 ## Startup
 
+## Syncing
+
+Your peer will need to synchronize with teh network by connecting to at least one existing peer.
+
+The following peers are available at time of writing for synchronisation:
+```
+peer.convex.live:18888
+```
+
+TODO: CLI commend top start peer with target host
 
 ## Shutdown
 
@@ -56,7 +78,9 @@ It may occur that a peer becomes temporarily disconnected from the peer network.
 
 Peers are designed to automatically recover from temporary network failure and re-establish connections when possible. Peers with normal configuration SHOULD periodically re-attempt to connect with other peers randomly until connection with the Network is re-established.
 
-Peer Operators SHOULD provide for an alternative way to connect to the main network, if only for the purposes of withrawing the Peer's Stake (Peers are at risk of penalisation if they remain disconnected for too long).
+Peer Operators SHOULD provide for an alternative way to connect to the main network, if only for the purposes of withdrawing the Peer's Stake. For example, a peer operator may monitor the connectivity of their peer and use Convex Desktop to de-stake the peer if it loses connections.
+
+TODO: describe best way to monitor this. Perhaps API peer health endpoint?
 
 ### Security Breach
 
@@ -68,22 +92,34 @@ Peer Operators SHOULD attempt to withdraw their Stake immediately, possibly thro
 
 Peers are required to post a Peer Stake to participate in consensus.
 
+### Setting a stake
+
+### Withdrawing stake
+
+### Stake penalties
+
+In Protonet, there is no slashing of stake (i.e. peers are not formally penalised for incorrect behaviour).
+
+In the future peers may be automatically penalised for provably incorrect behaviour.
+
 ## Key Management
 
 ## Connection Management
 
 A Convex Peer is designed to automatically manage P2P connections to the Network during normal operations. In most cases, assuming good network connectivity, a Peer should require no manual intervention to control connections to other Peers.
 
+
+
 ### Incoming Connections
 
-Peers treat incoming connections as regular Clients, i.e. they afford no particular special priviledges to incoming connections from other Peers. The purpose of this is to ensure that Bad Peers have no particular ability to influence a Peer that they connect to. 
+Peers treat incoming connections as regular Clients, i.e. they afford no particular special privileges to incoming connections from other Peers. The purpose of this is to ensure that Bad Peers have no particular ability to influence a Peer that they connect to. 
 
 ### Outgoing connections
 
 A Peer maintains a managed list of outgoing connections (i.e. connections to which they broadcast their Beliefs).
 
 Outgoing connections follow the following rules:
-- **Validated hosts**: Peers MUST only connect to Peers accesible on the network via the host address specified for the destination Peer in the current consensus, **or** if they are explicitly instructed to connect to a specific host address by the Peer Operator (e.g. when joining the Network). This minimises the chance of connecting to Bad Peers.
+- **Validated hosts**: Peers MUST only connect to Peers accessible on the network via the host address specified for the destination Peer in the current consensus, **or** if they are explicitly instructed to connect to a specific host address by the Peer Operator (e.g. when joining the Network). This minimises the chance of connecting to Bad Peers.
 - **Random elimination**: Peers SHOULD eliminate connections at random for low-staked Peers. This allows the Peer network to stay dynamic, and give an opportunity for new Peers to be connected to
 - **Stake-weighted selection** Peers MUST connect to other Peers preferentially according to stake, so that Bad Peers do not have a significant chance of isolating a Peer
 - **Target connection count**: Peers should attempt to maintain a number of outgoing connections to Peers as configured by the Peer Operator. This allows Peer Operators to control their bandwidth usage.