Skip to content

Commit

Permalink
Merge branch 'master' into feature/DOPS-2654/gdb
Browse files Browse the repository at this point in the history
  • Loading branch information
Cre-eD authored Sep 1, 2023
2 parents 8259e3c + a218191 commit 9f90dc8
Show file tree
Hide file tree
Showing 20 changed files with 460 additions and 112 deletions.
81 changes: 69 additions & 12 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,23 +11,79 @@ KAGOME is a [Polkadot Host](https://github.com/w3f/polkadot-spec/tree/master/hos

## Status

- [x] Syncing node
- Polkadot, Kusama and Rococo compatibility
- [x] Validating node
- [x] Polkadot JS apps support
- [ ] Parachains support
![kagome-components-Host drawio-light](https://github.com/soramitsu/kagome/assets/9370151/323c1677-b628-468c-adb8-5c09c2164fc3)


- [x] JSON-RPC (compatible with Polkadot JS)
- [ ] New [unstable JSON-RPC](https://paritytech.github.io/json-rpc-interface-spec/)
- [x] Scale codec
- [x] Synchronizer
- [x] Full sync
- [x] Fast sync
- [x] Warp sync
- [x] Transaction pool
- [x] Consensus
- [x] BABE
- [x] GRANDPA
- [ ] SASSAFRAS (Q4 2023)
- [x] Storage
- [x] Blockchain
- [x] Block storage
- [x] Block tree
- [x] Digest tracker
- [x] Trie storage (merkle trie)
- [x] RocksDB
- [x] Dynamic pruning
- [ ] Trie nodes caches (Q4 2023)
- [ ] State caches (Q4 2023)
- [x] Runtime
- [x] Host API
- [x] WASM engine
- [x] Binaryen
- [x] WAVM
- [ ] WasmEdge (planned for Q3 2023)
- [x] Parachains core
- [x] Non-asynchronous Backing
- [x] Data availability
- [x] Approval voting
- [ ] Disputes resolution
- [x] Disputes resolution
- [ ] Async-Backing (planned for Q3 2023)
- [x] Networking
- [x] Peer manager
- [x] /dot/block-announces/1
- [x] /paritytech/grandpa/1
- [x] /polkadot/validation/1
- [x] /polkadot/collation/1
- [x] /dot/transactions/1
- [x] /polkadot/req_collation/1
- [x] /dot/light/2
- [x] /polkadot/req_pov/1
- [x] /dot/state/2
- [x] /dot/sync/warp
- [x] /polkadot/req_statement/1
- [x] /dot/sync/2
- [x] /polkadot/req_available_data/1
- [x] /polkadot/req_chunk/1
- [x] /polkadot/send_dispute/1
- [x] Libp2p
- [x] Transport
- [x] TCP
- [ ] QUIC (Q4 2023)
- [ ] WebRTC (Q4 2023)
- [x] Secure connection
- [x] Noise
- [x] TLS
- [x] Multiplexing
- [x] Yamux
- [x] Multiselect protocol
- [x] Peer discovery
- [x] Kademlia
- [x] Ping protocol
- [x] Identify protocol
- [x] Offchain workers
- [x] Keystore
- [x] Telemetry support
- [x] Prometheus metrics
- [x] Fast sync
- [ ] Warp sync
- [x] Incoming connections
- [ ] Outcoming connections
- [x] Light client protocol

More details of KAGOME development can be found within the [supported features](./README.md/#supported-features) section and in [projects board](https://github.com/soramitsu/kagome/projects/2)

Expand Down Expand Up @@ -201,4 +257,5 @@ You can find more information about the components by checking [reference docume
* Press-release: [Soramitsu to implement Polkadot Runtime Environment in C++](https://medium.com/web3foundation/w3f-grants-soramitsu-to-implement-polkadot-runtime-environment-in-c-cf3baa08cbe6)
* [KAGOME: C++ implementation of PRE](https://www.youtube.com/watch?v=181mk2xvBZ4&t=) presentation at DOTCon (18.08.19)
* [KAGOME and consensus in Polkadot](https://www.youtube.com/watch?v=5OrevTjaiPA) presentation (in Russian) during Innopolis blockchain meetup (28.10.19)
* [Web3 Builders: Soramitsu | C++ Implementation of Polkadot Host](https://www.youtube.com/watch?v=We3kiGzg60w) Polkadot's Web3 builders online presentation
* [Web3 Builders: Soramitsu | C++ Implementation of Polkadot Host](https://www.youtube.com/watch?v=We3kiGzg60w) Polkadot's Web3 builders online presentation
* [Building alternative clients](https://youtu.be/TnENz6I9l8A?si=yF4oo2zQ6LdPW13N) Polkadot Decoded 2023 presentation
53 changes: 53 additions & 0 deletions core/consensus/beefy/sig.hpp
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
/**
* Copyright Soramitsu Co., Ltd. All Rights Reserved.
* SPDX-License-Identifier: Apache-2.0
*/

#pragma once

#include "consensus/beefy/types.hpp"
#include "crypto/ecdsa_provider.hpp"
#include "crypto/hasher/hasher_impl.hpp"

namespace kagome::consensus::beefy {
inline common::Hash256 prehash(const Commitment &commitment) {
// TODO(turuslan): #1768, scale encode to hash
return crypto::HasherImpl{}.keccak_256(scale::encode(commitment).value());
}

inline bool verify(const crypto::EcdsaProvider &ecdsa,
const VoteMessage &vote) {
auto r = ecdsa.verifyPrehashed(
prehash(vote.commitment), vote.signature, vote.id);
return r and r.value();
}

inline size_t threshold(size_t n) {
return n == 0 ? 0 : n - (n - 1) / 3;
}

inline bool verify(const crypto::EcdsaProvider &ecdsa,
const BeefyJustification &justification_v1,
const ValidatorSet &validators) {
auto &justification = boost::get<SignedCommitment>(justification_v1);
if (justification.commitment.validator_set_id != validators.id) {
return false;
}
auto total = validators.validators.size();
if (justification.signatures.size() != total) {
return false;
}
auto prehashed = prehash(justification.commitment);
size_t valid = 0;
for (size_t i = 0; i < total; ++i) {
if (auto &sig = justification.signatures[i]) {
if (auto r = ecdsa.verifyPrehashed(
prehashed, *sig, validators.validators[i]);
r and r.value()) {
++valid;
}
}
}
return valid >= threshold(total);
}
} // namespace kagome::consensus::beefy
114 changes: 114 additions & 0 deletions core/consensus/beefy/types.hpp
Original file line number Diff line number Diff line change
@@ -0,0 +1,114 @@
/**
* Copyright Soramitsu Co., Ltd. All Rights Reserved.
* SPDX-License-Identifier: Apache-2.0
*/

#pragma once

#include "common/unused.hpp"
#include "crypto/ecdsa_types.hpp"
#include "primitives/authority.hpp"

/**
* Test
* ConsensusDigest
* 0x0108020a1091341fe5664bfa1782d5e04779689068c916b04cb365ec3153755684d9a10390084fdbf27d2b79d26a4f13f0ccd982cb755a661969143c37cbc49ef5b91f270700000000000000
* BeefyJustification
* 0x01046d68803af1ad0102f711a7c08e589a1006e4f20c8853b12b5214a57a08cbb4c72cf2ce47000000070000000000000004c002000000080e0fa849fcd9ecfed1b1312e7a17bb4db4ec02761ac760b01a9fc7365c2f55a059125b6217943b561aa27c8b1f990eee1cc9b72ff6f4d6ddde467e33dd02142500f016a7aa597346546f0e799016c8a5302c7a6dce286c513bd69c60e1e77b1e2f6bff5c269369b4ede6fd6e41b32186faff8773158708b16a35d2afcdc9aeeaa500
*/

namespace kagome::consensus::beefy {
using MmrRootHash = common::Hash256;

struct ValidatorSet {
SCALE_TIE(2);

std::vector<crypto::EcdsaPublicKey> validators;
primitives::AuthoritySetId id;
};

using ConsensusDigest =
boost::variant<Unused<0>,
ValidatorSet, // AuthoritiesChange
primitives::AuthorityIndex, // OnDisabled
MmrRootHash>;

using PayloadId = common::Blob<2>;

struct Commitment {
SCALE_TIE(3);

std::vector<std::pair<PayloadId, common::Buffer>> payload;
primitives::BlockNumber block_number;
primitives::AuthoritySetId validator_set_id;
};

struct VoteMessage {
SCALE_TIE(3);

Commitment commitment;
crypto::EcdsaPublicKey id;
crypto::EcdsaSignature signature;
};

struct SignedCommitment {
Commitment commitment;
std::vector<std::optional<crypto::EcdsaSignature>> signatures;
};
scale::ScaleEncoderStream &operator<<(scale::ScaleEncoderStream &s,
const SignedCommitment &v) {
s << v.commitment;
size_t count = 0;
common::Buffer bits;
// https://github.com/paritytech/substrate/blob/55bb6298e74d86be12732fd0f120185ee8fbfe97/primitives/consensus/beefy/src/commitment.rs#L149-L152
bits.resize(v.signatures.size() / 8 + 1);
auto i = 0;
for (auto &sig : v.signatures) {
if (sig) {
++count;
bits[i / 8] |= 1 << (7 - i % 8);
}
++i;
}
s << bits;
s << static_cast<uint32_t>(v.signatures.size());
s << scale::CompactInteger{count};
for (auto &sig : v.signatures) {
if (sig) {
s << *sig;
}
}
return s;
}
scale::ScaleDecoderStream &operator>>(scale::ScaleDecoderStream &s,
SignedCommitment &v) {
s >> v.commitment;
common::Buffer bits;
s >> bits;
size_t expected_count = 0;
for (auto byte : bits) {
for (; byte; byte >>= 1) {
expected_count += byte & 1;
}
}
uint32_t total = 0;
s >> total;
if (bits.size() * 8 < total) {
scale::raise(scale::DecodeError::NOT_ENOUGH_DATA);
}
scale::CompactInteger actual_count;
s >> actual_count;
if (actual_count != expected_count) {
scale::raise(scale::DecodeError::TOO_MANY_ITEMS);
}
v.signatures.resize(total);
for (size_t i = 0; i < total; ++i) {
if ((bits[i / 8] & (1 << (7 - i % 8))) != 0) {
s >> v.signatures[i].emplace();
}
}
return s;
}

using BeefyJustification = boost::variant<Unused<0>, SignedCommitment>;
} // namespace kagome::consensus::beefy
69 changes: 38 additions & 31 deletions core/network/impl/protocols/protocol_base_impl.hpp
Original file line number Diff line number Diff line change
Expand Up @@ -3,8 +3,7 @@
* SPDX-License-Identifier: Apache-2.0
*/

#ifndef KAGOME_NETWORK_PROTOCOLBASEIMPL
#define KAGOME_NETWORK_PROTOCOLBASEIMPL
#pragma once

#include "network/protocol_base.hpp"

Expand Down Expand Up @@ -46,26 +45,34 @@ namespace kagome::network {
}

template <typename T>
bool start(std::weak_ptr<T> wptr) {
bool start(std::weak_ptr<T> wp) {
host_.setProtocolHandler(
protocols_,
[log{logger()}, wp(std::move(wptr))](auto &&stream_and_proto) {
network::streamReadBuffer(stream_and_proto);
if (auto peer_id = stream_and_proto.stream->remotePeerId()) {
SL_TRACE(log,
"Handled {} protocol stream from: {}",
stream_and_proto.protocol,
peer_id.value().toBase58());
if (auto self = wp.lock()) {
self->onIncomingStream(std::move(stream_and_proto.stream));
[wp = std::move(wp), log = logger()](auto &&stream_and_proto) {
if (auto self = wp.lock()) {
BOOST_ASSERT(stream_and_proto.stream);

network::streamReadBuffer(stream_and_proto);

auto &[stream, protocol] = stream_and_proto;
BOOST_ASSERT(stream);

if (auto peer_id = stream->remotePeerId()) {
SL_TRACE(log,
"Handled {} protocol stream from {}",
protocol,
peer_id);
BOOST_ASSERT(stream);
self->onIncomingStream(std::move(stream));
return;
}
} else {
log->warn("Handled {} protocol stream from unknown peer",
stream_and_proto.protocol);

SL_WARN(log,
"Handled {} protocol stream from unknown peer",
protocol);
BOOST_ASSERT(stream);
stream->close([](auto &&) {});
}
stream_and_proto.stream->close(
[stream{stream_and_proto.stream}](auto &&) {});
});
return true;
}
Expand All @@ -87,22 +94,24 @@ namespace kagome::network {
}

template <typename T>
void closeStream(std::weak_ptr<T> wptr, std::shared_ptr<Stream> stream) {
void closeStream(std::weak_ptr<T> wp, std::shared_ptr<Stream> stream) {
BOOST_ASSERT(stream);
stream->close([log{logger()}, wptr, stream](auto &&result) {
if (auto self = wptr.lock()) {
if (!result) {
stream->close([wp = std::move(wp),
log = logger(),
peer_id = stream->remotePeerId().value()](auto &&result) {
if (auto self = wp.lock()) {
if (result.has_value()) {
SL_DEBUG(log,
"Stream {} was not closed successfully with {}",
"Stream {} with {} was closed.",
self->protocolName(),
stream->remotePeerId().value());

} else {
SL_VERBOSE(log,
"Stream {} with {} was closed.",
self->protocolName(),
stream->remotePeerId().value());
peer_id);
return;
}
SL_DEBUG(log,
"Stream {} was not closed successfully with {}: {}",
self->protocolName(),
peer_id,
result.error());
}
});
}
Expand All @@ -115,5 +124,3 @@ namespace kagome::network {
};

} // namespace kagome::network

#endif // KAGOME_NETWORK_PROTOCOLBASEIMPL
Loading

0 comments on commit 9f90dc8

Please sign in to comment.