Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

GetRange for requests #1

Draft
wants to merge 57 commits into
base: without_middle_spend
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
57 commits
Select commit Hold shift + click to select a range
71433c2
chore(node): trust a parent spend with majority
maqi Jul 25, 2024
e9d536d
feat(networking): keep NetworkSwarmCmd sender to avoid blocking thread
joshuef Jul 30, 2024
ba065fa
feat(networking): move more send_request off thread
joshuef Jul 30, 2024
561e533
feat(networking): maintain record_store cache_map to increase efficiency
joshuef Jul 30, 2024
6c503f0
feat(networking): early exit on put if same record exists
joshuef Jul 30, 2024
232c49b
feat(networking): dedupe outgoing queries against existing inflight q…
joshuef Jul 30, 2024
bfa50a3
Merge pull request #2010 from joshuef/FurtherNetCodeSeperation
RolandSherwin Jul 30, 2024
14f8186
chore: correct logging of cash_note and confirmed_spend disk ops
maqi Jul 31, 2024
79cc0a6
Merge pull request #2015 from maqi/check_confirmed_spend
maqi Jul 31, 2024
6413eb7
chore(cli): [#2012] show download time per file
mickvandijke Jul 31, 2024
7729acb
chore(cli): [#2014] explicitly show the uploaded file's address in th…
mickvandijke Jul 31, 2024
78bf4fa
fix(node): check whether already received an incoming payment
maqi Jul 31, 2024
c268215
Merge pull request #2018 from mickvandijke/2014-cli-upload-output-fil…
RolandSherwin Jul 31, 2024
1136410
Merge pull request #2017 from maqi/cash_note_name
RolandSherwin Jul 31, 2024
59d8528
Merge pull request #2016 from mickvandijke/2012-cli-add-download-dura…
RolandSherwin Jul 31, 2024
4255e0d
Merge pull request #2007 from maqi/relax_on_parent_spend_verification
maqi Aug 1, 2024
6d184b1
chore(node): check confirmed_spend existence
maqi Aug 1, 2024
13b885a
Merge pull request #2020 from maqi/check_spent
joshuef Aug 2, 2024
05f3feb
chore(test): add more logs to the verify routing table test
RolandSherwin Aug 5, 2024
197c586
fix(ci): clean out target directory to save space
RolandSherwin Aug 5, 2024
82a2d6b
feat: strip debug info from dev profile
RolandSherwin Aug 5, 2024
898446f
Merge pull request #2022 from RolandSherwin/fix_ci
RolandSherwin Aug 5, 2024
bd05f53
docs: update readme about transfers
grumbach Aug 5, 2024
cd09145
feat(metrics): track the bad node count
RolandSherwin Aug 1, 2024
4026edd
feat(metrics): track the number of times we've been shunned
RolandSherwin Aug 1, 2024
6dc63b4
chore(node): use PeerId for the reporting node
RolandSherwin Aug 5, 2024
492f50d
fix(node): terminate make_payment process during unrecoverable error
maqi Aug 5, 2024
816ef88
chore(node): [#1999] panic on upnp event `GatewayNotFound`
mickvandijke Jul 24, 2024
5e61746
Merge pull request #2003 from mickvandijke/1999-node-should-panic-on-…
mickvandijke Aug 5, 2024
b4da842
Merge pull request #2021 from RolandSherwin/bad_node_metric
RolandSherwin Aug 5, 2024
5200a00
Merge pull request #2023 from maqi/upload_un-recoverable_error_during…
maqi Aug 5, 2024
176f63c
fix(peers): timeout if we cannot get the network contacts
RolandSherwin Jul 30, 2024
8222681
fix(manager): set the node startup interval based on the time to conn…
RolandSherwin Jul 30, 2024
603c1a6
feat(manager): implement a moving average of sleep intervals
RolandSherwin Jul 30, 2024
8a4155d
feat(manager): implement dynamic interval during node upgrade
RolandSherwin Jul 30, 2024
0b7496f
feat(manager): provide rpc connection timeout during start and upgrade
RolandSherwin Jul 30, 2024
bc73a09
fix(peers): remove timeout from wasm32
RolandSherwin Jul 31, 2024
10bf30c
fix(test): update the node manager tests to expect the new rpc call
RolandSherwin Jul 31, 2024
2eda868
fix(manager): make 'interval' and 'connection-timeout' args mutually …
RolandSherwin Aug 1, 2024
60700bd
fix(manager): check if node has been connected to the network
RolandSherwin Aug 1, 2024
42a1368
chore(manager): update docs based on comment
RolandSherwin Aug 1, 2024
b0fa120
fix(manager): do not sleep if we're already waiting for the node to c…
RolandSherwin Aug 6, 2024
52b1faa
Merge pull request #2024 from grumbach/update_readme_transfers
b-zee Aug 6, 2024
d5efb2f
Merge pull request #2013 from RolandSherwin/dynamic_startup_time
RolandSherwin Aug 6, 2024
807e2ca
fix(network): add peer to blocklist only after informing it
RolandSherwin Aug 7, 2024
cf8e671
fix(node): avoid deserialise error cause a confirmed_spend to be non-…
maqi Aug 7, 2024
a1f8fa3
Merge pull request #2026 from RolandSherwin/inform_peer_before_blocklist
RolandSherwin Aug 7, 2024
4afef00
Merge pull request #2027 from maqi/check_confirmed_spend_existence_wi…
RolandSherwin Aug 7, 2024
7eb71d3
fix(metrics): update current wallet balance after forwarding rewards
RolandSherwin Aug 7, 2024
74af759
Merge pull request #2029 from RolandSherwin/metrics_update_current_wa…
RolandSherwin Aug 7, 2024
b99479e
refactor(network): remove some unused network events
RolandSherwin Aug 7, 2024
15e1def
feat(network): expose bandwidth per transport protocol
RolandSherwin Aug 7, 2024
edb5d3b
feat(networking): initial range based gets implementation
joshuef Jul 15, 2024
2f60aa6
chore(networking): rename sort_peers fns to clarify the accepted limi…
joshuef Jul 15, 2024
3a45bf6
chore(networking): remove REPLICATION_PEER_COUNT
joshuef Jul 25, 2024
5978f72
chore(transfers): cleanup DoubleSpendAttempt error string usage
joshuef Aug 7, 2024
5c85d95
chore(networking): update get accumulation to use get_request_range
joshuef Aug 7, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 8 additions & 1 deletion .github/workflows/merge.yml
Original file line number Diff line number Diff line change
Expand Up @@ -77,6 +77,9 @@ jobs:
shell: bash
run: if [[ ! $(cargo metadata --no-deps --format-version 1 | jq -r '.packages[].features.default[]? | select(. == "local-discovery")') ]]; then echo "local-discovery is not a default feature in any package."; else echo "local-discovery is a default feature in at least one package." && exit 1; fi

- name: Clean out the target directory
run: cargo clean

# In a cargo workspace, feature unification can occur, allowing a crate to be built successfully even if it
# doesn't explicitly specify a feature it uses, provided another crate in the workspace enables that feature.
# To detect such cases, we must build each crate using `--package` flag, building all packages at once does not work.
Expand Down Expand Up @@ -737,6 +740,10 @@ jobs:
echo "SAFE_PEERS has been set to $SAFE_PEERS"
fi

- name: Wait from network to stabilise
shell: bash
run: sleep 30

- name: Verify the routing tables of the nodes
run: cargo test --release -p sn_node --features="local-discovery" --test verify_routing_table -- --nocapture
env:
Expand All @@ -762,7 +769,7 @@ jobs:
uses: maidsafe/sn-local-testnet-action@main
with:
action: stop
log_file_prefix: safe_test_logs_data_location
log_file_prefix: safe_test_logs_data_location_routing_table
platform: ${{ matrix.os }}

- name: Verify restart of nodes using rg
Expand Down
3 changes: 3 additions & 0 deletions Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,9 @@ clone_on_ref_ptr = "warn"
debug = 0
strip = "debuginfo"

[profile.dev]
debug = 0
strip = "debuginfo"

[workspace.metadata.release]
pre-release-commit-message = "chore(release): release commit, tags, deps and changelog updates"
Expand Down
17 changes: 11 additions & 6 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -210,22 +210,27 @@ cargo run --bin safe --features local-discovery -- folders download <target dir

Use your local wallet to demonstrate sending tokens and receiving transfers.

First, get your wallet address:
First, get your wallet address, this address can be safely shared publicly

```
cargo run --bin safe -- wallet address
```

Now send some tokens to that address:
You can also get your balance with:

```
cargo run --bin safe --features local-discovery -- wallet send 2 [address]
cargo run --bin safe -- wallet balance
```

This will output a transfer as a hex string, which should be sent to the recipient out-of-band.
Now to send some tokens to an address:

```
cargo run --bin safe --features local-discovery -- wallet send 2 [address]
```

For demonstration purposes, copy the transfer string and use it to receive the transfer in your own
wallet:
This will output a transfer as a hex string, which should be sent to the recipient.
This transfer is encrypted to the recipient so only the recipient can read and redeem it.
To receive a transfer, simply paste it after the wallet receive command:

```
cargo run --bin safe --features local-discovery -- wallet receive [transfer]
Expand Down
4 changes: 2 additions & 2 deletions node-launchpad/src/components/home.rs
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,6 @@ use std::{
};
use tokio::sync::mpsc::UnboundedSender;

const NODE_START_INTERVAL: usize = 10;
const NODE_STAT_UPDATE_INTERVAL: Duration = Duration::from_secs(5);
/// If nat detection fails for more than 3 times, we don't want to waste time running during every node start.
const MAX_ERRORS_WHILE_RUNNING_NAT_DETECTION: usize = 3;
Expand Down Expand Up @@ -660,6 +659,7 @@ fn maintain_n_running_nodes(
if let Err(err) = sn_node_manager::cmd::node::maintain_n_running_nodes(
false,
true,
120,
count,
None,
true,
Expand All @@ -680,7 +680,7 @@ fn maintain_n_running_nodes(
None,
None,
VerbosityLevel::Minimal,
NODE_START_INTERVAL as u64,
None,
)
.await
{
Expand Down
9 changes: 9 additions & 0 deletions sn_cli/src/files/download.rs
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@ use indicatif::ProgressBar;
use walkdir::WalkDir;
use xor_name::XorName;

use crate::utils::duration_to_minute_seconds_miliseconds_string;
use sn_client::{
protocol::storage::{Chunk, ChunkAddress, RetryStrategy},
FilesApi, FilesDownload, FilesDownloadEvent,
Expand Down Expand Up @@ -97,6 +98,8 @@ pub async fn download_file(
batch_size: usize,
retry_strategy: RetryStrategy,
) {
let start_time = std::time::Instant::now();

let mut files_download = FilesDownload::new(files_api.clone())
.set_batch_size(batch_size)
.set_show_holders(show_holders)
Expand All @@ -110,6 +113,7 @@ pub async fn download_file(

let progress_handler = tokio::spawn(async move {
let mut progress_bar: Option<ProgressBar> = None;

// The loop is guaranteed to end, as the channel will be closed when the download completes or errors out.
while let Some(event) = download_events_rx.recv().await {
match event {
Expand Down Expand Up @@ -145,6 +149,7 @@ pub async fn download_file(
}
}
}

if let Some(progress_bar) = progress_bar {
progress_bar.finish_and_clear();
}
Expand All @@ -158,6 +163,8 @@ pub async fn download_file(
)
.await;

let duration = start_time.elapsed();

// await on the progress handler first as we want to clear the progress bar before printing things.
let _ = progress_handler.await;
match download_result {
Expand All @@ -170,6 +177,8 @@ pub async fn download_file(
"Saved {file_name:?} at {}",
downloaded_file_path.to_string_lossy()
);
let elapsed_time = duration_to_minute_seconds_miliseconds_string(duration);
println!("File downloaded in {elapsed_time}");
}
Err(error) => {
error!("Error downloading {file_name:?}: {error}");
Expand Down
13 changes: 4 additions & 9 deletions sn_cli/src/files/files_uploader.rs
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@
// permissions and limitations relating to use of the SAFE Network Software.

use super::get_progress_bar;
use crate::utils::duration_to_minute_seconds_string;
use crate::ChunkManager;
use bytes::Bytes;
use color_eyre::{eyre::eyre, Report, Result};
Expand Down Expand Up @@ -424,13 +425,7 @@ impl FilesUploadStatusNotifier for StdOutPrinter {
elapsed_time: Duration,
chunks_to_upload_len: usize,
) {
let elapsed_minutes = elapsed_time.as_secs() / 60;
let elapsed_seconds = elapsed_time.as_secs() % 60;
let elapsed = if elapsed_minutes > 0 {
format!("{elapsed_minutes} minutes {elapsed_seconds} seconds")
} else {
format!("{elapsed_seconds} seconds")
};
let elapsed = duration_to_minute_seconds_string(elapsed_time);

println!(
"Among {chunks_to_upload_len} chunks, found {} already existed in network, uploaded \
Expand Down Expand Up @@ -462,10 +457,10 @@ impl StdOutPrinter {
for (_, file_name, addr) in completed_files {
let hex_addr = addr.to_hex();
if let Some(file_name) = file_name.to_str() {
println!("\"{file_name}\" {hex_addr}");
println!("Uploaded \"{file_name}\" to address {hex_addr}");
info!("Uploaded {file_name} to {hex_addr}");
} else {
println!("\"{file_name:?}\" {hex_addr}");
println!("Uploaded \"{file_name:?}\" to address {hex_addr}");
info!("Uploaded {file_name:?} to {hex_addr}");
}
}
Expand Down
25 changes: 25 additions & 0 deletions sn_cli/src/utils.rs
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,32 @@
// KIND, either express or implied. Please review the Licences for the specific language governing
// permissions and limitations relating to use of the SAFE Network Software.

use std::time::Duration;

/// Returns whether a hex string is a valid secret key in hex format.
pub fn is_valid_key_hex(hex: &str) -> bool {
hex.len() == 64 && hex.chars().all(|c| c.is_ascii_hexdigit())
}

pub fn duration_to_minute_seconds_string(duration: Duration) -> String {
let elapsed_minutes = duration.as_secs() / 60;
let elapsed_seconds = duration.as_secs() % 60;
if elapsed_minutes > 0 {
format!("{elapsed_minutes} minutes {elapsed_seconds} seconds")
} else {
format!("{elapsed_seconds} seconds")
}
}

pub fn duration_to_minute_seconds_miliseconds_string(duration: Duration) -> String {
let elapsed_minutes = duration.as_secs() / 60;
let elapsed_seconds = duration.as_secs() % 60;
let elapsed_millis = duration.subsec_millis();
if elapsed_minutes > 0 {
format!("{elapsed_minutes} minutes {elapsed_seconds} seconds {elapsed_millis} milliseconds")
} else if elapsed_seconds > 0 {
format!("{elapsed_seconds} seconds {elapsed_millis} milliseconds")
} else {
format!("{elapsed_millis} milliseconds")
}
}
37 changes: 24 additions & 13 deletions sn_client/src/api.rs
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ use super::{
use bls::{PublicKey, SecretKey, Signature};
use libp2p::{
identity::Keypair,
kad::{Quorum, Record},
kad::{KBucketDistance, Quorum, Record},
Multiaddr, PeerId,
};
#[cfg(feature = "open-metrics")]
Expand All @@ -24,7 +24,7 @@ use sn_networking::{
get_signed_spend_from_record, multiaddr_is_global,
target_arch::{interval, spawn, timeout, Instant},
GetRecordCfg, GetRecordError, NetworkBuilder, NetworkError, NetworkEvent, PutRecordCfg,
VerificationKind, CLOSE_GROUP_SIZE,
VerificationKind,
};
use sn_protocol::{
error::Error as ProtocolError,
Expand All @@ -33,7 +33,7 @@ use sn_protocol::{
try_deserialize_record, try_serialize_record, Chunk, ChunkAddress, RecordHeader,
RecordKind, RegisterAddress, RetryStrategy, SpendAddress,
},
NetworkAddress, PrettyPrintRecordKey,
NetworkAddress, PrettyPrintRecordKey, CLOSE_GROUP_SIZE,
};
use sn_registers::{Permissions, SignedRegister};
use sn_transfers::{
Expand Down Expand Up @@ -307,6 +307,11 @@ impl Client {
self.events_broadcaster.subscribe()
}

/// Return the underlying network GetRange
pub async fn get_range(&self) -> Result<KBucketDistance> {
self.network.get_range().await.map_err(Error::from)
}

/// Sign the given data.
///
/// # Arguments
Expand Down Expand Up @@ -450,9 +455,9 @@ impl Client {
) -> Result<SignedRegister> {
let key = NetworkAddress::from_register_address(address).to_record_key();
let get_quorum = if is_verifying {
Quorum::Majority
Quorum::All
} else {
Quorum::One
Quorum::Majority
};
let retry_strategy = if is_verifying {
Some(RetryStrategy::Balanced)
Expand Down Expand Up @@ -889,17 +894,25 @@ impl Client {

// When there is retry on Put side, no need to have a retry on Get
let verification_cfg = GetRecordCfg {
get_quorum: Quorum::Majority,
get_quorum: Quorum::All,
retry_strategy: None,
target_record: record_to_verify,
expected_holders,
};

let verification = if verify_store {
Some((VerificationKind::Network, verification_cfg))
} else {
None
};

let put_cfg = PutRecordCfg {
put_quorum: Quorum::Majority,
put_quorum: Quorum::All,
retry_strategy: Some(RetryStrategy::Persistent),
use_put_record_to: None,
verification: Some((VerificationKind::Network, verification_cfg)),
verification,
};

Ok(self.network.put_record(record, &put_cfg).await?)
}

Expand Down Expand Up @@ -936,7 +949,7 @@ impl Client {
self.try_fetch_spend_from_network(
address,
GetRecordCfg {
get_quorum: Quorum::Majority,
get_quorum: Quorum::All,
retry_strategy: Some(RetryStrategy::Balanced),
target_record: None,
expected_holders: Default::default(),
Expand Down Expand Up @@ -967,7 +980,7 @@ impl Client {
self.try_fetch_spend_from_network(
address,
GetRecordCfg {
get_quorum: Quorum::Majority,
get_quorum: Quorum::All,
retry_strategy: None,
target_record: None,
expected_holders: Default::default(),
Expand Down Expand Up @@ -1023,9 +1036,7 @@ impl Client {
}
Err(err) => {
warn!("Invalid signed spend got from network for {address:?}: {err:?}.");
Err(Error::CouldNotVerifyTransfer(format!(
"Verification failed for spent at {address:?} with error {err:?}"
)))
Err(Error::from(err))
}
}
}
Expand Down
7 changes: 4 additions & 3 deletions sn_client/src/error.rs
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@ use crate::UploadSummary;
use super::ClientEvent;
use sn_protocol::NetworkAddress;
use sn_registers::{Entry, EntryHash};
use sn_transfers::SpendAddress;
use std::collections::BTreeSet;
use thiserror::Error;
use tokio::time::Duration;
Expand Down Expand Up @@ -46,6 +47,9 @@ pub enum Error {
#[error("Chunks error {0}.")]
Chunks(#[from] super::chunks::Error),

#[error("No cashnote found at {0:?}.")]
NoCashNoteFound(SpendAddress),

#[error("Decrypting a Folder's item failed: {0}")]
FolderEntryDecryption(EntryHash),

Expand All @@ -64,9 +68,6 @@ pub enum Error {
#[error(transparent)]
JoinError(#[from] tokio::task::JoinError),

/// A general error when verifying a transfer validity in the network.
#[error("Failed to verify transfer validity in the network {0}")]
CouldNotVerifyTransfer(String),
#[error("Invalid DAG")]
InvalidDag,
#[error("Serialization error: {0:?}")]
Expand Down
2 changes: 1 addition & 1 deletion sn_client/src/test_utils.rs
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ use tracing::{info, warn};
pub const AMOUNT_TO_FUND_WALLETS: u64 = 100 * 1_000_000_000;

// The number of times to try to load the faucet wallet
const LOAD_FAUCET_WALLET_RETRIES: usize = 6;
const LOAD_FAUCET_WALLET_RETRIES: usize = 10;

// mutex to restrict access to faucet wallet from concurrent tests
static FAUCET_WALLET_MUTEX: Mutex<()> = Mutex::const_new(());
Expand Down
Loading