Skip to content

Commit

Permalink
Track certificates signed by each validator (#3038)
Browse files Browse the repository at this point in the history
## Motivation

We want to track validators that are potentially lagging behind (not signing as many blocks).

## Proposal

Add a Prometheus counter that counts for each validator, how many of the confirmed certificates we have processed they signed.
If we really want to keep an eye on this, we could even add an alert later on if the validator with the most signatures minus the validator with the least signatures is greater than some threshold.

## Test Plan

Did a local run and checked the metrics:

![Screenshot 2024-12-17 at 23.23.23.png](https://graphite-user-uploaded-assets-prod.s3.amazonaws.com/HlciHFAoHZW62zn13apJ/b9386742-d093-4cc9-808d-25683fe687d5.png)

## Release Plan

- These changes should be backported to the latest `testnet` branch
  • Loading branch information
ndr-ds authored Dec 18, 2024
1 parent adef7e0 commit 69268f8
Showing 1 changed file with 15 additions and 0 deletions.
15 changes: 15 additions & 0 deletions linera-core/src/worker.rs
Original file line number Diff line number Diff line change
Expand Up @@ -92,6 +92,15 @@ static NUM_BLOCKS: LazyLock<IntCounterVec> = LazyLock::new(|| {
register_int_counter_vec("num_blocks", "Number of blocks added to chains", &[])
});

#[cfg(with_metrics)]
static CERTIFICATES_SIGNED: LazyLock<IntCounterVec> = LazyLock::new(|| {
register_int_counter_vec(
"certificates_signed",
"Number of certificates signed by each validator",
&["validator_name"],
)
});

/// Instruct the networking layer to send cross-chain requests and/or push notifications.
#[derive(Default, Debug)]
pub struct NetworkActions {
Expand Down Expand Up @@ -847,6 +856,12 @@ where
.with_label_values(&[])
.inc_by(confirmed_transactions);
}

for (validator_name, _) in certificate.signatures() {
CERTIFICATES_SIGNED
.with_label_values(&[&validator_name.to_string()])
.inc();
}
}

self.process_confirmed_block(certificate, &blobs, notify_when_messages_are_delivered)
Expand Down

0 comments on commit 69268f8

Please sign in to comment.