feat(node): file-based observability configuration #1078

karlem · 2024-07-17T21:17:33Z

Close #1071

raulk

Could you post an example of what the traces look like with the console and file outputs?

ipc/observability/src/lib.rs

ipc/observability/src/traces_settings.rs

ipc/observability/src/traces.rs

raulk

Just some nits.

fendermint/app/src/cmd/rpc.rs

fendermint/vm/topdown/src/proxy.rs

karlem · 2024-07-22T16:27:18Z

Output from file:

{"timestamp":"2024-07-22T16:23:58.163639Z","level":"INFO","fields":{"message":"Connecting to Tendermint at http://127.0.0.1:26657/"},"filename":"fendermint/app/src/cmd/run.rs","line_number":70}
{"timestamp":"2024-07-22T16:23:58.164816Z","level":"DEBUG","fields":{"message":"validator key not configured"},"filename":"fendermint/app/src/cmd/run.rs","line_number":104}
{"timestamp":"2024-07-22T16:23:58.166078Z","level":"INFO","fields":{"message":"opening database","path":"/Users/karlem/.fendermint/data/rocksdb"},"filename":"fendermint/app/src/cmd/run.rs","line_number":395}
{"timestamp":"2024-07-22T16:23:58.212059Z","level":"INFO","fields":{"message":"IPLD Resolver disabled."},"filename":"fendermint/app/src/cmd/run.rs","line_number":241}
{"timestamp":"2024-07-22T16:23:58.212092Z","level":"INFO","fields":{"message":"topdown finality disabled"},"filename":"fendermint/app/src/cmd/run.rs","line_number":272}
{"timestamp":"2024-07-22T16:23:58.212098Z","level":"INFO","fields":{"message":"snapshots disabled"},"filename":"fendermint/app/src/cmd/run.rs","line_number":298}
{"timestamp":"2024-07-22T16:23:58.212450Z","level":"INFO","fields":{"message":"serving metrics","listen_addr":"127.0.0.1:9184"},"filename":"fendermint/app/src/cmd/run.rs","line_number":343}
{"timestamp":"2024-07-22T16:23:58.213254Z","level":"INFO","fields":{"message":"starting ABCI server","addr":"\"127.0.0.1:26658\""},"filename":"/Users/karlem/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tower-abci-0.7.0/src/v037/server.rs","line_number":127}
{"timestamp":"2024-07-22T16:23:58.213260Z","level":"TRACE","fields":{"message":"worker polling for next message"},"filename":"/Users/karlem/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tower-0.4.13/src/buffer/worker.rs","line_number":108}
{"timestamp":"2024-07-22T16:23:58.213302Z","level":"INFO","fields":{"message":"bound tcp listener","local_addr":"127.0.0.1:26658"},"filename":"/Users/karlem/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tower-abci-0.7.0/src/v037/server.rs","line_number":130}

Output from console:

2024-07-22T16:23:58.154879Z  INFO fendermint/app/src/cmd/mod.rs:88: reading configuration path="/Users/karlem/.fendermint/config"
2024-07-22T16:23:58.163563Z  INFO fendermint/app/src/cmd/run.rs:70: Connecting to Tendermint at http://127.0.0.1:26657/
2024-07-22T16:23:58.164808Z DEBUG fendermint/app/src/cmd/run.rs:104: validator key not configured
2024-07-22T16:23:58.166069Z  INFO fendermint/app/src/cmd/run.rs:395: opening database path="/Users/karlem/.fendermint/data/rocksdb"
2024-07-22T16:23:58.212041Z  INFO fendermint/app/src/cmd/run.rs:241: IPLD Resolver disabled.
2024-07-22T16:23:58.212089Z  INFO fendermint/app/src/cmd/run.rs:272: topdown finality disabled
2024-07-22T16:23:58.212096Z  INFO fendermint/app/src/cmd/run.rs:298: snapshots disabled
2024-07-22T16:23:58.212444Z  INFO fendermint/app/src/cmd/run.rs:343: serving metrics listen_addr="127.0.0.1:9184"
2024-07-22T16:23:58.213176Z  INFO /Users/karlem/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tower-abci-0.7.0/src/v037/server.rs:127: starting ABCI server addr="127.0.0.1:26658"
2024-07-22T16:23:58.213219Z TRACE /Users/karlem/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tower-0.4.13/src/buffer/worker.rs:108: worker polling for next message
2024-07-22T16:23:58.213297Z  INFO /Users/karlem/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tower-abci-0.7.0/src/v037/server.rs:130: bound tcp listener local_addr=127.0.0.1:26658

raulk · 2024-07-22T16:28:43Z

@karlem do we have logs/outputs showing journal event traces? 🙏

raulk · 2024-07-22T16:43:37Z

A metrics dump would also be very welcome, to ensure everything is working as expected 🙏

karlem · 2024-07-22T16:57:39Z

Metrics

# HELP bottomup_checkpoint_created_confignum Configuration number of the checkpoint created
# TYPE bottomup_checkpoint_created_confignum gauge
bottomup_checkpoint_created_confignum 0
# HELP bottomup_checkpoint_created_height Height of the checkpoint created
# TYPE bottomup_checkpoint_created_height gauge
bottomup_checkpoint_created_height 0
# HELP bottomup_checkpoint_created_msgcount Number of messages in the checkpoint created
# TYPE bottomup_checkpoint_created_msgcount gauge
bottomup_checkpoint_created_msgcount 0
# HELP bottomup_checkpoint_created_total Bottom-up checkpoint produced
# TYPE bottomup_checkpoint_created_total counter
bottomup_checkpoint_created_total 0
# HELP bottomup_checkpoint_finalized_height Height of the checkpoint finalized
# TYPE bottomup_checkpoint_finalized_height gauge
bottomup_checkpoint_finalized_height 0
# HELP exec_fvm_apply_execution_time_secs Execution time of FVM apply in seconds
# TYPE exec_fvm_apply_execution_time_secs histogram
exec_fvm_apply_execution_time_secs_bucket{le="0.005"} 0
exec_fvm_apply_execution_time_secs_bucket{le="0.01"} 0
exec_fvm_apply_execution_time_secs_bucket{le="0.025"} 0
exec_fvm_apply_execution_time_secs_bucket{le="0.05"} 0
exec_fvm_apply_execution_time_secs_bucket{le="0.1"} 0
exec_fvm_apply_execution_time_secs_bucket{le="0.25"} 0
exec_fvm_apply_execution_time_secs_bucket{le="0.5"} 0
exec_fvm_apply_execution_time_secs_bucket{le="1"} 0
exec_fvm_apply_execution_time_secs_bucket{le="2.5"} 0
exec_fvm_apply_execution_time_secs_bucket{le="5"} 0
exec_fvm_apply_execution_time_secs_bucket{le="10"} 0
exec_fvm_apply_execution_time_secs_bucket{le="+Inf"} 0
exec_fvm_apply_execution_time_secs_sum 0
exec_fvm_apply_execution_time_secs_count 0
# HELP exec_fvm_call_execution_time_secs Execution time of FVM call in seconds
# TYPE exec_fvm_call_execution_time_secs histogram
exec_fvm_call_execution_time_secs_bucket{le="0.005"} 0
exec_fvm_call_execution_time_secs_bucket{le="0.01"} 0
exec_fvm_call_execution_time_secs_bucket{le="0.025"} 0
exec_fvm_call_execution_time_secs_bucket{le="0.05"} 0
exec_fvm_call_execution_time_secs_bucket{le="0.1"} 0
exec_fvm_call_execution_time_secs_bucket{le="0.25"} 0
exec_fvm_call_execution_time_secs_bucket{le="0.5"} 0
exec_fvm_call_execution_time_secs_bucket{le="1"} 0
exec_fvm_call_execution_time_secs_bucket{le="2.5"} 0
exec_fvm_call_execution_time_secs_bucket{le="5"} 0
exec_fvm_call_execution_time_secs_bucket{le="10"} 0
exec_fvm_call_execution_time_secs_bucket{le="+Inf"} 0
exec_fvm_call_execution_time_secs_sum 0
exec_fvm_call_execution_time_secs_count 0
# HELP exec_fvm_check_execution_time_secs Execution time of FVM check in seconds
# TYPE exec_fvm_check_execution_time_secs histogram
exec_fvm_check_execution_time_secs_bucket{le="0.005"} 0
exec_fvm_check_execution_time_secs_bucket{le="0.01"} 0
exec_fvm_check_execution_time_secs_bucket{le="0.025"} 0
exec_fvm_check_execution_time_secs_bucket{le="0.05"} 0
exec_fvm_check_execution_time_secs_bucket{le="0.1"} 0
exec_fvm_check_execution_time_secs_bucket{le="0.25"} 0
exec_fvm_check_execution_time_secs_bucket{le="0.5"} 0
exec_fvm_check_execution_time_secs_bucket{le="1"} 0
exec_fvm_check_execution_time_secs_bucket{le="2.5"} 0
exec_fvm_check_execution_time_secs_bucket{le="5"} 0
exec_fvm_check_execution_time_secs_bucket{le="10"} 0
exec_fvm_check_execution_time_secs_bucket{le="+Inf"} 0
exec_fvm_check_execution_time_secs_sum 0
exec_fvm_check_execution_time_secs_count 0
# HELP exec_fvm_estimate_execution_time_secs Execution time of FVM estimate in seconds
# TYPE exec_fvm_estimate_execution_time_secs histogram
exec_fvm_estimate_execution_time_secs_bucket{le="0.005"} 0
exec_fvm_estimate_execution_time_secs_bucket{le="0.01"} 0
exec_fvm_estimate_execution_time_secs_bucket{le="0.025"} 0
exec_fvm_estimate_execution_time_secs_bucket{le="0.05"} 0
exec_fvm_estimate_execution_time_secs_bucket{le="0.1"} 0
exec_fvm_estimate_execution_time_secs_bucket{le="0.25"} 0
exec_fvm_estimate_execution_time_secs_bucket{le="0.5"} 0
exec_fvm_estimate_execution_time_secs_bucket{le="1"} 0
exec_fvm_estimate_execution_time_secs_bucket{le="2.5"} 0
exec_fvm_estimate_execution_time_secs_bucket{le="5"} 0
exec_fvm_estimate_execution_time_secs_bucket{le="10"} 0
exec_fvm_estimate_execution_time_secs_bucket{le="+Inf"} 0
exec_fvm_estimate_execution_time_secs_sum 0
exec_fvm_estimate_execution_time_secs_count 0
# HELP proposals_block_committed Block committed
# TYPE proposals_block_committed counter
proposals_block_committed{height="1"} 1
proposals_block_committed{height="10"} 1
proposals_block_committed{height="11"} 1
proposals_block_committed{height="12"} 1
proposals_block_committed{height="13"} 1
proposals_block_committed{height="14"} 1
proposals_block_committed{height="15"} 1
proposals_block_committed{height="16"} 1
proposals_block_committed{height="17"} 1
proposals_block_committed{height="18"} 1
proposals_block_committed{height="19"} 1
proposals_block_committed{height="2"} 1
proposals_block_committed{height="20"} 1
proposals_block_committed{height="21"} 1
proposals_block_committed{height="22"} 1
# HELP proposals_block_proposal_accepted Block proposal accepted
# TYPE proposals_block_proposal_accepted counter

proposals_block_proposal_accepted{height="55"} 1
proposals_block_proposal_accepted{height="56"} 1
proposals_block_proposal_accepted{height="57"} 1
proposals_block_proposal_accepted{height="58"} 1
proposals_block_proposal_accepted{height="59"} 1
proposals_block_proposal_accepted{height="6"} 1
proposals_block_proposal_accepted{height="60"} 1
proposals_block_proposal_accepted{height="61"} 1
proposals_block_proposal_accepted{height="62"} 1
proposals_block_proposal_accepted{height="63"} 1
proposals_block_proposal_accepted{height="64"} 1
proposals_block_proposal_accepted{height="65"} 1
proposals_block_proposal_accepted{height="66"} 1
proposals_block_proposal_accepted{height="67"} 1
proposals_block_proposal_accepted{height="68"} 1
proposals_block_proposal_accepted{height="69"} 1
proposals_block_proposal_accepted{height="7"} 1
proposals_block_proposal_accepted{height="70"} 1
proposals_block_proposal_accepted{height="71"} 1
proposals_block_proposal_accepted{height="72"} 1
proposals_block_proposal_accepted{height="73"} 1
proposals_block_proposal_accepted{height="74"} 1
proposals_block_proposal_accepted{height="75"} 1
proposals_block_proposal_accepted{height="76"} 1
proposals_block_proposal_accepted{height="77"} 1
proposals_block_proposal_accepted{height="78"} 1
proposals_block_proposal_accepted{height="79"} 1
proposals_block_proposal_accepted{height="8"} 1
proposals_block_proposal_accepted{height="80"} 1
proposals_block_proposal_accepted{height="81"} 1
proposals_block_proposal_accepted{height="82"} 1
proposals_block_proposal_accepted{height="9"} 1
# HELP proposals_block_proposal_received Block proposal received
# TYPE proposals_block_proposal_received counter

proposals_block_proposal_received{height="70"} 1
proposals_block_proposal_received{height="71"} 1
proposals_block_proposal_received{height="72"} 1
proposals_block_proposal_received{height="73"} 1
proposals_block_proposal_received{height="74"} 1
proposals_block_proposal_received{height="75"} 1
proposals_block_proposal_received{height="76"} 1
proposals_block_proposal_received{height="77"} 1
proposals_block_proposal_received{height="78"} 1
proposals_block_proposal_received{height="79"} 1
proposals_block_proposal_received{height="8"} 1
proposals_block_proposal_received{height="80"} 1
proposals_block_proposal_received{height="81"} 1
proposals_block_proposal_received{height="82"} 1
proposals_block_proposal_received{height="9"} 1
# HELP proposals_block_proposal_sent Block proposal sent
# TYPE proposals_block_proposal_sent counter
proposals_block_proposal_sent{height="61"} 1
proposals_block_proposal_sent{height="62"} 1
proposals_block_proposal_sent{height="63"} 1
proposals_block_proposal_sent{height="64"} 1
proposals_block_proposal_sent{height="65"} 1
proposals_block_proposal_sent{height="66"} 1
proposals_block_proposal_sent{height="67"} 1
proposals_block_proposal_sent{height="68"} 1
proposals_block_proposal_sent{height="69"} 1
proposals_block_proposal_sent{height="7"} 1
proposals_block_proposal_sent{height="70"} 1
proposals_block_proposal_sent{height="71"} 1
proposals_block_proposal_sent{height="72"} 1
proposals_block_proposal_sent{height="73"} 1
proposals_block_proposal_sent{height="74"} 1
proposals_block_proposal_sent{height="75"} 1
proposals_block_proposal_sent{height="76"} 1
proposals_block_proposal_sent{height="77"} 1
proposals_block_proposal_sent{height="78"} 1
proposals_block_proposal_sent{height="79"} 1
proposals_block_proposal_sent{height="8"} 1
proposals_block_proposal_sent{height="80"} 1
proposals_block_proposal_sent{height="81"} 1
proposals_block_proposal_sent{height="82"} 1
proposals_block_proposal_sent{height="9"} 1
# HELP topdown_parent_finality_committed_height Parent finality committed on chain
# TYPE topdown_parent_finality_committed_height gauge
topdown_parent_finality_committed_height 0
# HELP topdown_parent_finality_voting_latest_sent_height Parent finality peer
# TYPE topdown_parent_finality_voting_latest_sent_height gauge
topdown_parent_finality_voting_latest_sent_height 0
# HELP topdown_parent_finality_voting_quorum_height Parent finality vote tally new agreement; recorded whenever the latest epoch with quorum
# TYPE topdown_parent_finality_voting_quorum_height gauge
topdown_parent_finality_voting_quorum_height 0
# HELP topdown_parent_finality_voting_quorum_weight Parent finality vote tally new agreement; recorded whenever the latest epoch with quorum
# TYPE topdown_parent_finality_voting_quorum_weight gauge
topdown_parent_finality_voting_quorum_weight 0

karlem · 2024-07-22T17:02:46Z

Some consensus related traces:

Console:

2024-07-22T16:58:48.490982Z  INFO ipc/observability/src/lib.rs:42: domain="Proposals" event=BlockProposalSent { validator: account::Id(44D6E9254B04951164143C6D688C2036FA593F41), height: 173, size: 0, tx_count: 0 }
2024-07-22T16:58:48.508255Z  INFO ipc/observability/src/lib.rs:42: domain="Proposals" event=BlockProposalReceived { height: 173, hash: f87b67959f9c2f4119cfa2921ad8d121f0c23d7aa57fe0e363cf42a4f42fb36c, size: 0, tx_count: 0, validator: account::Id(44D6E9254B04951164143C6D688C2036FA593F41) }
2024-07-22T16:58:48.508269Z  INFO ipc/observability/src/lib.rs:42: domain="Proposals" event=BlockProposalEvaluated { height: 173, hash: f87b67959f9c2f4119cfa2921ad8d121f0c23d7aa57fe0e363cf42a4f42fb36c, size: 0, tx_count: 0, validator: account::Id(44D6E9254B04951164143C6D688C2036FA593F41), accept: true, reason: None }
2024-07-22T16:58:48.544056Z  INFO ipc/observability/src/lib.rs:42: domain="Proposals" event=BlockCommitted { height: 173, app_hash: 0171a0e40220d36a47bfd38c81b0acd8a87a5d473875dc31cdbba81b5d3157b2ae4c009a81f4 }
2024-07-22T16:58:49.526968Z  INFO ipc/observability/src/lib.rs:42: domain="Proposals" event=BlockProposalSent { validator: account::Id(44D6E9254B04951164143C6D688C2036FA593F41), height: 174, size: 0, tx_count: 0 }
2024-07-22T16:58:49.547365Z  INFO ipc/observability/src/lib.rs:42: domain="Proposals" event=BlockProposalReceived { height: 174, hash: 466a6d4efbd7cd78553e493844c69a661c917d30a25dd56a42e3a02539c24014, size: 0, tx_count: 0, validator: account::Id(44D6E9254B04951164143C6D688C2036FA593F41) }
2024-07-22T16:58:49.547385Z  INFO ipc/observability/src/lib.rs:42: domain="Proposals" event=BlockProposalEvaluated { height: 174, hash: 466a6d4efbd7cd78553e493844c69a661c917d30a25dd56a42e3a02539c24014, size: 0, tx_count: 0, validator: account::Id(44D6E9254B04951164143C6D688C2036FA593F41), accept: true, reason: None }
2024-07-22T16:58:49.582128Z  INFO ipc/observability/src/lib.rs:42: domain="Proposals" event=BlockCommitted { height: 174, app_hash: 0171a0e402206e947f59c0f0746fd8dacd2d4c8eb30055233ce89f93bd68e86194e8d83a761d }
2024-07-22T16:58:50.565149Z  INFO ipc/observability/src/lib.rs:42: domain="Proposals" event=BlockProposalSent { validator: account::Id(44D6E9254B04951164143C6D688C2036FA593F41), height: 175, size: 0, tx_count: 0 }
2024-07-22T16:58:50.582157Z  INFO ipc/observability/src/lib.rs:42: domain="Proposals" event=BlockProposalReceived { height: 175, hash: 4696b5a37bf003d1f4468059bf96c34a7d79a0c93203601aadadf921c17f782a, size: 0, tx_count: 0, validator: account::Id(44D6E9254B04951164143C6D688C2036FA593F41) }
2024-07-22T16:58:50.582189Z  INFO ipc/observability/src/lib.rs:42: domain="Proposals" event=BlockProposalEvaluated { height: 175, hash: 4696b5a37bf003d1f4468059bf96c34a7d79a0c93203601aadadf921c17f782a, size: 0, tx_count: 0, validator: account::Id(44D6E9254B04951164143C6D688C2036FA593F41), accept: true, reason: None }
2024-07-22T16:58:50.616325Z  INFO ipc/observability/src/lib.rs:42: domain="Proposals" event=BlockCommitted { height: 175, app_hash: 0171a0e40220093a9e7b9d7c6bb3e540710b560f6f5d9baccc51b5dc257be127c8926fc99f66 }

File:

{"timestamp":"2024-07-22T16:58:47.453943Z","level":"INFO","fields":{"domain":"Proposals","event":"BlockProposalSent { validator: account::Id(44D6E9254B04951164143C6D688C2036FA593F41), height: 172, size: 0, tx_count: 0 }"},"filename":"ipc/observability/src/lib.rs","line_number":42}
{"timestamp":"2024-07-22T16:58:47.473650Z","level":"INFO","fields":{"domain":"Proposals","event":"BlockProposalReceived { height: 172, hash: ac51519dbb9241bd40ff1ffea0cd89f2a0c97e123f47a26545e253e17fe85439, size: 0, tx_count: 0, validator: account::Id(44D6E9254B04951164143C6D688C2036FA593F41) }"},"filename":"ipc/observability/src/lib.rs","line_number":42}
{"timestamp":"2024-07-22T16:58:47.473673Z","level":"INFO","fields":{"domain":"Proposals","event":"BlockProposalEvaluated { height: 172, hash: ac51519dbb9241bd40ff1ffea0cd89f2a0c97e123f47a26545e253e17fe85439, size: 0, tx_count: 0, validator: account::Id(44D6E9254B04951164143C6D688C2036FA593F41), accept: true, reason: None }"},"filename":"ipc/observability/src/lib.rs","line_number":42}
{"timestamp":"2024-07-22T16:58:47.512034Z","level":"INFO","fields":{"domain":"Proposals","event":"BlockCommitted { height: 172, app_hash: 0171a0e40220ef7f37a1c3ea1c58f78fecc94f591551222c2a4bd9e3876346268e0f53a9d7b3 }"},"filename":"ipc/observability/src/lib.rs","line_number":42}
{"timestamp":"2024-07-22T16:58:48.490986Z","level":"INFO","fields":{"domain":"Proposals","event":"BlockProposalSent { validator: account::Id(44D6E9254B04951164143C6D688C2036FA593F41), height: 173, size: 0, tx_count: 0 }"},"filename":"ipc/observability/src/lib.rs","line_number":42}
{"timestamp":"2024-07-22T16:58:48.508260Z","level":"INFO","fields":{"domain":"Proposals","event":"BlockProposalReceived { height: 173, hash: f87b67959f9c2f4119cfa2921ad8d121f0c23d7aa57fe0e363cf42a4f42fb36c, size: 0, tx_count: 0, validator: account::Id(44D6E9254B04951164143C6D688C2036FA593F41) }"},"filename":"ipc/observability/src/lib.rs","line_number":42}
{"timestamp":"2024-07-22T16:58:48.508535Z","level":"INFO","fields":{"domain":"Proposals","event":"BlockProposalEvaluated { height: 173, hash: f87b67959f9c2f4119cfa2921ad8d121f0c23d7aa57fe0e363cf42a4f42fb36c, size: 0, tx_count: 0, validator: account::Id(44D6E9254B04951164143C6D688C2036FA593F41), accept: true, reason: None }"},"filename":"ipc/observability/src/lib.rs","line_number":42}
{"timestamp":"2024-07-22T16:58:48.544060Z","level":"INFO","fields":{"domain":"Proposals","event":"BlockCommitted { height: 173, app_hash: 0171a0e40220d36a47bfd38c81b0acd8a87a5d473875dc31cdbba81b5d3157b2ae4c009a81f4 }"},"filename":"ipc/observability/src/lib.rs","line_number":42}
{"timestamp":"2024-07-22T16:58:49.527006Z","level":"INFO","fields":{"domain":"Proposals","event":"BlockProposalSent { validator: account::Id(44D6E9254B04951164143C6D688C2036FA593F41), height: 174, size: 0, tx_count: 0 }"},"filename":"ipc/observability/src/lib.rs","line_number":42}
{"timestamp":"2024-07-22T16:58:49.547372Z","level":"INFO","fields":{"domain":"Proposals","event":"BlockProposalReceived { height: 174, hash: 466a6d4efbd7cd78553e493844c69a661c917d30a25dd56a42e3a02539c24014, size: 0, tx_count: 0, validator: account::Id(44D6E9254B04951164143C6D688C2036FA593F41) }"},"filename":"ipc/observability/src/lib.rs","line_number":42}
{"timestamp":"2024-07-22T16:58:49.547420Z","level":"INFO","fields":{"domain":"Proposals","event":"BlockProposalEvaluated { height: 174, hash: 466a6d4efbd7cd78553e493844c69a661c917d30a25dd56a42e3a02539c24014, size: 0, tx_count: 0, validator: account::Id(44D6E9254B04951164143C6D688C2036FA593F41), accept: true, reason: None }"},"filename":"ipc/observability/src/lib.rs","line_number":42}
{"timestamp":"2024-07-22T16:58:49.582133Z","level":"INFO","fields":{"domain":"Proposals","event":"BlockCommitted { height: 174, app_hash: 0171a0e402206e947f59c0f0746fd8dacd2d4c8eb30055233ce89f93bd68e86194e8d83a761d }"},"filename":"ipc/observability/src/lib.rs","line_number":42}
{"timestamp":"2024-07-22T16:58:50.565155Z","level":"INFO","fields":{"domain":"Proposals","event":"BlockProposalSent { validator: account::Id(44D6E9254B04951164143C6D688C2036FA593F41), height: 175, size: 0, tx_count: 0 }"},"filename":"ipc/observability/src/lib.rs","line_number":42}
{"timestamp":"2024-07-22T16:58:50.582163Z","level":"INFO","fields":{"domain":"Proposals","event":"BlockProposalReceived { height: 175, hash: 4696b5a37bf003d1f4468059bf96c34a7d79a0c93203601aadadf921c17f782a, size: 0, tx_count: 0, validator: account::Id(44D6E9254B04951164143C6D688C2036FA593F41) }"},"filename":"ipc/observability/src/lib.rs","line_number":42}
{"timestamp":"2024-07-22T16:58:50.582304Z","level":"INFO","fields":{"domain":"Proposals","event":"BlockProposalEvaluated { height: 175, hash: 4696b5a37bf003d1f4468059bf96c34a7d79a0c93203601aadadf921c17f782a, size: 0, tx_count: 0, validator: account::Id(44D6E9254B04951164143C6D688C2036FA593F41), accept: true, reason: None }"},"filename":"ipc/observability/src/lib.rs","line_number":42}
{"timestamp":"2024-07-22T16:58:50.616333Z","level":"INFO","fields":{"domain":"Proposals","event":"BlockCommitted { height: 175, app_hash: 0171a0e40220093a9e7b9d7c6bb3e540710b560f6f5d9baccc51b5dc257be127c8926fc99f66 }"},"filename":"ipc/observability/src/lib.rs","line_number":42}

karlem · 2024-07-22T17:11:01Z

@raulk Please see the output above. I have only set up a simple node without a connection to a parent, so it only includes consensus metrics/traces. Additionally, I took a random sample, so don't worry if they don't look coherent (for instance, they all refer to different block heights, etc.).

karlem changed the base branch from main to new-observability July 17, 2024 21:17

karlem mentioned this pull request Jul 18, 2024

feat(node): new observability architecture + events #1053

Merged

karlem force-pushed the new-observability-configuration branch from 77adbfc to 9f8f27a Compare July 18, 2024 22:21

raulk changed the title ~~New observability configuration~~ feat(node): file-based observability configuration Jul 19, 2024

karlem force-pushed the new-observability-configuration branch from 9f8f27a to 378d72c Compare July 19, 2024 13:41

Base automatically changed from new-observability to main July 19, 2024 13:41

raulk reviewed Jul 19, 2024

View reviewed changes

karlem and others added 23 commits July 22, 2024 15:45

feat: introduce metrics crate

ccde386

feat: improve library with macros, add top down events

6facf63

feat: emit metrics

1dcf517

feat: add remaining top down events

d34590d

feat: remove unused code & fix build

75e45e4

feat: address comments

c2465bc

feat: use latency wrapper instead

599bcc3

feat: add config filters

e80f3df

feat: add config as a cmd flags

79de523

feat: fix comments

2f40eef

feat: add consensus traces

a97b16d

feat: add execution metrics

f16f8c5

feat: add proposals metrics & reason

a164472

feat: add mempool event

1f5a38a

feat: remove

da84485

feat: address comments

c1ded52

feat: address comments

59f3ea7

feat: fix clippy

13bb9c6

minor cleanup.

d3b22d6

feat: fix typo

094e9b6

feat: standartize mpool trace

27e7e36

feat: address comments

e6e79f3

lint: clippy

01843f0

karlem added 5 commits July 22, 2024 16:01

feat: revert risky changes & small cleanup

6f67fe1

feat: use tracing config

4295027

feat: add tracing to each command

b77bc36

feat: introduce cleanups

edb7dc2

feat: adress comments

4e7d1c6

karlem force-pushed the new-observability-configuration branch from cfbeab6 to 4e7d1c6 Compare July 22, 2024 14:04

karlem added 2 commits July 22, 2024 16:25

feat: fix after rebase

ad1bd14

fix: rebase && comments

c544c46

karlem marked this pull request as ready for review July 22, 2024 15:59

raulk approved these changes Jul 22, 2024

View reviewed changes

fendermint/app/src/cmd/rpc.rs Outdated Show resolved Hide resolved

fendermint/vm/topdown/src/proxy.rs Outdated Show resolved Hide resolved

fix: select proper default

22a065a

feat: setup tracing in single place

68b2dbc

fix: do no register tracing twice in eth

73a4f4d

karlem merged commit c5e6b07 into main Jul 23, 2024
20 checks passed

karlem deleted the new-observability-configuration branch July 23, 2024 10:01

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(node): file-based observability configuration #1078

feat(node): file-based observability configuration #1078

karlem commented Jul 17, 2024 •

edited

Loading

raulk left a comment

raulk left a comment

karlem commented Jul 22, 2024

raulk commented Jul 22, 2024

raulk commented Jul 22, 2024

karlem commented Jul 22, 2024 •

edited

Loading

karlem commented Jul 22, 2024 •

edited

Loading

karlem commented Jul 22, 2024

feat(node): file-based observability configuration #1078

feat(node): file-based observability configuration #1078

Conversation

karlem commented Jul 17, 2024 • edited Loading

raulk left a comment

Choose a reason for hiding this comment

raulk left a comment

Choose a reason for hiding this comment

karlem commented Jul 22, 2024

raulk commented Jul 22, 2024

raulk commented Jul 22, 2024

karlem commented Jul 22, 2024 • edited Loading

karlem commented Jul 22, 2024 • edited Loading

karlem commented Jul 22, 2024

karlem commented Jul 17, 2024 •

edited

Loading

karlem commented Jul 22, 2024 •

edited

Loading

karlem commented Jul 22, 2024 •

edited

Loading