Integrate tracing #6339

ThreeHrSleep · 2024-09-02T05:02:56Z

Issue Addressed

Tracing Integration

reference

Proposed Changes

replace slog & log with tracing throughout the codebase
implement custom crit log
make relevant changes in the formatter
replace sloggers
re-write SSE logging components

Add retropgf funding (sigp#6324) Migrate from `ethereum-types` to `alloy-primitives` (sigp#6078) * Remove use of ethers_core::RlpStream * Merge branch 'unstable' of https://github.com/sigp/lighthouse into remove_use_of_ethers_core * Remove old code * Simplify keccak call * Remove unused package * Merge branch 'unstable' of https://github.com/ethDreamer/lighthouse into remove_use_of_ethers_core * Merge branch 'unstable' into remove_use_of_ethers_core * Run clippy * Merge branch 'remove_use_of_ethers_core' of https://github.com/dospore/lighthouse into remove_use_of_ethers_core * Check all cargo fmt * migrate to alloy primitives init * fix deps * integrate alloy-primitives * resolve dep issues * more changes based on dep changes * add TODOs * Merge branch 'unstable' of https://github.com/sigp/lighthouse into remove_use_of_ethers_core * Revert lock * Add BeaconBlocksByRange v3 * continue migration * Revert "Add BeaconBlocksByRange v3" This reverts commit e3ce7fc. * impl hash256 extended trait * revert some uneeded diffs * merge conflict resolved * fix subnet id rshift calc * rename to FixedBytesExtended * debugging * Merge branch 'unstable' of https://github.com/sigp/lighthouse into migrate-to-alloy-primitives * fix failed test * fixing more tests * Merge branch 'unstable' of https://github.com/sigp/lighthouse into remove_use_of_ethers_core * introduce a shim to convert between the two u256 types * move alloy to wrokspace * align alloy versions * update * update web3signer test certs * refactor * resolve failing tests * linting * fix graffiti string test * fmt * fix ef test * resolve merge conflicts * remove udep and revert cert * cargo patch * cyclic dep * fix build error * Merge branch 'unstable' of https://github.com/sigp/lighthouse into migrate-to-alloy-primitives * resolve conflicts, update deps * merge unstable * fmt * fix deps * Merge branch 'unstable' of https://github.com/sigp/lighthouse into migrate-to-alloy-primitives * resolve merge conflicts * resolve conflicts, make necessary changes * Remove patch * fmt * remove file * merge conflicts * sneaking in a smol change * bump versions * Merge remote-tracking branch 'origin/unstable' into migrate-to-alloy-primitives * Updates for peerDAS * Update ethereum_hashing to prevent dupe * updated alloy-consensus, removed TODOs * cargo update * endianess fix * Merge branch 'unstable' of https://github.com/sigp/lighthouse into migrate-to-alloy-primitives * fmt * fix merge * fix test * fixed_bytes crate * minor fixes * convert u256 to i64 * panic free mixin to_low_u64_le * from_str_radix * computbe_subnet api and ensuring we use big-endian * Merge branch 'unstable' of https://github.com/sigp/lighthouse into migrate-to-alloy-primitives * fix test * Simplify subnet_id test * Simplify some more tests * Add tests to fixed_bytes crate * Merge branch 'unstable' into migrate-to-alloy-primitives update libp2p to version 0.54 (sigp#6249) * update libp2p to version 0.54.0 * address review * Merge branch 'unstable' of github.com:sigp/lighthouse into update-libp2p * Merge branch 'update-libp2p' of github.com:sigp/lighthouse into update-libp2p Drop block data from BlockError and BlobError (sigp#5735) * Drop block data from BlockError and BlobError * Debug release tests * Fix release tests * Revert unnecessary changes to lighthouse_metrics Update manual checkpoint sync content in Lighthouse book (sigp#6338) * Update manual checkpiont sync * Update faq * Minor revision * Minor revision Light Client Bug Fix (sigp#6299) * Light Client Bug Fix Ignore Rust 1.82 warnings about void patterns (sigp#6357) * Ignore Rust 1.82 warnings about void patterns Enable `large_stack_frames` lint (sigp#6343) * Enable `large_stack_frames` lint Add blob count label to `DATA_COLUMN_SIDECAR_COMPUTATION` metric (sigp#6340) * Add blob count label to `DATA_COLUMN_SIDECAR_COMPUTATION` metric, and move metrics into the compute function, recording only successful computation. * Move `discard_timer_on_break` usage to caller site. * Merge branch 'unstable' into compute-data-column-metric * Merge branch 'unstable' into compute-data-column-metric Metadata request ordering (sigp#6336) * Send metadata request ordering * Merge branch 'unstable' into metadata-order Clarify validator monitor block log (sigp#6342) * Clarify validator monitor block log * Merge branch 'unstable' into clarify-block-log Check known parent on rpc blob process (sigp#5893) * Check known parent on rpc blob process * fix test * Merge branch 'unstable' of https://github.com/sigp/lighthouse into blob-unknown-parent Remove beta tag from gossipsub 1.2 (sigp#6344) * Remove the beta tag from gossipsub v1.2 * fix clippy * Merge branch 'unstable' into remove-beta-tag Fix lints for Rust 1.81 (sigp#6363) * Fix lints for Rust 1.81 Use tikv-jemallocator instead of jemallocator (sigp#6354) * Use tikv-jemallocator instead of jemallocator * Merge branch 'unstable' into tikv-jemallocator * Bump tikv-jemallocator and tikv-jemalloc-ctl Improve `get_custody_columns` validation, caching and error handling (sigp#6308) * Improve `get_custody_columns` validation, caching and error handling. * Merge branch 'unstable' into get-custody-columns-error-handing * Fix failing test and add more test. * Fix failing test and add more test. * Merge branch 'unstable' into get-custody-columns-error-handing * Add unit test to make sure the default specs won't panic on the `compute_custody_requirement_subnets` function. * Add condition when calling `compute_custody_subnets_from_metadata` and update logs. * Validate `csc` when returning from enr. Remove `csc` computation on connection since we get them on metadata anyway. * Add `peers_per_custody_subnet_count` to track peer csc and supernodes. * Disconnect peers with invalid metadata and find other peers instead. * Fix sampling tests. * Merge branch 'unstable' into get-custody-columns-error-handing * Merge branch 'unstable' into get-custody-columns-error-handing Delete legacy payload reconstruction (sigp#6213) * Delete legacy payload reconstruction * Delete unneeded failing test * Merge remote-tracking branch 'origin/unstable' into remove-more-ethers * Merge remote-tracking branch 'origin/unstable' into remove-more-ethers * Cleanups simplify rpc codec logic (sigp#6304) * simplify rpc codec logic * Merge branch 'unstable' of github.com:sigp/lighthouse into simplify-rpc-codec * Merge branch 'unstable' of github.com:sigp/lighthouse into simplify-rpc-codec * Merge branch 'unstable' of github.com:sigp/lighthouse into simply-rpc-codec * Merge branch 'unstable' into simplify-rpc-codec * Merge branch 'unstable' into simplify-rpc-codec

Revert "Merge branch 'sigp:stable' into tracing-integration" This reverts commit 668157c, reversing changes made to ee25fbe.

eserilev · 2024-10-01T17:44:58Z

Tracing

Slog

eserilev · 2024-10-01T19:40:31Z

at the moment this is what a crit log looks like

its labeled as ERROR but with an error_type = crit. Im wondering if we can tweak the formatting here to display CRIT in purple instead of ERROR (as we have currently for our slog impl)

eserilev

Great work so far!

Most of my suggested changes revolve around making sure we stick to display formatting vs debug formatting.

I think we have some options to simplify the crit! macro a bit

For crit level errors we'll want to display the Crit log similarly to how they are currently displayed with slog. This will involve implementing a custom tracing layer that takes in those crit logs and formats them appropriately.

In situations where we use slog:o! I think we can replace those with a tracing span

Theres some instances where we are still importing slog, we should try removing those if possible

beacon_node/beacon_chain/src/builder.rs

beacon_node/beacon_chain/src/light_client_server_cache.rs

beacon_node/beacon_chain/src/migrate.rs

beacon_node/beacon_chain/src/otb_verification_service.rs

beacon_node/beacon_chain/src/state_advance_timer.rs

validator_client/src/sync_committee_service.rs

validator_client/src/validator_store.rs

eserilev

I think this PR should be ready for another round of reviews!

beacon_node/beacon_chain/src/beacon_block_streamer.rs

beacon_node/beacon_chain/src/builder.rs

beacon_node/beacon_chain/src/data_availability_checker/overflow_lru_cache.rs

beacon_node/beacon_chain/src/fetch_blobs.rs

CLAassistant · 2024-12-02T10:24:58Z

All committers have signed the CLA.

Merge unstable into anchor

mergify · 2025-01-08T23:53:03Z

⚠️ The sha of the head commit of this PR conflicts with #6772. Mergify cannot evaluate rules on this PR. ⚠️

macladson

About halfway through now. I've left a question but mostly has been minor formatting issues where cargo fmt isn't doing its job.

I went ahead and pushed a commit to fix what I found, I figured it was faster that way

beacon_node/beacon_chain/src/beacon_block_streamer.rs

macladson · 2025-01-12T08:27:07Z

beacon_node/lighthouse_network/src/rpc/mod.rs

-        let log = self
-            .log
-            .new(slog::o!("peer_id" => peer_id.to_string(), "connection_id" => connection_id.to_string()));
+


Are we losing logging verbosity here if we aren't using connection_id or peer_id anymore?

macladson · 2025-01-12T08:27:42Z

beacon_node/lighthouse_network/src/rpc/mod.rs

@@ -306,8 +334,8 @@ where

    fn handle_established_outbound_connection(
        &mut self,
-        connection_id: ConnectionId,
-        peer_id: PeerId,
+        _connection_id: ConnectionId,


Same thing here

ThreeHrSleep marked this pull request as draft September 2, 2024 05:04

chong-he added work-in-progress PR is a work-in-progress UX-and-logs labels Sep 3, 2024

ThreeHrSleep changed the title ~~feat: tracing integration~~ feat: Integrate tracing Sep 4, 2024

ThreeHrSleep force-pushed the tracing-integration branch 2 times, most recently from 23491cd to 4b63358 Compare September 9, 2024 11:07

ThreeHrSleep added 17 commits September 22, 2024 07:47

add tracing to beacon_node

f59246e

add tracing to common

c5ab874

add tracing to consensus

2b2577b

add tracing to database_manager

131e939

add tracing to lcli

a3c4a4d

add tracing to lighthouse

9749a5e

add tracing to slasher

18b7a6a

add tracing to validator_client

80ee046

add tracing to beacon_node

f558244

add tracing to beacon_node/lighthouse_network

52e0471

add tracing to beacon_node/network

c361e0e

add tracing to boot_node

eaf3174

nit

497e04a

Revert "Merge branch 'sigp:stable' into tracing-integration" This reverts commit 668157c, reversing changes made to ee25fbe.

suggested changes from last review

df09d4e

remove redundant loggers (partial)

8d7a009

remove more loggers

49745c0

ThreeHrSleep force-pushed the tracing-integration branch from 4b63358 to 49745c0 Compare September 22, 2024 02:45

ThreeHrSleep and others added 3 commits September 22, 2024 08:26

minor fixes + cargo fmt

4aebccb

remove junk

4ca9e28

Tracing executor and metrics rename

73743a5

eserilev requested changes Oct 1, 2024

View reviewed changes

eserilev reviewed Dec 1, 2024

View reviewed changes

suggested changes

f7016f2

ThreeHrSleep added 2 commits December 2, 2024 16:10

conflicts

066a9e9

nit

8ff6dc4

michaelsproul added v6.1.0 New release c. Q1 2025 backwards-incompat Backwards-incompatible API change labels Dec 2, 2024

michaelsproul mentioned this pull request Dec 13, 2024

Re-implement short format for block roots/hashes #6689

Open

ThreeHrSleep added 2 commits December 16, 2024 08:46

Merge branch 'unstable' into tracing-integration

16a1a60

merge conflicts

75150aa

dknopik mentioned this pull request Dec 16, 2024

Modularize validator store #6705

Draft

ThreeHrSleep added 6 commits December 17, 2024 11:03

resolve more merge conflicts

ddf80ee

remove slog from some tests

54aafb8

moreeee conflicts

5d10bd9

clean up

7dbde61

Merge branch 'unstable' into tracing-integration

bc6bdf8

Merge branch 'unstable' into tracing-integration

9f4772f

mergify bot mentioned this pull request Jan 8, 2025

Anchor pre-PR: Tracing integration #6772

Closed

jimmygchen added 2 commits January 9, 2025 10:47

Merge pull request sigp#6768 from sigp/unstable

3208430

Merge unstable into anchor

Merge branch 'anchor' into tracing-integration

b9aa8fb

jimmygchen requested a review from jxs as a code owner January 8, 2025 23:52

ThreeHrSleep and others added 4 commits January 10, 2025 23:02

Merge branch 'unstable' into tracing-integration

843c675

fix compilation errors

103c944

cargo sort

ac51378

Minor formatting

e87898d

macladson reviewed Jan 12, 2025

View reviewed changes

macladson added 3 commits January 13, 2025 15:48

Fix test compliation errors

b536036

make cli

7efe331

Remove unused dependencies

84fc3ac

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Integrate tracing #6339

Integrate tracing #6339

ThreeHrSleep commented Sep 2, 2024 •

edited

Loading

eserilev commented Oct 1, 2024

eserilev commented Oct 1, 2024

eserilev left a comment •

edited

Loading

eserilev left a comment

CLAassistant commented Dec 2, 2024 •

edited

Loading

mergify bot commented Jan 8, 2025

macladson left a comment •

edited

Loading

macladson Jan 12, 2025

macladson Jan 12, 2025

Integrate tracing #6339

Are you sure you want to change the base?

Integrate tracing #6339

Conversation

ThreeHrSleep commented Sep 2, 2024 • edited Loading

Issue Addressed

Proposed Changes

eserilev commented Oct 1, 2024

eserilev commented Oct 1, 2024

eserilev left a comment • edited Loading

Choose a reason for hiding this comment

eserilev left a comment

Choose a reason for hiding this comment

CLAassistant commented Dec 2, 2024 • edited Loading

mergify bot commented Jan 8, 2025

macladson left a comment • edited Loading

Choose a reason for hiding this comment

macladson Jan 12, 2025

Choose a reason for hiding this comment

macladson Jan 12, 2025

Choose a reason for hiding this comment

ThreeHrSleep commented Sep 2, 2024 •

edited

Loading

eserilev left a comment •

edited

Loading

CLAassistant commented Dec 2, 2024 •

edited

Loading

macladson left a comment •

edited

Loading