Enhance Prometheus metrics

[arijitAD]

Issue summary

The metrics listed below outline all the metrics published in Substrate. Gossamer should use this list to track similar metrics. To view all Substrate metrics and view them in a Grafana dashboard follow the Visualizing Node Metrics tutorial.

Metrics published by Substrate

Authority node metrics

• substrate_authority_discovery_amount_external_addresses_last_published : Number of external addresses published when authority discovery last published addresses.
• substrate_authority_discovery_authority_address_requests_pending: Number of pending authority address requests.
• substrate_authority_discovery_authority_addresses_requested_total: Number of times authority discovery has requested external addresses of a single authority.
• substrate_authority_discovery_dht_event_received: Number of dht events received by authority discovery.
• substrate_authority_discovery_handle_value_found_event_failure: Number of times handling a dht value found event failed.
• substrate_authority_discovery_known_authorities_count: Number of authorities known by authority discovery.
• substrate_authority_discovery_times_published_total: Number of times authority discovery has published external addresses.

Block metrics

• substrate_block_height: Block height info of the chain
• substrate_block_verification_and_import_time: Time taken to verify and import blocks
• substrate_block_verification_and_import_time_bucket
• substrate_proposer_block_constructed_bucket{le="0.005"}: time taken to construct new block
• substrate_process_start_time_seconds: Number of seconds between the UNIX epoch and the moment the process started

Build info metrics

• substrate_build_info: A metric with a constant '1' value labeled by name, version
• substrate_node_roles: The roles the node is running as
• substrate_number_leaves: Number of known chain leaves (aka forks)
• substrate_sync_fork_targets: Number of fork sync targets

Database metrics

• substrate_database_cache_bytes: RocksDB cache size in bytes

Grandpa finality metrics

• substrate_finality_grandpa_precommits_total: Total number of GRANDPA precommits cast locally.
• substrate_finality_grandpa_prevotes_total: Total number of GRANDPA prevotes cast locally.

Grandpa Finality metrics

• substrate_finality_grandpa_round: Highest completed GRANDPA round.
• substrate_finality_grandpa_until_imported_waiting_messages_number: Number of finality grandpa messages waiting within the until imported queue.
• substrate_import_queue_blocks_submitted: Number of blocks submitted to the import queue.
• substrate_import_queue_justifications_submitted: Number of justifications submitted to the import queue.
• substrate_issued_light_requests: Number of light client requests that our node has issued.
• substrate_justification_import_time: Time taken to import justifications
• substrate_justification_import_time_bucket
• substrate_sync_extra_justifications: Number of extra justifications requests
• substrate_sync_extra_justifications{status="active"}

Networking metrics

• substrate_network_gossip_expired_messages_total: Number of expired messages by the gossip service.
• substrate_network_gossip_registered_messages_total: Number of registered messages by the gossip service.
• substrate_sub_libp2p_distinct_peers_connections_closed_total: Total number of connections closed with distinct peers
• substrate_sub_libp2p_distinct_peers_connections_opened_total: Total number of connections opened with distinct peers
• substrate_sub_libp2p_incoming_connections_total: Total number of incoming connections on the listening sockets
• substrate_sub_libp2p_is_major_syncing: Whether the node is performing a major sync or not.
• substrate_sub_libp2p_kademlia_query_duration: Duration of Kademlia queries per query type
• substrate_sub_libp2p_kademlia_query_duration_bucket{type="value-put-failed",le="0.5"}
• substrate_sub_libp2p_kademlia_random_queries_total: Number of random Kademlia queries started
• substrate_sub_libp2p_kademlia_records_count: Number of records in the Kademlia records store
• substrate_sub_libp2p_kademlia_records_sizes_total: Total size of all the records in the Kademlia records store
• substrate_sub_libp2p_listeners_errors_total: Total number of non-fatal errors reported by a listener
• substrate_sub_libp2p_listeners_local_addresses: Number of local addresses we're listening on
• substrate_sub_libp2p_network_bytes_total: Total bandwidth usage
• substrate_sub_libp2p_network_bytes_total{direction="in"}
• substrate_sub_libp2p_out_events_events_total: Number of broadcast network events that have been sent or received across all channels
• substrate_sub_libp2p_out_events_events_total{action="received",event_name="dht",name="authority-discovery"}
• substrate_sub_libp2p_out_events_num_channels: Number of internal active channels that broadcast network events
• substrate_sub_libp2p_out_events_num_channels{name="authority-discovery"}
• substrate_sub_libp2p_peers_count: Number of connected peers
• substrate_sub_libp2p_peerset_num_discovered: Number of nodes stored in the peerset manager
• substrate_sub_libp2p_peerset_num_requested: Number of nodes that the peerset manager wants us to be connected to
• substrate_sub_libp2p_pending_connections: Number of connections in the process of being established

Tx metrics

• substrate_proposer_number_of_transactions: Number of transactions included in block
• substrate_ready_transactions_number: Number of transactions in the ready queue
• substrate_sub_txpool_block_transactions_pruned: Total number of transactions that was requested to be pruned by block events
• substrate_sub_txpool_block_transactions_resubmitted: Total number of transactions that was requested to be resubmitted by block events
• substrate_sub_txpool_submitted_transactions: Total number of transactions submitted
• substrate_sub_txpool_validations_finished: Total number of transactions that finished validation
• substrate_sub_txpool_validations_invalid: Total number of transactions that were removed from the pool as invalid
• substrate_sub_txpool_validations_scheduled: Total number of transactions scheduled for validation

State metrics

• substrate_state_cache_bytes: State cache size in bytes
• substrate_state_db_cache_bytes: State DB cache in bytes
• substrate_state_db_cache_bytes{subtype="non_canonical"}

Sync metrics

• substrate_sync_peers: Number of peers we sync with
• substrate_sync_propagated_transactions: Number of transactions propagated to at least one peer
• substrate_sync_queued_blocks: Number of blocks in import queue

Other information and links

[EclesioMeloJunior]

@arijitAD could you explain me where I can get these metrics?