Database and Table Level Metrics

[breezewish]

Feature Request

Is your feature request related to a problem? Please describe:

Many customers use a single TiDB cluster to serve multiple & hybrid payloads (in different databases). Currently TiDB only supports metrics of the whole TiDB cluster or single TiDB instance. Database or table metrics are missing.

Describe the feature you'd like:

For critical metrics like QPS, latency, errors, a more detailed, per-table and per-database metrics is needed.

Describe alternatives you've considered:

Teachability, Documentation, Adoption, Migration Strategy:

The technical implementation needs to be further investigated. A possible solution can be using Prometheus labels. The corresponding label in memory metrics need to be deleted when table or database is deleted. Notice that there might be a lot of databases and tables and attaching metrics for each one may affect performance, so that it may not be suitable to adapt all metrics. Also histograms and multi-label metrics need to be very carefully considered, since they notably amplify the number of metrics. Another good idea can be allowing users to config what tables and databases are needed.

The new metrics should be added to the Grafana monitor.

Notice that TiDB already have a similar feature: #9151. However it may lead to memory leaks due to always keeping database names in the memory, as well as not work well when database numbers are huge. The new implementation could refine and improve it.

[zz-jason]

@breeswish Could you describe more about the use cases? Seems it's useful in a multi-tenant scenario?

[breezewish]

@zz-jason Simply speaking, yes. We already have many customers use a single TiDB cluster to serve multiple & hybrid payloads. These payloads are usually stayed in different databases. But anyway, they are on the same TiDB cluster or even same TiDB instance.

[jackysp]

emm... something like #9151 ?

[breezewish]

@jackysp Yes, similar to it. The current implementation in #9151 has memory leaks (when database is deleted) and database level metrics is usually not precise enough as well.

[jackysp]

The key point is that still there are performance issues when there are many databases. It seems like no one uses this feature so that many people don't even know about it.

[breezewish]

@jackysp I received real-world feature requests from our clients for this one, where they don't deploy multiple TiDB clusters :)

Yes, in addition to the memory leak issue, performance is another problem. I think we can simply let user configure what they want to collect, in order to not suffer from performance problems in default scenarios. It is netural that the more user wants to know, the more cost there will be. The important part is to let user decide.

[pepezzzz]

@breeswish Please help develop a simple one, end-user dba can oberseve the duration per database.
like #19360