Introduce metrics to track each Module Manifest reconcile performance

[ruanxin]

Description:

At the moment, LM has the internal implemented performance metrics for manifest CR reconcile, however, it lacks of detailed view to tell how is each Module individually performed, which makes the troubleshooting difficult if some problematic modules are introduced.

AC Updates

• Introduce Gauge metrics reconcile_duration_seconds with Manifest name as label to track each Manifest success reconciliation duration time

  • configure a duration threshold value (1mins), only add new metric above the threshold, also take care to delete existing metrics if duration time is lower than the threshold.

• Add a new metric diagram in the Plutono dashboard.