Metrics dashboard for the TAGs themselves

[jberkus]

Starting a discussion from the Feb 6 TOC meeting. Several TOC members suggested that we should have TAG metrics the same as we have them for projects. This discussion is to resolve what metrics we would track, and how we would track them.

[raravena80]

Some metric ideas:

• Average meeting participants
• Yearly (or some cadence) actual scheduled meetings -- non canceled
• Number of monthly project reach outs (CNCF and non CNCF)
• Usual github type of metrics (Issues/PRs opened, closed)
• Active working group initiatives

(Figuring out an automated way to collect these may not be that trivial)

[jberkus]

Yeah, that's why I started a discussion. Even monitoring whitepaper work would be quite difficult, since most of that is activity in googledocs, which have no real API for stats.

[parispittman]

could you have all of the authors of the white paper sign as co-authors to the commit for the white paper? or have them all engage with the PR in some way, comments, et al?

[jberkus]

Here's more on my dream list of metrics:

• Activity stats for the TAG's own repo (easy)
• Activity stats of TAG/WG leads in other repos (harder)
• Projects assigned to the tag and their relative levels (soon to be easy)
• Project TOC activity (e.g. matriculation, etc.) for projects assigned to the TAG (e.g. how many projects under this TAG are applying for incubation?) (harder)
• How many open processes belong to this TAG (e.g. DDs in review) and how old are they?
• TAG and WG leads (easy), their terms in office (hard), and their relative levels of activity across the CNCF (very hard)
• Participation by assigned TOC liaisons in TAG activities (hard)
• TAG participation by non-leads (nearly impossible, too much of this activity is not on GH)

Sample user stories:

1. I'm a TAG chair. I would like to see who among one WG's leads are actually active running the WG. For those who are not active, I would like to see if the WG has active contributors who could be promoted to WG lead.

2. I'm the TOC chair. I'd like to see if it's time to split a TAG based on it having both too many projects and an adequate supply of active leads.

3. I'm a TOC liaison to a TAG. I'd like to see, at a glance, if the TAG is keeping up with it's "duties", or if some kind of intervention is needed.

[leonardpahlke]

We used this tool in Kubernetes sig-release to get an overview of the contributors. Probably this does not help too much here. Sharing this anyways. https://github.com/kubernetes-sigs/maintainers. A CLI like this may be helpful.

[leonardpahlke]

cc @tegioz @cynthia-sg (maintainers of the CNCF CLO projects like https://github.com/cncf/clomonitor)

[chira001]

We should not think of metrics as an end goal, but rather, what the benefit of the metrics are, and who consumes them.

For many of the TAGs, a simple 5 minute conversation between TOC and TAG chair may be able to make decisions for the sample user stories suggested above.

I'd love the metrics to tie into the actual roles that are being discussed and any functions that get delegated e.g. reviews, project updates etc ...

For the avoidance of doubt, I'm not against metrics, but I'd like their purpose defined up front, and tied to specific things we need, otherwise it is just more process for the sake of process.

[angellk]

When replying in email, it doesn't thread correctly - so copying my reply here:

Agree with Alex

Many time we have spreadsheets or other conversations where
discussions/decisions are being made that are difficult to track

Also, each TAG operates differently, so this would imply standardization
across TAGs in order for the metrics to 'make sense'

[jberkus]

@chira001 hence the user stories. The list of metrics above are metrics I'd like to have for my own TAG, because I would use them.

[riaankleinhans]

Much of these metrics are collected by LFX Community Management and help with a lot of the collection of this data.

@jberkus we can start implementing with TAG CS as soon as we are ready.
I will add it to this week's meeting agenda.

[jberkus]

Would be happy to use LFX if they can fix their data integrity problems. Does it have any ability to pull data from Zoom or GoogleDocs?

[riaankleinhans]

If a TAG move to LFX all the meetings get scheduledon the LFX calendar, which integrate with zoom and create a lfx.zoom link.
Meeting attendance based on zoom name, recording, transcription. Youtube automation is a BIG ask from the PMO team but is still in development. What do you have in mind with GoogleDoc integration?

This does however mean Zoom links would need to be updated everywhere. (CNCF Calendar, Meeting minutes, GH etc.)

[jberkus]

Also, last I checked, LFX calendar required all attendees to have LFX accounts ... which is a major blocker, since TAG meetings need to be open to the whole community.

[johnwalicki]

@jberkus Not necessarily. If you have an LFX account and are logged in, you'll automatically be joined to the Meeting. You can also optionally join with an email address or anonymously as a guest.

[angellk]

Agree with Alex Many time we have spreadsheets or other conversations where discussions/decisions are being made that are difficult to track Also, each TAG operates differently, so this would imply standardization across TAGs in order for the metrics to 'make sense'

…

On Sun, Mar 3, 2024 at 4:13 AM Alex Chircop ***@***.***> wrote: We should not think of metrics as an end goal, but rather, what the benefit of the metrics are, and who consumes them. For many of the TAGs, a simple 5 minute conversation between TOC and TAG chair may be able to make decisions for the sample user stories suggested above. I'd love the metrics to tie into the actual roles that are being discussed and any functions that get delegated e.g. reviews, project updates etc ... For the avoidance of doubt, I'm not against metrics, but I'd like their purpose defined up front, and tied to specific things we *need*, otherwise it is just more process for the sake of process. — Reply to this email directly, view it on GitHub <#1256 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAGMEV6NEMMZSB6ODRLM7PTYWMAWVAVCNFSM6AAAAABC4JZG2WVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4DMNJWHA4TE> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>