Charter review process

[mnot]

Right now the Process is pretty lax about how charters are initiated. Conceivably, a charter could be worked on for a week or two and then go to the AC without anyone outside a limited subset of the community being any wiser.

While in some aspects that's great, if we want broad review of charters to assure that there's strong consensus on them and that they align with values, we should make sure that they get broad review, and people who might have feedback have adequate chance to do so.

Some proposals:

• A new term, charter under consideration be introduced to the process. While a charter can be under development privately, once it's under consideration it is acknowledged that the W3C is considering adoption of the charter and commencing discussion and evaluation of it.
• Charters under consideration are announced to the entire community (e.g., using an e-mail list + RSS feed + web page), not just the AC.
• A community-wide venue be established for discussion of charters under consideration. Probably a GitHub repo, maybe a mailing list. Needs to be a community space, not a Team-oriented tracking mechanism like the strategy funnel.
• There is a minimum limit on the time a charter is under consideration baked into the process, to allow ample notice and time for discussion inside the W3C as well as with any external stakeholders. I'd suggest four weeks.
• Horizontal review be required by the Process, not just the Guide, and it be performed by designated CGs or IGs as appropriate, not just delegated to Team members.

[OR13]

@mnot thanks for raising this.

What are the required criteria for exiting, charter under consideration ?

I would hope to see a written acknowledgement that it had been reviewed, but that might not be realistic....

And then I can see folks still just ignoring the charter under consideration phase and waiting till the end of the process to object to deliverables, and then pointing back to the charter and saying, I never would have approved that.

This should be mitigated by the charter review process, at the very least, folks should be able to say, you / your organization was made aware, at this time, you had this many weeks to respond, you did not.

work items that are defined by charters that survive this period should have a higher bar for formal objections that are focused on the charter deliverables (since they have been reviewed, and should have been objected too earlier in the process), but I am not sure how or to what extent this can or should be formalized...

I am struggling to understand the relationship between revisions to the charters and formal objections process...

I have heard W3C is trending towards narrow charters, which should theoretically be easier to review, but that also means that objections of the form "we wish the charter was less narrow" need to be addressed in that context.

I think its excellent for us to raise the bar on chartering, the more rigorous we can make that process the less chance of wasted time, and higher the probability of better deliverables.

Are narrow charters a good or bad idea? how do we define a narrow charter?

For example, the web app sec wg seems to have a really large number of items, doesn't that raise the burden on getting a good review of the charter?... it also decreases the likely hood of passing through without objections...

[dwsinger]

I think having the Charters Under Consideration/Development "obviously visible" to the community, and especially the AC and the horizontal review groups and AB makes a lot of sense. I am not sure I want to add mandatory workload (e.g. getting HR signoffs); I feel we should try and see what visibility does for us, and allowing people to comment before it goes to the AC. The AC (and everyone, including team) would then be able to see "hang on, the i18n group has a major concern" and we could handle it.

This sounds eminently do-able using modern tooling.

[nigelmegitt]

If it encourages AC engagement in Charter development, it is probably a Good Thing.

[cwilso]

@OR13 I see a lot of analogies to the incubation process, with many of the same difficulties.

1. You can't demand review until the details are somewhat set; otherwise the review is inappropriate or just plain wrong
2. Once the details are somewhat set, there is resistance to change.

In short, we DO have this today - it's the charter review that the AC gets. "Less chance of wasted time" has to include "less chance of wasting the time of all the people we demand review from" too. I'm not against making changes here - just calling out that you have to minimize the time required from all reviewers, and you need to expect that there WILL be objections and redesigns needed along the way.

[wseltzer]

Inviting @w3c/strategy to take a look here. I'd be very willing to extend the scope and participation of the Funnel or to adopt new methods to make charter review more interactive.

[OR13]

@cwilso

In short, we DO have this today - it's the charter review that the AC gets.

My point was not to burden the AC with more work, it was to give them more than 1 chance to do that work well.

just calling out that you have to minimize the time required from all reviewers,

Yes, part of that might involve spreading reviews out, and giving more than 1 chance...

Imagine 3 opportunities to review the charter instead of 1....

If a wg asks 3 times for a review, and the reviewers still don't have time... they have still done the same amount of work as if they were asked once, but the chances of them having really bad weeks during all 3 asks is less....

Though some folks might never have enough time to review... at least giving them a few chances is better than 1 chance... especially if their review might prevent a WG from getting started with deliverables they could never do anything but object too.... WGs should not be capable of operating without knowing such objections are coming... since AC reviews seem like the only way to mitigate that, asking for more AC reviews up front (before charter / after charter) seems like the only defense against this...

Is there a less expensive solution?

[jeffjaffe]

Imagine 3 opportunities to review the charter instead of 1....

If a wg asks 3 times for a review, and the reviewers still don't have time... they have still done the same amount of work as if they were asked once, but the chances of them having really bad weeks during all 3 asks is less....

Though some folks might never have enough time to review... at least giving them a few chances is better than 1 chance... especially if their review might prevent a WG from getting started with deliverables they could never do anything but object too.... WGs should not be capable of operating without knowing such objections are coming... since AC reviews seem like the only way to mitigate that, asking for more AC reviews up front (before charter / after charter) seems like the only defense against this...

Is there a less expensive solution?

Without addressing Orie's ideas directly, I would like to point out:

• Charter reviews last for 4 weeks, generally. We hope that balances (a) the need to get started with new work and (b) the fact that AC reps can be busy. We hope that 4 weeks gives them time to fit in reviews for charters that are relevant for them.
• With Process 2021, we have a new facility where an AC rep may temporarily delegate AC rep privileges to someone else in their organization. That, too, can help with spreading the effort.

[torgo]

There's been sone discussion on AC-Forum on what role the TAG should or shouldn't play regarding charter development and review. Shall we fold that in here as well?

[torgo]

Actually I see this was already discussed in #328 ?

[tateshelby28]

👍

[frivoal]

A few more considerations:

• Once a charter-under-consideration moves to being proposed to the AC, the communication announcing it to the AC (mail, wbs poll intro…) needs to point to a disposition of comments of some sort. I don't strongly care about whether it's a manually crafted document or a properly labelled set of github issues, but it must be possible for any AC rep casting their ballot to observe:
  • that broad discussion and review have happened
  • that issues raised were addressed
  • whether the way issues were addressed were to the satisfaction of the people who raised them
• We should formally identify who's chairing this under-consideration phase. Particularly, it's important to tell the difference between those who are expected to express subjective opinions about should or shouldn't be written in the charter from those who're trying to asses whether we have consensus. Closing an issue against the wishes of the person who raised it may still happen, but it's very different whether that happens when a chair has determined that this was the only way to make progress while noting that doing so might cause a formal objection to be raised, versus when it's a proponent of the thing being proposed who wants to close because they like it the way it is.

Assuming we ensure the above, it may well be that the current set up of discussing charters in https://github.com/w3c/charter-drafts/ (and announcing that we do so) could largely fulfill the intent of @mnot 's proposal in the OC. Or maybe we do need more formality.

Either way, I do feel pretty strongly that we should make it trivially observable by the AC whether a charter that is being proposed is the result of extensive (and successful) consensus building, of bitter and unresolved discussions, or something few people have looked at.

[mnot]

Just checking in - is this something we consider as part of director-free (I can see it through that lens), or separate? Any chance of getting it into process-2022?

[frivoal]

I'd consider it fuzzily related to director-free. We could improve this while still having a director, and we could switch away from having a director without fixing this. But it's a thing that the Director used to do that would be good to improve into something that no longer involves him, so yeah, it's mostly part of director-free. As for landing it in process-2022, I'd say we can, if we manage to find consensus on exactly what it is we want to do.

I see no reason to tie it to other parts of director-free, so even if the rest were not to be ready, if we agree on what to do here, there's no reason to refrain from landing it.

[dwsinger]

I don't see a need to tie either ProtoCharter visibility or better sign-off visibility (e.g. HR groups, TAG, AB) to Director-Free.

[plehegar]

Discussed on June 8.

Suggested approach to move forward was a 2 steps:

1. replace the Director by the Team (FOs would go to the Council in any case)
2. look into fixing the pre-AC review period to avoid surprising the AC when starting a review. (no proposal yet).

Step 1 is now waiting on @frivoal to create a new pull request.
We'll come back to step 2 after that (in the fall most probably).

[plehegar]

Discussion with @frivoal on how to address the pre-AC review period for charters:

3 phases of chartering:

• phase 1 - prior to Advance Notice
• phase 2 - between Advance Notice and AC Review
• phase 3 - start at the beginning of AC review (already documented in Process)

To be added to the /Guide (with some in the Process as well?):

• Members have a right to ask the Team to start chartering (prior to phase 1), can Formally Object if the Team Decision disagree or the Team fails to answer within 3 months
• phase 2 starts when the Team sends the Advance Notice.
  • The chair of phase 2 MUST be identified by the Team in the advance notice (NOTE: this is NOT expected to be the Chair of the Working Group if it exists)
  • Phase 2 is public, and anyone can send comments
• During phase 2, the Chair:
  • MUST lead wide review (including HR)
• At the end of phase 2, the Chair:
  • MUST formally address comments
  • MUST produce a DoC
  • MUST identify the changes since the Advance Notice (diff)
• Chair of phase 2 may proceed to AC review even if there is no consensus (dissent will be identified in the DoC)
• Default duration of phase 2 is 6 months. After 6 months, the chair must either:
  • Send the charter to the Team, requesting an AC Review
  • decide to extend the chartering (for a max of 6? months [at a time|combined])
  • decide to abandon the chartering if they see no reasonable path forward. (A lack of decision is equivalent to decision to abandon)
• Decision to extend or abandon must be announced to the AC, and may be FOed.
• The Team may deny an AC Review request by the Chair if the proposed charter does not fulfill the Process's requirements about Charters, or if the Chair did not fullfill the requirements of phase 2. That is a Team decision, which must be announced, and can be Formally Objected to.
• Any FO raised during phase 2 is processed together with FOs raised during the AC review (i.e. they do not block phase 2), except FOs against the decision to abandon or extend chartering, which are processed according to the usual deadlines (See Process 5.6.1.)

[plehegar]

Question: if a Council agrees with an FO against decision to abandon, who decides whether that means we must continue working on the charter, or if it means we must send the charter to AC Review as it is? Council? Team (=TILT)?
The Council itself deciding that would be unusual: Councils typically only decide uphold / override, not what to do instead. But putting that decision elsewhere would be a weird, though TILT could handle that (but then if you get an FO against TILT's decision, you get a second council in a row…)
The same question applies to an AC appeal against the Council's decision.
Similarly, if the Council (or an AC Appeal) agrees with an FO against decision to extend, who decides if we must abandon or if must move to AC?

[plehegar]

we should probably have a "last call" period before the end of phase 2.

[mnot]

When you say 'Default duration of chartering' I think you mean 'Default duration of phase 2' -- correct?

'Chair' is confusing -- 'facilitator'? 'chartering lead'?

Other than that, this looks like a promising start. I'd like to see more about community engagement and discussion -- so many of our interactions are one-way.

[cwilso]

(as @plehegar and I were talking about this when he wrote it) yes, default duration of phase two, and I had the same feedback as you - that "chair" was confusing in context. "Chartering facilitator" seems good.

[frivoal]

you mean 'Default duration of phase 2' -- correct?

+1

'Chair' is confusing -- 'facilitator'? 'chartering lead'?

Well, I'm not too attached to what we call the person, but my idea is that we need to be clear about who is doing what, and not confuse the person(s) who are championing an idea, with the one who's trying to make sure that we address feedback properly and try to get to consensus where we can.

A "lead proponent" of a charter might want to close a change request with "yeah, no, that's not what we're trying to do here", while a chair-like figure would try to find the compromise that create the weakest objections. They might decide to go ahead without consensus if it seems unreachable, but they should strive for it, and not try to drive through a particular take that they personally favor.

To me, "chair" works for that. It's also convenient in a Process sense, as things like "Chairs decision" are already defined. If that doesn't work, I think "facilitator" could be ok too, but "lead" is too easily confused with the main person who is trying to make a particular take on the idea move forward. Charters obviously need proponents—if nobody cares it's not going to move forward—but I think distinguishing that from the person who's trying to make sure that the chartering activity is happening as it should is important.

[frivoal]

we should probably have a "last call" period before the end of phase 2.

Yes, but just like for specs, it doesn't need to be formally encoded in the Process. It can be left as a good practice: when you're wrapping up, say so and call for a last round of feedback, otherwise you might miss important comments an only get them as FOs later.

[css-meeting-bot]

The Revising W3C Process CG just discussed Charter Review Process.

The full IRC log of that discussion

<fantasai> Topic: Charter Review Process
<fantasai> github: https://github.com//issues/580
<fantasai> -> https://github.com//pull/851/files
<fantasai> florian: We've been discussing several months
<fantasai> ... currently it's informal, but this formalizes it
<fantasai> ... creates a formally identified phase (calling "charter refinement" currently)
<fantasai> ... to start it, is a Team Decision -- so can object
<fantasai> ... and has a Facilitator (Chair) to drive progress
<fantasai> ... progress requires wide review, addressing issues, etc.
<fantasai> ... and then decision to take to AC Review
<fantasai> ... if anyone files an FO prior to AC Review, we group that objection with the AC Review FOs, so the Council can address everything together
<fantasai> ... (this is something we learned from experience)
<fantasai> plh: Looking at "Advance Notice" requirement, a few things not comfortable
<fantasai> ... 1st is to identify location of charter draft which must be public
<fantasai> ... means we can't do anything if we're thinking about this, but don't have draft yet
<fantasai> florian: We're using the term 'Advance Notice" for the start of this new phase. But if we want an earlier notice... we can have two different words
<fantasai> ... which one is called 'Advance Review Notes' ... probably it's this one because can't "review" if nothing to "review"
<fantasai> ... but making a notice "we're thinking about a charter, come talk to us" is also fine
<fantasai> plh: Today we have two phases
<fantasai> ... 1. We're working on a charter, should tell AC about it
<fantasai> ... 2. We have a draft, and asking ppl to review it
<fantasai> ... horizontal review, etc.
<fantasai> ... which one is it?
<fantasai> florian: in between. You might not have a completed draft, but you have something
<fantasai> ... you could have a mostly blank draft, but probably premature
<fantasai> plh: had an AC meeting session on identity, trying to evaluate whether to look into it
<fantasai> ... sent advance notice about working on a charter
<fantasai> ... and some of us spent some time working on a draft
<fantasai> ... then went around asking for comments
<fantasai> ... hoping to start AC Review in May
<plh> ack fan
<fantasai> florian: I think at the end of these two days, you would have sent the Advance Review Notice saying "we have a thing, want ppl to work with us to make it ready"
<florian_irc> fantasai: the point of this notice to to provide more structure
<florian_irc> fantasai: if we're in a hazy phase where we don't really need formal feedback because there's no controversy, or because you don't yet know what you're doing, it's too early
<florian_irc> fantasai: but if you have put a draft together, or if you're having trouble putting a draft together because of controversy…
<cpn> q+
<florian_irc> fantasai: then having this phase with formal helps, thanks to formal chairing
<fantasai> plh: I like that, but the proposal removes requirement on Team to tell AC that they're working on a charter
<fantasai> ... we often can send draft, but some cases where we know the WG is talking about a charter and basically send a signal to AC to say, there are conversations happening over there
<plh> ack cpn
<fantasai> ... not sure that's a good thing or bad thing
<fantasai> cpn: Another benefit in this earlier review
<fantasai> ... when charters get to AC Review, you have a community very invested in the charter
<fantasai> ... if objections from AC, better to bring up earlier
<fantasai> ... so in general supportive of any effort to bring concerns earlier
<fantasai> ... in addition to other benefits described
<fantasai> cpn: wrt your concerns of Advance Notice, anything stopping from doing both things?
<fantasai> ... you can send an early notification that working on a charter, and another that it's time for wider review
<fantasai> plh: nothing stopping us other than maybe concern about too many emails to AC
<fantasai> ... a little uncomfortable about removing requirement on staff about this early notification
<florian_irc> q+
<cpn> q+
<plh> ack florian_irc
<fantasai> florian_irc: Not removing requirement on Team to communicate, just shifts timing
<fantasai> ... if Team is wondering whether it should try to write a draft, want to ask feedback, send a notice
<fantasai> ... but if you already know, then just start it and announce the draft
<fantasai> ... if you anticipate that what you draft will be controversial, make a draft with lots of "fill in the blank" and send the notice earlier to ask for help
<plh> ack cpn
<fantasai> cpn: I think I'd prefer to keep the requirement for earlier advance notice
<fantasai> ... even if that means multiple notifications
<fantasai> ... uncomfortable about later in the phase only
<plh> ack fantasai
<florian_irc> fantasai: I don't have a strong feeling either way on the earlier notice
<florian_irc> fantasai: we can keep that req if people want it
<florian_irc> fantasai: if that's too much traffic, we can set up a separate mailing list that people can subscribe to
<florian_irc> fantasai: we can also kick that question to the AB or AC
<florian_irc> q+
<fantasai> plh: I don't have strong feelings either, want to hear from AC
<fantasai> ... if early notice and advance notice coincide because we receive a charter that's fairly complete, can merge into one
<plh> ack florian_irc
<fantasai> florian_irc: if we go this way, I suggest we amend PR to include both
<fantasai> ... that way we can have stable names for them
<fantasai> ... and then include a provision that says if they happen at the same time, can send one mail
<fantasai> plh: corner case, Verifiable Credentials has been discussing their charter
<fantasai> ... only change is, reason they do rechartering because can't extend more than 6 months
<fantasai> ... and that requires an AC Review
<fantasai> ... so nothing much changed, and here in this advance review notice, expected duration for phase is 28 days
<fantasai> ... so basically before 28 days of AC review, force 28 days of pre-review
<fantasai> ... most cases it's fine, but some cases where nothing changed in the charter
<fantasai> ... corner case, idk if we care enough
<fantasai> florian_irc: 6-month limit, is that good practice or basis in the Process?
<fantasai> plh: might be good practice, maybe W3M committed to AC
<fantasai> ... AC didn't like e.g. 2-year extensions
<fantasai> florian_irc: should this be in the Process?
<fantasai> ... we could say charter extension is by Team Decision, but longer than six months requires AC Review
<fantasai> ... [missed]
<fantasai> fantasai: How about just if it's under the Team Decision scope of minor decisions, then Team MAY request AC Review (and then doesn't require the 28-day wide review phase)
<fantasai> florian_irc: could do that, and if later want to forbid Team from e.g. longer extensions then can request a separate change
<fantasai> fantasai: At Time
<fantasai> florian_irc: ok, I'll make adjustments and we'll come back to it

[frivoal]

Ask requested by @tantek , pasting here some points that were made in the AB discussion of this topic:

There, @fantasai said:

Tantek asked to clarify the problems this proposal is trying to solve. There are 3 I can think of:

• Reduce the number of FOs at AC Review that could have been avoided by solving small problems simply and developing consensus on larger ones through dialog.
• Ensure that comments get addressed, not ignored, by those developing the charter and submitting to the AC.
• Make the chartering process more understandable, and therefore easier to participate in (whether drafting/proposing or commenting on the charters).

[frivoal]

Landed PR #851 into the Editor's draft to address this issue. Additional concerns not yet discussed should be filed as separate issues.