Internal testing for within a8c and rollout mechanism

[cpapazoglou]

Before launching the Nav Unification project to the audience, we will test this feature internally. To do so we need to find a mechanism that will allow us:

1. Target only a11ns or allow a11ns to opt in for testing
2. By opting in, an a11n should not need to add any ?flags=nav-unification in calypso or blog sticker for wp-admin
3. By opting in, both interfaces (calypso and wp-admin) should load the new UI / UX directly (without the intermediate blue masterbar in calypso)
4. Targeting Users (and not sites) will make the experience smoother, so that user will not see different UIs while navigating across their sites in calypso.

We can possibly leverage the experimentation platform. Here is a guide and here is a workshop video pbmo2S-yu-p2.

Rollout targets have been decided here #47050

Internal Testing Phase	Rollout Phase 1	Rollout Phase 2	Rollout Phase 3	Rollout Phase n
isA11n	isA11n OR UserId%100 < 5 && singleSiteUser	isA11n OR UserId%100 < 20 && singleSiteUser	isA11n OR UserId%100 < 50 && singleSiteUser OR UserId%100 < 5 && !singleSiteUser	20%>50%>100%

[cpapazoglou]

It seems that the only way to avoid the the intermediate blue masterbar is by :

1. removing it from the initial state

   wp-calypso/client/document/index.jsx

   Line 146 in 76ae957

   <div className="masterbar" />

2. or parsing a cookie set from the backend document.cookie = 'flags=nav-unification;max-age=1209600;path=/';

[cpapazoglou]

After doing some research I came down to the following solutions:

Solutions which may be also useful for rolling out #47047:

1. Using ExPlat:

I have an open discussion here pbmo2S-zB-p2 so that can learn more about the abilities of ExPlat. Till then

Pros:

• a11ns are excluded by default, they can opt in through the platform
  
• We can target users (audience) by: new/existing user, locale, country
• We can possibly? change the affected percentage of the audience through the platform, without deploying changes again WIP to altering the audience programmatically PoC: A quick demonstration of adapting the ExPlat components into the Plan step page and the checkout page. #47088 , D52254-code
• Works on both calypso & wpcom by using these snippets https://github.com/Automattic/experimentation-platform/wiki/Experimenter's-Guide#writing-the-controlvariant-code-experiences
• we can a/b test the nudges performance

Cons:

• Not sure yet it supports atomic (answer pending). In which case, we can filter out atomic users and only launch them when we go 100%
• Audience criteria are limited

2. Combination of WET, wpcomsh, wpcom as proposed here p1604532994070700-slack-C014TPFLP4Z?thread_ts=1604500291.058600&cid=C014TPFLP4Z

Pros:

Cons:

• not agile enough on changing audience cause we need to deploy calypso, WET & wpcomsh all at once!
• wpcomsh is no longer suggested for implementing new features

3. Developing a wpcom endpoint to serve remote feature flags

We may develop something like this endpoint D48213-code which will return all feature flags and their state (enabled, disabled). Also, we may hook our audience logic so it returns different state per user.

We can then consume this endpoint by calypso, wpcom, atomic possibly via Jetpack code.

Pros:

• We can create the audience programatically and therefore we have more control

Cons:

• No infrastructure yet
• Will need a wpcom deploy every time we need changes
• We have to built the opt-in/ opt-out mechanism (possibly through calypso /me account preferences)

Using horizon is not an option as there is not any wpcom equivalent environment.

[cpapazoglou]

It seems that the only way to avoid the the intermediate blue masterbar is by :

1. removing it from the initial state

   wp-calypso/client/document/index.jsx

   Line 146 in 76ae957

   <div className="masterbar" />

2. or parsing a cookie set from the backend document.cookie = 'flags=nav-unification;max-age=1209600;path=/';

How would you feel in removing the masterbar from the initial loading state screen? At least until we have rolled out nav unification to 100% of the users. It will certainly make things easier!

Before	After

cc: @davemart-in, @sfougnier

[ghost]

How would you feel in removing the masterbar from the initial loading state screen? At least until we have rolled out nav unification to 100% of the users. It will certainly make things easier!

I think it makes sense to remove it.

[davemart-in]

How would you feel in removing the masterbar from the initial loading state screen? At least until we have rolled out nav unification to 100% of the users. It will certainly make things easier!

@cpapazoglou Yes please!

[cpapazoglou]

In order to get more stability / performance, it is advised that we don't do requests each time an Atomic User reloads /wp-admin. In that case we can use add_filter( 'jetpack_load_admin_menu_class', '__return_true' ); that is already defined here Automattic/jetpack#17629.

It will reside in wpcomsh for atomic Users and wpcom-experiment-custom-segments.php in wpcom D52254-code. This way:

1. we just need to deploy wpcom and wpcomsh each time we need to change audience (10%, then 20% etc)
2. we utilise experiments in calypso and wpcom which can provide as with nudge metrics and instant opt in / opt out for a8c users

[cpapazoglou]

After a discussion with @withinboredom (p1605882949141200-slack-CJS75TX3R) it seems that syncing the experiment in calypso / wpcom (simple sites) /wpcomsh (atomic sites) poses some challenges when using an experiment. We will use it initially for development and internal testing (easy opt-in / opt-out) and we will decide in the future if and how we can leverage it for production rollouts. For the record, it would be great if we had the following:

• atomic integration.
• broaden the audience without deploy (or at least with minimum effort).

So, let's move to Plan B

Development & Internal Testing Phase

We will leverage the experiment for development and internal testing of calypso and wpcom simple sites - you just need to opt-in to the nav-unification experiment. In order to develop or test atomic sites:

• Uncomment 562-gh-Automattic/wpcomsh/files#diff-afccf5f632a6c554163a3276a51ee7c4f2bb1159a5b44654fc1c820c96f990d9R62-R65
• Add define('IS_ATOMIC', true); in wpcomsh-loader.php

Production Rollout

We will need to deploy all calypso - wpcom - wpcomsh that will contain the following changes:

• remove the experiment code (not applicable to wpcomsh)
• instead add a check that UserIsAutomattician() || ( 1 === $user_site_count && $user_id % 100 < 5) which will eventually target all a12s and all 5% of single site users.

[cpapazoglou]

cc @Automattic/ajax

[obenland]

@davemart-in FYI ^