Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DuckPlayer Launch Experiment for iOS #3328

Merged
merged 19 commits into from
Sep 6, 2024
Merged

Conversation

afterxleep
Copy link
Collaborator

@afterxleep afterxleep commented Sep 5, 2024

Task/Issue URL: https://app.asana.com/0/1204099484721401/1208091265812750/f
Tech Design URL: https://app.asana.com/0/1204099484721401/1207989846862252/f

Description:
We're running a launch experiment to try measure retention impact with DuckPlayer launch. This is a pixel experiment, consisting in a few pixels sent in these scenarios:

  • Search Performed
  • Youtube Page visited

Steps to test this PR:
Code includes tests covering the correct pixel firing on different dates/weeks, etc.

Setup

  • Make yourself Internal user
  • Drag content-scope-scripts to project from branch pr-releases/release-993
  • Add custom privacy config from https://www.jsonblob.com/api/1276215786342309888
  • Go to Settings and confirm 'Duck Player' is NOT visible
  • Filter your console messages with duckplayer.experiment (No category)

Control Group

  • Modify this, so it always returns control (let cohort: Cohort = .control)
  • Build & Run
  • Go to Youtube.com and search for a 'metallica'
  • Tap the first video.
  • Check your console and confirm the following two pixels were fired
  • Confirm variant is control, state is disabled and date matches today's date
  • Confirm that youtube.page pixel referer is youtube

Example Pixels:

Pixel fired duckplayer.experiment.cohort.assign ["enrollment": "20240905", "variant": "control"]
Pixel fired duckplayer.experiment.youtube.page.view ["variant": "control", "day": "0", "referrer": "youtube", "state": "alwaysAsk", "enrollment": "20240905"]
  • Refresh the page multiple times and confirm the pixel is not fired multiple times
  • Go back and select another video
  • Confirm the duckplayer.experiment.youtube.page.view pixel is fired again
  • Search for black sabbathusing the URL Bar
  • Confirm the following 2 pixels are fired in the console

Example pixels

Pixel fired duckplayer.experiment.search ["enrollment": "20240905", "variant": "control", "day": "0"]
Pixel fired duckplayer.experiment.daily.search ["enrollment": "20240905", "variant": "control", "day": "0"]
  • Run another search for black sabbath war pigs and confirm only one pixel is fired in the console

Pixel fired duckplayer.experiment.search ["day": "0", "variant": "control", "enrollment": "20240905"]

  • Tap the first video from the results
  • Confirm the following pixel is fired in the console, and referrer is serp

Example Pixel

Pixel fired duckplayer.experiment.youtube.page.view ["enrollment": "20240905", "state": "disabled", "variant": "control", "day": "0", "referrer": "serp"]
  • Go to google.com and search for number of the beast
  • Tap the first link to Youtube video (make sure it opens in Youtube)
  • Confirm the experiment.search pixel is fired and referrer is other

Example Pixel

Pixel fired duckplayer.experiment.youtube.page.view ["variant": "control", "day": "0", "state": "disabled", "referrer": "other", "enrollment": "20240905"]

Experiment Group

  • Modify this, so it always returns experiment (let cohort: Cohort = .experiment)
  • Go to Settings > Debug > All Debug Options > Reset DuckPlayer Experiment
  • Close all tabs
  • Go to Settings, and confirm DuckPlayer is NOT visible (This is expected)
  • Go to Youtube.com and search for 'creedence'
  • Tap the first video.
  • Confirm DuckPlayer Overlay is visible and both the enrollment pixel and video view pixels are there
  • Confirm that the referrer in the view pixel is 'alwaysAsk' or whatever setting you had for DuckPlayer before
  • Tap Turn On Duck Player
  • Confirm DuckPlayer shows up and No additional pixel is fired
  • Add page to favorites
  • Go back and navigate to another video
  • Close all tabs and open the favorite you've added before.. Confirm a view pixel is fired again
  • Go to Settings, and confirm DuckPlayer IS now visible

Example Pixel for experimen view

Pixel fired duckplayer.experiment.youtube.page.view ["variant": "control", "day": "0", "state": "alwaysAsk", "referrer": "other", "enrollment": "20240905"]

Other

  • Smoke test DuckPlayer (Disable, enable, watch videos, etc)

@@ -110,8 +123,9 @@ final class DuckPlayer: DuckPlayerProtocol {
static let featureNameKey = "featureName"
}

private(set) var settings: DuckPlayerSettingsProtocol
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Renamed the protocol to DuckPlayerSettings, which makes more sense

UserValues(
duckPlayerMode: settings.mode,
let experiment = DuckPlayerLaunchExperiment()
let isEnabled = experiment.isEnrolled && experiment.isExperimentCohort && featureFlagger.isFeatureOn(.duckPlayer)
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These values are the ones passed to the FE, so account for experiment group and feature flag

let currentDate = dateProvider.currentDate
let calendar = Calendar.current
let dayDifference = calendar.dateComponents([.day], from: enrollmentDate, to: currentDate).day ?? 0
let weekDifference = (dayDifference / 7) + 1
Copy link
Collaborator Author

@afterxleep afterxleep Sep 5, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Week "0" is not a thing. We expect it to be 1 or 2, but Day "0" is OK



// Handle custom events
func handleEvent(event: DuckPlayerNavigationEvent, url: URL?, navigationAction: WKNavigationAction?) {
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The idea here is to handle 'Custom' events that are unrelated to DuckPlayer as a feature but not with presentation.-

@@ -175,7 +175,6 @@ extension MainViewController {

private func fireBrokenSiteReportShown() {
let parameters = [
PrivacyDashboardEvents.Parameters.variant: PixelExperiment.privacyDashboardVariant.rawValue,
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There was a bunch of old PixelExperiment stuff that was not cleaned up before 🤷

@afterxleep afterxleep marked this pull request as ready for review September 5, 2024 16:05
@afterxleep afterxleep requested a review from Bunn September 5, 2024 16:05
Copy link

github-actions bot commented Sep 5, 2024

Warnings
⚠️ PR has more than 500 lines of code changing. Consider splitting into smaller PRs if possible.

Generated by 🚫 dangerJS against b621678

Copy link
Contributor

@Bunn Bunn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code is OK for experiment.

@afterxleep afterxleep merged commit 3cedd20 into main Sep 6, 2024
13 checks passed
@afterxleep afterxleep deleted the daniel/duckplayer.experiment branch September 6, 2024 07:19
samsymons added a commit that referenced this pull request Sep 9, 2024
# By Christopher Brind (4) and others
# Via Alessandro Boron (1) and others
* main: (27 commits)
  Bump C-S-S to 6.14.1 (#3331)
  DuckPlayer Launch Experiment for iOS (#3328)
  defer loading the tab switcher button until view did load (#3326)
  Release 7.136.0-3 (#3324)
  Release PR: Check for the negative attribution case (#3311)
  fix tab switcher crashes (speculative fix) (#3319)
  Onboarding highlights feature flag setup (#3308)
  Release 7.136.0-2 (#3320)
  Attempt to fix dissapearing privacy icon (#3317)
  Fix bookmarks toolbar behaviour with Sync Promo on iOS 15 (#3313)
  Bump BSK with C-S-S to 6.14.0 (#3314)
  ensure no atb or app version sent with pixel (#3315)
  Fix #3298: Add support for Xcode 16 (#3299)
  Fix Keychain Debug view controller segue (#3310)
  Release 7.136.0-1 (#3309)
  Fix an issue that causes the fire dialog to show multiple times for the same website after dismissing it (#3305)
  [DuckPlayer] 28. Open in Youtube -> Youtube App (#3290)
  usage segmentation (#3263)
  Fix wrong URL displayed for auth dialog (#3307)
  iOS Integration of BSK Onboarding (#3282)
  ...

# Conflicts:
#	DuckDuckGo.xcodeproj/project.pbxproj
#	DuckDuckGo.xcodeproj/project.xcworkspace/xcshareddata/swiftpm/Package.resolved
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants