Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[P3A] New question about Search providers #18224

Closed
mattmcalister opened this issue Sep 20, 2021 · 2 comments · Fixed by brave/brave-core#10300 or brave/brave-core#10627
Closed

[P3A] New question about Search providers #18224

mattmcalister opened this issue Sep 20, 2021 · 2 comments · Fixed by brave/brave-core#10300 or brave/brave-core#10627
Assignees
@rillian
Labels
feature/new-metric Proposing a new metric for Privacy Preserving Product Analytics (P3A) features/P3A OS/Android Fixes related to Android browser functionality OS/Desktop priority/P3 The next thing for us to work on. It'll ride the trains. QA Pass - Android ARM QA Pass-Linux QA Pass-macOS QA Pass-Win64 QA/Test-All-Platforms QA/Yes release-notes/exclude
Milestone
1.31.x - Release #2

Comments

@mattmcalister
Copy link

mattmcalister commented Sep 20, 2021 •
edited by rillian
Loading

Question:
Did you switch search engines?

Answers:

(0) Default was not Brave, did not switch
(1) Default was Brave, did not switch
(0) Did not switch
(1) Default was Brave, switched to Google
(2) Default was Brave, switched to DDG
(3) Default was Brave, switched to Other (NOT Google OR DDG)
(4) Default was Google, switched to Brave
(5) Default was DDG, switched to Brave
(6) Default was Other (NOT Google OR DDG), switched to Brave
(7) Default was Other (not Brave), switched to Other (not Brave)

(Answers amended to meet the 8-option limit for P3A questions.)

Justification:
Data about switching behavior and retention will inform our feature prioritization for Brave Search and integration with the browser.
@mattmcalister mattmcalister added OS/Android Fixes related to Android browser functionality features/P3A OS/Desktop feature/new-metric Proposing a new metric for Privacy Preserving Product Analytics (P3A) labels Sep 20, 2021
rillian added a commit to brave/brave-core that referenced this issue Sep 28, 2021
@rillian
Record default search engine changes.
aba3b26 
Add a P3A probe to record when the default search engine is changed,
and whether it's a transition between Brave Search or one of the
other major services. This helps understand what options are seem
better to users.

Closes brave/brave-browser#18224
This was referenced Sep 28, 2021
Merged
Closed
@rebron rebron assigned rillian and unassigned aekeus and kkuehlz Sep 30, 2021
@rebron rebron added the priority/P3 The next thing for us to work on. It'll ride the trains. label Sep 30, 2021
rillian added a commit to brave/brave-core that referenced this issue Oct 4, 2021
@rillian
Record default search engine changes.
0fd784a 
Add a P3A probe to record when the default search engine is changed,
and whether it's a transition between Brave Search or one of the
other major services. This helps understand what options are seem
better to users.

Closes brave/brave-browser#18224
rillian added a commit to rillian/brave-core that referenced this issue Oct 12, 2021
@rillian
Record default search engine changes.
c3c2365 
Add a P3A probe to record when the default search engine is changed,
and whether it's a transition between Brave Search or one of the
other major services. This helps understand what options are seem
better to users.

Closes brave/brave-browser#18224
rillian added a commit to brave/brave-core that referenced this issue Oct 13, 2021
@rillian
Record default search engine changes.
edeb3cf 
Add a P3A probe to record when the default search engine is changed,
and whether it's a transition between Brave Search or one of the
other major services. This helps understand what options are seem
better to users.

Keep track of the previous default search engine url. The event
the tracker observes only indicates something has changed, and
by then the previous state is no longer accessible; we need this
to correctly mark the transition in our matrix, separate from
whatever the original default was at startup.

Move the actual record step into an object method so it can access
the saved previous default. This simplifies calling it at startup
to set the initial kNoSwitch answer.

Note: this means we only report a change if the ping is sent from
the same application instance. We don't reset to kNoSwitch at the
end of the weekly sample cadence, and restarting the browser always
resets.

Closes brave/brave-browser#18224
rillian added a commit to brave/brave-core that referenced this issue Oct 13, 2021
@rillian
P3A: Record default search engine changes.
a732abd 
Add a P3A probe to record when the default search engine is changed,
and whether it's a transition between Brave Search or one of the
other major services. This helps understand what options are seem
better to users.

Keep track of the previous default search engine url. The event
the tracker observes only indicates something has changed, and
by then the previous state is no longer accessible; we need this
to correctly mark the transition in our matrix, separate from
whatever the original default was at startup.

Move the actual record step into an object method so it can access
the saved previous default. This simplifies calling it at startup
to set the initial kNoSwitch answer.

Note: this means we only report a change if the ping is sent from
the same application instance. We don't reset to kNoSwitch at the
end of the weekly sample cadence, and restarting the browser always
resets.

Closes brave/brave-browser#18224
@rillian rillian added this to the 1.33.x - Nightly milestone Oct 20, 2021
rillian added a commit to brave/brave-core that referenced this issue Oct 20, 2021
@rillian
P3A: Record default search engine changes.
adc0a81 
Add a P3A probe to record when the default search engine is changed,
and whether it's a transition between Brave Search or one of the
other major services. This helps understand what options are seem
better to users.

Keep track of the previous default search engine url. The event
the tracker observes only indicates something has changed, and
by then the previous state is no longer accessible; we need this
to correctly mark the transition in our matrix, separate from
whatever the original default was at startup.

Move the actual record step into an object method so it can access
the saved previous default. This simplifies calling it at startup
to set the initial kNoSwitch answer.

Note: this means we only report a change if the ping is sent from
the same application instance. We don't reset to kNoSwitch at the
end of the weekly sample cadence, and restarting the browser always
resets.

Closes brave/brave-browser#18224
@rillian rillian mentioned this issue Oct 20, 2021
Merged
24 tasks
rillian added a commit to brave/brave-core that referenced this issue Oct 20, 2021
@rillian
P3A: Record default search engine changes.
e81898d 
Add a P3A probe to record when the default search engine is changed,
and whether it's a transition between Brave Search or one of the
other major services. This helps understand what options are seem
better to users.

Keep track of the previous default search engine url. The event
the tracker observes only indicates something has changed, and
by then the previous state is no longer accessible; we need this
to correctly mark the transition in our matrix, separate from
whatever the original default was at startup.

Move the actual record step into an object method so it can access
the saved previous default. This simplifies calling it at startup
to set the initial kNoSwitch answer.

Note: this means we only report a change if the ping is sent from
the same application instance. We don't reset to kNoSwitch at the
end of the weekly sample cadence, and restarting the browser always
resets.

Closes brave/brave-browser#18224
@rillian rillian mentioned this issue Oct 20, 2021
Merged
24 tasks
@GeetaSarvadnya GeetaSarvadnya mentioned this issue Oct 21, 2021
Closed
@LaurenWags
Copy link
Member

LaurenWags commented Oct 29, 2021 •
edited
Loading

Verified passed with

Brave | 1.31.88 Chromium: 95.0.4638.69 (Official Build) (x86_64)
-- | --
Revision | 6a1600ed572fedecd573b6c2b90a22fe6392a410-refs/branch-heads/4638@{#984}
OS | macOS Version 11.6 (Build 20G165)

Verified test plan from brave/brave-core#10300 in US region
Confirmed each of the P3A responses listed in the description

Brave is default (did not switch) - PASSED 
         "Brave.Search.SwitchEngine": {
            "sent": false,
            "value": "0"
         },
Step4
Switch from Brave to Google - PASSED 
         "Brave.Search.SwitchEngine": {
            "sent": false,
            "value": "1"
         },
brave to google
Switch from Google to Brave - PASSED 
         "Brave.Search.SwitchEngine": {
            "sent": false,
            "value": "4"
         },
Google to Brave
Switch from Brave to DDG - PASSED 
         "Brave.Search.SwitchEngine": {
            "sent": true,
            "timestamp": 1635510311.293804,
            "value": "2"
         },
Brave to DDG
Switch from DDG to Brave - PASSED 
         "Brave.Search.SwitchEngine": {
            "sent": true,
            "timestamp": 1635510311.293804,
            "value": "5"
         },
DDG to Brave
Switch from Brave to Ecosia - PASSED 
         "Brave.Search.SwitchEngine": {
            "sent": true,
            "timestamp": 1635510311.293804,
            "value": "3"
         },
Brave to Ecosia
Switch from Ecosia to Startpage - PASSED 
         "Brave.Search.SwitchEngine": {
            "sent": true,
            "timestamp": 1635510311.293804,
            "value": "7"
         },
Ecosia to Startpage
Switch from Startpage to Brave - PASSED 
         "Brave.Search.SwitchEngine": {
            "sent": true,
            "timestamp": 1635510311.293804,
            "value": "6"
         },
Startpage to Brave

Verification passed on


Brave | 1.31.88 Chromium: 95.0.4638.69 (Official Build) (64-bit)
-- | --
Revision | 6a1600ed572fedecd573b6c2b90a22fe6392a410-refs/branch-heads/4638@{#984}
OS | Windows 10 Version 21H1 (Build 19043.1288)

Verified test plan from brave/brave-core#10300 in FR region
Confirmed each of the P3A responses listed in the description

New User
Clean profile, default SE is brave - PASSED
  • Installed 1.31.88 RC build and ensured that the default SE is set as Brave
  • Confirmed that the response value for Brave.Search.SwitchEngine is 0 as expected
1.31.88 Brave Brave.Search.SwitchEngine - Response value is 0
image image
Clean profile, default is Brave, switch to Google - PASSED
  • Installed 1.31.88 RC build and ensured that the default SE is set as Brave
  • Change default SE to Google
  • Confirmed that the response value for Brave.Search.SwitchEngine is changed from 0 to 1 as expected
1.31.88 Brave Brave.Search.SwitchEngine - Response value is 1
image image
Clean profile, default is Brave, switch to Google and again switch back to Brave- PASSED
  • Installed 1.31.88 RC build and ensured that the default SE is set as Brave
  • Change default SE to Google and then switch back to Brave
  • Confirmed that the response value for Brave.Search.SwitchEngine is changed from 0 ---> 1-->4 as expected
1.31.88 Brave Brave.Search.SwitchEngine - Response value is 0 1.31.88 Google Brave.Search.SwitchEngine - Response value is 1 1.31.88 Brave Brave.Search.SwitchEngine - Response value is 4
image image image image image image
Clean profile, default is Brave, switch to DDG - PASSED
  • Installed 1.31.88 nightly build and ensured that the default SE is set as Brave
  • Change default SE to DDG
  • Confirmed that the response value for Brave.Search.SwitchEngine is changed from 0 to 2 as expected
1.31.88 Brave Brave.Search.SwitchEngine - Response value is 0 1.31.88 DDG Brave.Search.SwitchEngine - Response value is 2
image image image image
Clean profile, default is Brave, switched to Other (NOT Google OR DDG) - PASSED
  • Installed 1.31.88 nightly build and ensured that the default SE is set as Brave
  • Change default SE to Ecosia
  • Confirmed that the response value for Brave.Search.SwitchEngine is changed from 0 to 3 as expected
1.31.88 Brave Brave.Search.SwitchEngine - Response value is 0 1.31.88 Ecosia Brave.Search.SwitchEngine - Response value is 3
image image image image
Clean profile, default is Brave, switch to Other (NOT Google OR DDG), switched to Brave- PASSED
  • Installed 1.31.88 nightly build and ensured that the default SE is set as Brave
  • Change default SE to Startpage and then switch back to Brave
  • Confirmed that the response value for Brave.Search.SwitchEngine is changed from 0 ---> 3-->6 as expected
1.31.88 Brave Brave.Search.SwitchEngine - Response value is 0 1.31.88 Startpage Brave.Search.SwitchEngine - Response value is 3 1.31.88 Brave Brave.Search.SwitchEngine - Response value is 6
image image image image image image
Clean profile, default to Other (not Brave), switched to Other (not Brave)- PASSED
  • Installed 1.31.88 nightly build and ensured that the default SE is set as Brave
  • Change default SE to Bing and then switch to Qwant
  • Confirmed that the response value for Brave.Search.SwitchEngine is changed from 0 ---> 3-->7 as expected
1.31.88 Brave Brave.Search.SwitchEngine - Response value is 0 1.31.88 Bing Brave.Search.SwitchEngine - Response value is 3 1.31.88 Qwant Brave.Search.SwitchEngine - Response value is 7
image image image image image image
Existing user
Default was not Brave, did not switch - PASSED
  • Installed 1.30.x RC build and ensured that the default SE is set as Google
  • Upgraded to 1.31.x nightly build and ensured that Google was still the default SE after upgrading
  • Confirmed that the response value for Brave.Search.SwitchEngine is 0 as expected
1.30.x Google 1.31.x Google Brave.Search.SwitchEngine - Response value is 0
image image image
Default was Brave, switched to Google - PASSED
  • Installed 1.30.x RC build and ensured that the default SE is set as Brave
  • Upgraded to 1.31.88 nightly build and changed SE to Google after upgrade
  • Confirmed that the response value for Brave.Search.SwitchEngine is 1 as expected
1.30.x Brave 1.31.88 Google Brave.Search.SwitchEngine - Response value is 1
image image image
Default was Brave, switched to Other (NOT Google OR DDG) - PASSED
  • Installed 1.30.x RC build and ensured that the default SE is set as Brave
  • Upgraded to 1.31.88 nightly build and changed SE to Bing after upgrade
  • Confirmed that the response value for Brave.Search.SwitchEngine is 3 as expected
1.30.x Brave 1.31.x Bing Brave.Search.SwitchEngine - Response value is 3
image image image

Verified passed with

Brave	1.31.88 Chromium: 95.0.4638.69 (Official Build) (64-bit)
Revision	6a1600ed572fedecd573b6c2b90a22fe6392a410-refs/branch-heads/4638@{#984}
OS	Linux

Verified test plan from brave/brave-core#10300 in CAN region
Confirmed each of the P3A responses listed in the description

Brave is default (did not switch) - PASSED 
         "Brave.Search.SwitchEngine": {
            "sent": false,
            "value": "0"
         },
1 Brave Default
Switch from Brave to Google - PASSED 
         "Brave.Search.SwitchEngine": {
            "sent": false,
            "value": "1"
         },
2 Brave to Google
Switch from Google to Brave - PASSED 
         "Brave.Search.SwitchEngine": {
            "sent": true,
            "timestamp": 1635520704.800581,
            "value": "4"
         },
3 Google to Brave
Switch from Brave to DDG - PASSED 
         "Brave.Search.SwitchEngine": {
            "sent": true,
            "timestamp": 1635520704.800581,
            "value": "2"
         },
4 Brave to DDG
Switch from DDG to Brave - PASSED 
         "Brave.Search.SwitchEngine": {
            "sent": true,
            "timestamp": 1635520704.800581,
            "value": "5"
         },
5 DDG to Brave
Switch from Brave to Ecosia - PASSED 
         "Brave.Search.SwitchEngine": {
            "sent": true,
            "timestamp": 1635520704.800581,
            "value": "3"
         },
6 Brave to Ecosia
Switch from Ecosia to Bing - PASSED 
         "Brave.Search.SwitchEngine": {
            "sent": true,
            "timestamp": 1635520704.800581,
            "value": "7"
         },
7 Ecosia to Bing
Switch from Bing to Brave - PASSED 
         "Brave.Search.SwitchEngine": {
            "sent": true,
            "timestamp": 1635520704.800581,
            "value": "6"
         },
8 Bing to Brave

@srirambv
Copy link
Contributor

Verification passed on Oppo Reno 5 with Android 11 running 1.31.88 x64 build
(0) Did not switch (1) Default was Brave, switched to Google (2) Default was Brave, switched to DDG (3) Default was Brave, switched to Other (NOT Google OR DDG)
image image image image
(4) Default was Google, switched to Brave (5) Default was DDG, switched to Brave (6) Default was Other (NOT Google OR DDG), switched to Brave (7) Default was Other (not Brave), switched to Other (not Brave)
image image image image

@rillian rillian mentioned this issue Nov 8, 2021
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@rillian rillian

Labels
feature/new-metric Proposing a new metric for Privacy Preserving Product Analytics (P3A) features/P3A OS/Android Fixes related to Android browser functionality OS/Desktop priority/P3 The next thing for us to work on. It'll ride the trains. QA Pass - Android ARM QA Pass-Linux QA Pass-macOS QA Pass-Win64 QA/Test-All-Platforms QA/Yes release-notes/exclude
Projects
None yet
Milestone
1.31.x - Release #2
9 participants
@mattmcalister @aekeus @rillian @kjozwiak @rebron @srirambv @LaurenWags @kkuehlz @GeetaSarvadnya