Overriding referrer: publisherDomain vs pageUrl

[bretg]

Description

We have two overlapping ways for publishers to override the results of Prebid's attempt to derive the page's URL that's stored in bidderRequest.refererInfo as described in https://docs.prebid.org/dev-docs/bidder-adaptor.html#referrers

Another use case here is that the publisher may want to suppress the full page URL from bidders.

I'd like to propose that we fix this in 6.0:

1. Allow the pub to define the pageUrl with setConfig
2. Prebid.js core reads the pageUrl value and overrides bidderRequest.refererInfo.canonicalUrl
3. Adapters cannot call getConfig or any other function (e.g. window.location), only retrieving the values from bidderRequest.refererInfo
4. The publisherDomain config goes away. If someone needs to have just the domain, we could add it to bidderRequest.refererInfo

Here are the two current setConfig options:

publisherDomain

As documented at https://docs.prebid.org/dev-docs/publisher-api-reference.html#setConfig-Publisher-Domain

Set the publisher’s domain where Prebid is running, for cross-domain iframe communication:

pbjs.setConfig({ publisherDomain: "https://www.theverge.com" )

This is used by the avocet and readpeak adapters, but they appear to expect the value to be "example.com" rather than "https://example.com"

pageUrl

There is a setConfig value used by 16 adapters in addition to the refererInfo object or as an override.

e.g.

const page = utils.deepAccess(bidderRequest, 'refererInfo.canonicalUrl') || config.getConfig('pageUrl') || utils.deepAccess(window, 'location.href');

config.getConfig('pageUrl') || bidderRequest.refererInfo.referer

[patmmccann]

Requested not to be used at https://github.com/prebid/Prebid.js/pull/5674/files#r480260677

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[gglas]

Current plan is to copy over pageURL for current 6 adapters, will also explore a utility function that copies one value to the other, also for docs :

1. deprecation of pubDomain
2. Bidder Doucmentation - domain needs to be retrieved from bidderRequest.refererInfo. Nowhere else.

[jbartek25]

@bretg Would you please update the doc for pageUrl. The doc falsely claims that the pageUrl param overrides the Prebid.js page referrer algorithm.

[bretg]

@jbartek25 - why do you say "The doc falsely claims"? I just made two test pages that show pageUrl does in fact get passed to both client and server adapters.

That said, I fixed a couple of annoying typos in that section.

[jbartek25]

@bretg You are right in that the pageUrl gets passed to adapters but with exception of a few adapters, pageUrl is not checked. It's publisher reference stating that pageUrl will override the page referrer but right now that's true for some adapters and not others, not even for Appnexus/Xandr:

[bretg]

Hmm. That's a problem. What's supposed to happen is that it overrides the bidrequest refererinfo.

Anyhow, I tweaked the docs to make this an unappealing feature as it works only for some bidders. prebid/prebid.github.io#2864

[jbartek25]

@bretg thank you

[gglas]

relevant deprecation addressed in #6591, we need to alter the relevant adapters that consume this.

Need a separate PR to alter these adapters, raising at 5.0 meeting

[patmmccann]

I see several more adapters than avocet and readpeak using this including synacor, adwmg, interactiveoffers, adyoulike, and apacdex. I also see an issue with a test on test/spec/config_spec.js

[dgirardi]

I believe bidderRequest.refererInfo is simply broken at the moment - the referer it provides is not actually the referrer in most cases (see #8294 (comment))

Here's what I intend to do for 7: (out of date, see below)

• RemoverefererInfo.referer;
• add refererInfo.ref, pointing to our best guess of who the referrer is;
• add refererInfo.page that resolves to refererInfo.canonicalUrl || window.top.location.href (canonicalUrl now respects setConfig({pageUrl}), but is more or less left undefined otherwise)
• refactor every adapter to use one or the other - I'll have to do some guesswork to understand if when they pick refererInfo.referer they actually want the referer, or they are expecting it to be the pageUrl.

[philipwatson]

Hi @dgirardi
To me, it looks like it is working as designed. If you look at the original PR:

#3067

The 'referer' field is the "detect top url". So with cross-domain iframes, it'll be the top-most accessible window's document.referrer (the top-most url we can get). And the functions utils.getTopWindowLocation and utils.getTopFrameReferrer were deprecated.

I must admit, the chosen name 'referer' is confusing.

I kinda agree with @robertrmartinez #8294 (comment) It'll be nice to have the page referer (may be called pageReferer). I see adapters use the same bit of code:

function getTopWindowReferrer() {
  try {
    return window.top.document.referrer;
  } catch (e) {
    return '';
  }
}

This implementation makes more sense than his example.
As refererInfo.referer could give iframe referrer. We can know this by using refererInfo.reachedTop.

[patmmccann]

it seems this could just be presented to adapters on ortb2.site.page and ortb2.site.ref without requiring looking at referInfo ?

[dgirardi]

After working on it for a while, here's what I am actually doing:

• rename referer to topmostLocation, keeping it for backwards compatibility.
• add location - the value of window.top.location, or null when that is unavailable (blocked by cross-origin protections)
• add ref - the value of window.top.document.referrer, or null when that is unavailable
• add page - canonicalUrl falling back to location
• add domain - the domain portion of page
• refactor all uses of refererInfo to reflect the new names, but keeping behavior unchanged in most cases. Because of the unfortunate choice of names, it's often not possible for me to guess if uses of referrerInfo.referer should be replaced with ref, location, or topmostLocation. Unless it's very clear (e.g. they are actually setting an ortb2 field for which I know the meaning), I'll just keep the behavior unchanged (but mark it with a TODO comment).

@philipwatson, you are correct - I was not aware that iframes have their document.referrer set to their parent frame.

[bretg]

I would like to see an analysis of use-cases and what value each attribute would have in that scenario.

e.g.

1. Prebid.js is directly in the page example.com. The user got to the page from a bookmark.

• topmostLocation=https://example.com/
• location=https://example.com/
• ref=NULL
• page=https://example.com/
• domain=example.com

2. Prebid.js is run on example.com within an tag manager iframed in from tagmanager.com

• topmostLocation=https://example.com/
• location=https://example.com/ (?)
• ref=NULL
• page=https://example.com/
• domain=example.com

3. Prebid.js is run post-bid on example.com from adserver.com and iframed in from managedservice.com

[dgirardi]

topmostLocation is the location of the parent of the top-most accessible iframe. (the parent is inaccessible, but its location can be inferred by looking at the referrer it sets for the child - mostly, see scenario #6).

So for scenarios 1 and 2 you have the right values.

In all cases, if setConfig({pageUrl}) or <link rel="canonical"> are provided by the publisher, canonicalUrl, page, and domain will respect it (and disregard the location values).

3:

• topmostLocation = https://adserver.com/creative/url
• location = NULL
• ref = NULL
• page = NULL
• domain = NULL

4:

Same scenario as 3, but from a webkit browser. Webkit makes the origin of cross-origin frames available. (Origin is not exactly the same as the URL, but we pick it up nonetheless).

If the actual publisher page is at "www.example.com/article", in Chrome the values are

• topmostLocation = https://www.example.com
• location = https://www.example.com
• ref = NULL
• page = https://www.example.com
• domain = example.com

5:

AMP. I do not know how it works, but Prebid's frame has a global context defined by it, we pick it up. This is what it looks like according to the unit tests:
with the browser being at "https://example-com.amp-cache.example.com/some/page/amp/", we find an automagical window.context = {sourceUrl: "https://example.com/some/page/amp/", canonicalUrl: 'https://example.com/some/page/'}, and the values are

• location = https://example.com/some/page/amp
• topmostLocation = https://example.com/some/page/amp
• ref = NULL
• page = https://example.com/some/page
• domain = example.com

6.

Nested in-frame navigation. I don't know if/how this happens in the real world, but if prebid lives inside a navigable cross-origin iframe, then most values would be incorrect.

If example.com serves a page with <iframe src="https://components.com/some/landing/page />, and that component renders clickable links; with one going to "publisher.com" (which serves Prebid); then clicking on that link would give values such as

• topmostLocation = https://components.com/some/landing/page
• location = https://components.com/some/landing/page
• ref = NULL
• page = https://components.com/some/landing/page
• domain = components.com

This is because when we find the top-most accessible frame, we assume that the referrer to it was the parent frame. That heuristic fails if any navigation happens within frames.

[dgirardi]

7:

Like example #1, prebid lives directly under example.com. But the user gets to it from Google. then the values are

• topmostLocation = https://example.com
• location = https://example.com
• ref = https://www.google.com
• page = https://example.com
• domain = components.com

That is, ref is populated only in the simple case where window.top.document.referrer is accessible.