PrebidServerBidAdapter issues with First Party Data

[pm-harshad-mane]

Type of issue

There are some issues with the latest (4.30) implementation of FPD in PrebidServerBidAdapter,
Issues are with the commit 8c686a9#diff-3375f581a781deb711719820144dc635c7852299259ba806911532c829349687

Description

Over-writing instead of Merging.

In PrebidServerBidAdapter, request.site is over-written with FPD site object hence the pre-populated request.site object is lost, we should have used utils.mergeDeep than using utils.deepsetValue, same issue is fore request.user object.
Refer code: https://github.com/prebid/Prebid.js/blob/master/modules/prebidServerBidAdapter/index.js#L772 and https://github.com/prebid/Prebid.js/blob/master/modules/prebidServerBidAdapter/index.js#L775

If we simply pass the FPD config as following,

pbjs.setConfig({
    "ortb2": {
        "site": {
            "name": "example",
        }
    }
});

this is how the request to PrebidServer looks, returning HTTP 400 Bad request

{
  "id": "ddbe7943-60f4-47e3-9e05-f365d5077452",
  "source": {
    "tid": "ddbe7943-60f4-47e3-9e05-f365d5077452"
  },
  "tmax": 1000,
  "imp": [
    {
      "id": "/19968336/header-bid-tag-0",
      "ext": {
        "appnexus": {
          "use_pmt_rule": false,
          "placement_id": 13144370
        },
        "data": {
          "adserver": {
            "name": "gam",
            "adslot": "/19968336/header-bid-tag-0"
          },
          "pbadslot": "/19968336/header-bid-tag-0"
        }
      },
      "banner": {
        "format": [
          {
            "w": 300,
            "h": 250
          },
          {
            "w": 300,
            "h": 600
          }
        ]
      }
    },
    {
      "id": "/19968336/header-bid-tag-1",
      "ext": {
        "appnexus": {
          "use_pmt_rule": false,
          "placement_id": 13144370
        },
        "data": {
          "adserver": {
            "name": "gam",
            "adslot": "/19968336/header-bid-tag-1"
          },
          "pbadslot": "/19968336/header-bid-tag-1"
        }
      },
      "banner": {
        "format": [
          {
            "w": 728,
            "h": 90
          },
          {
            "w": 970,
            "h": 250
          }
        ]
      }
    }
  ],
  "test": 1,
  "ext": {
    "prebid": {
      "auctiontimestamp": 1615918349413,
      "targeting": {
        "includewinners": true,
        "includebidderkeys": false
      }
    }
  },
  "site": {
    "name": "example"
  },
  "device": {
    "w": 1680,
    "h": 324
  }
}

PBS is not accepting imp[0].ext.data

With the above example we get the following error returned in PBS response

Invalid request: request.imp[0].ext contains unknown bidder: data. Did you forget an alias in request.ext.prebid.aliases?

Root cause for the issue is imp object in request, data object is placed along with appnexus in imp[0].ext

"imp": [
    {
      "id": "/19968336/header-bid-tag-0",
      "ext": {
        "appnexus": {
          "use_pmt_rule": false,
          "placement_id": 13144370
        },
        "data": {
          "adserver": {
            "name": "gam",
            "adslot": "/19968336/header-bid-tag-0"
          },
          "pbadslot": "/19968336/header-bid-tag-0"
        }
      },
      "banner": {
        "format": [
          {
            "w": 300,
            "h": 250
          },
          {
            "w": 300,
            "h": 600
          }
        ]
      }
    }
]

PBS returns HTTP 400 Bad request even for the FPD example shown in documentation

Refer: https://docs.prebid.org/features/firstPartyData.html#first-party-data---prebidjs
FPD Config used here is

pbjs.setConfig({
  "ortb2": {
    "site": {
      "name": "example",
      "domain": "page.example.com",
      "cat": [
        "IAB2"
      ],
      "sectioncat": [
        "IAB2-2"
      ],
      "pagecat": [
        "IAB2-2"
      ],
      "page": "https://page.example.com/here.html",
      "ref": "https://ref.example.com",
      "keywords": "power tools, drills",
      "search": "drill",
      "content": {
        "userrating": 4,
        "data": [
          {
            "name": "www.dataprovider1.com",
            "ext": {
              "taxonomyname": "iab_content_taxonomy"
            },
            "segment": [
              {
                "id": "687"
              },
              {
                "id": "123"
              }
            ]
          }
        ]
      },
      "ext": {
        "data": {
          "pageType": "article",
          "category": "repair"
        }
      }
    },
    "user": {
      "yob": 1985,
      "gender": "m",
      "keywords": "a,b",
      "data": [
        {
          "name": "dataprovider.com",
          "ext": {
            "taxonomyname": "iab_audience_taxonomy"
          },
          "segment": [
            {
              "id": "1"
            }
          ]
        }
      ],
      "ext": {
        "data": {
          "registered": true,
          "interests": [
            "cars"
          ]
        }
      }
    }
  }
});

PBS request in this case,

{
  "id": "03b44c86-6a4f-4137-940c-b2b797631f27",
  "source": {
    "tid": "03b44c86-6a4f-4137-940c-b2b797631f27"
  },
  "tmax": 1000,
  "imp": [
    {
      "id": "/19968336/header-bid-tag-0",
      "ext": {
        "appnexus": {
          "use_pmt_rule": false,
          "placement_id": 13144370
        },
        "data": {
          "adserver": {
            "name": "gam",
            "adslot": "/19968336/header-bid-tag-0"
          },
          "pbadslot": "/19968336/header-bid-tag-0"
        }
      },
      "banner": {
        "format": [
          {
            "w": 300,
            "h": 250
          },
          {
            "w": 300,
            "h": 600
          }
        ]
      }
    },
    {
      "id": "/19968336/header-bid-tag-1",
      "ext": {
        "appnexus": {
          "use_pmt_rule": false,
          "placement_id": 13144370
        },
        "data": {
          "adserver": {
            "name": "gam",
            "adslot": "/19968336/header-bid-tag-1"
          },
          "pbadslot": "/19968336/header-bid-tag-1"
        }
      },
      "banner": {
        "format": [
          {
            "w": 728,
            "h": 90
          },
          {
            "w": 970,
            "h": 250
          }
        ]
      }
    }
  ],
  "test": 1,
  "ext": {
    "prebid": {
      "auctiontimestamp": 1615920270104,
      "targeting": {
        "includewinners": true,
        "includebidderkeys": false
      }
    }
  },
  "site": {
    "name": "example",
    "domain": "page.example.com",
    "cat": [
      "IAB2"
    ],
    "sectioncat": [
      "IAB2-2"
    ],
    "pagecat": [
      "IAB2-2"
    ],
    "page": "https://page.example.com/here.html",
    "ref": "https://ref.example.com",
    "keywords": "power tools, drills",
    "search": "drill",
    "content": {
      "userrating": 4,
      "data": [
        {
          "name": "www.dataprovider1.com",
          "ext": {
            "taxonomyname": "iab_content_taxonomy"
          },
          "segment": [
            {
              "id": "687"
            },
            {
              "id": "123"
            }
          ]
        }
      ]
    },
    "ext": {
      "data": {
        "pageType": "article",
        "category": "repair"
      }
    }
  },
  "device": {
    "w": 1680,
    "h": 324
  },
  "user": {
    "yob": 1985,
    "gender": "m",
    "keywords": "a,b",
    "data": [
      {
        "name": "dataprovider.com",
        "ext": {
          "taxonomyname": "iab_audience_taxonomy"
        },
        "segment": [
          {
            "id": "1"
          }
        ]
      }
    ],
    "ext": {
      "data": {
        "registered": true,
        "interests": [
          "cars"
        ]
      }
    }
  }
}

PBS responds with HTTP 400 Bad Request.
PBS response is

Invalid request: json: cannot unmarshal number into Go struct field Content.site.content.userrating of type string

[pm-harshad-mane]

Please note that the changes were made in multiple bidders along with PBS-bid-adapter, we should check if all updated bidders are working as expected.

[pm-harshad-mane]

Validity of the data

Here I tried to pass FPD

                pbjs.setConfig({
                    "ortb2": {
                        "site": {
                            "buzzinga": "example",
                        }
                    }
                });

The PBS-bid-adapter passes the buzzinga property in request to PBS.
We are over-writing request.site with whatever we get from FPD, we should make sure that we pass only the valid keys in ORTB request.

Data type check for valid keys

pbjs.setConfig({
                    "ortb2": {
                        "site": {
                            "domain": 22,
                        }
                    }
                });

The PBS-bid-adapter passes the domain property as integer in request to PBS.
Response from PBS
Invalid request: json: cannot unmarshal number into Go struct field Site.site.domain of type string
such mistakes might happen accidentally by users while setting up FPD and it will result in loss of opportunity to bid (revenue impact), better if we type-check the FPD data that we are passing to bidders.

[bretg]

Thanks for catching these @pm-harshad-mane .

Over-writing instead of Merging

Opened a ticket for @mmoschovas to address this part.

PBS is not accepting imp[0].ext.data

This is supported in PBS-Java. Opened a discussion with @SyntaxNode to address the issue in PBS-Go.

PBS returns HTTP 400 Bad request even for the FPD example shown in documentation

Doc example fixed. userrating value turned to a string.

Validity of the data

We did not want to weigh down core PBJS with a hundred FPD validations. That will happen in the FPD module tracked by #6099

[bretg]

@pm-harshad-mane - can you inspect the fix for the PBS adapter? #6428

[pm-harshad-mane]

PBS is not accepting imp[0].ext.data
This is supported in PBS-Java. Opened a discussion with @SyntaxNode to address the issue in PBS-Go.

@bretg, @mmoschovas I feel in this scenario as PBS-Go breaks and ignores whole request, we should not pass imp[0].ext.data from PBS-bid-adapter until PBS-Go starts supporting it and also inform pubs to make sure they have respective PBS-GO and PBJS versions to make it work.

[pm-harshad-mane]

PBS is not accepting imp[0].ext.data
This is supported in PBS-Java. Opened a discussion with @SyntaxNode to address the issue in PBS-Go.

@SyntaxNode am assuming you will mark the data key-word as a known not-a-bidder in the Go Code. Should we move all bidders from imp[0].ext to under imp[0].ext.bidders? like instead of imp[0].ext.appnexus it will be imp[0].ext.bidders.appnexus. This will avoid such problems in future. As ext are made for adding unknown params, next time if we add anything under imp[0].ext.new-key we will not need changes on Server side code.

[bretg]

@pm-harshad-mane - there's a huge difference in the release schedules between PBJS and PBS. Pubs will get on PBJS 4.31 and be stuck there for years, while PBS host companies will upgrade in the next couple of weeks and everything will be good. I think PBJS should continue sending imp.ext.data. We can place a note somewhere that PBJS 4.30 and 4.31 are temporarily incompatible with PBS-Go, but I'm sure that situation will be resolved soon.

As for moving bidders, yes, that's been done with prebid/prebid-server#1478... both PBS-Go and PBS-Java now support imp[].ext.prebid.bidder.BIDDER. So the pbsBidAdapter could be updated to make use of this.

[pm-harshad-mane]

Yup @bretg , I agree that PBS host companies will upgrade faster than pubs using PBJS.