ortb2 FPD object contains a ortb 2.6 field

[jbartek25]

Type of issue

Bug

Description

enrichmentFpdModule with the help of sua.js adds ortb 2.6 field device.sue into ortb2 object.
Improve Digital bidder adapter was recently migrated to benefit from the common ortbConverter logic which fills in all the usual fields in a oRTB bid request. Now, Improve Digital ad server endpoint is oRTB 2.5 and rejects any request containing device.sue.
I believe most (all) SSPs with oRTB endpoints are on oRTB 2.5 and the ortbConverter should only set valid 2.5 fields. This is a big issue for us. I would think our adapter is not the right place for the fix and it should be either in ortbConverter or elsewhere in the core/common logic. What do you think? @dgirardi

[dgirardi]

The 2.6 spec states:

implementers need not specifically treat the minor version of OpenRTB 2.x as significant, in most circumstances. For example:
Bidders and exchanges must tolerate receiving new or unexpected fields and enumerated list
values gracefully, treating them as unknown or ignoring them

However, that section is not in the 2.5 spec, so it's not clear if that is recommended only for 2.6 onwards (the wording seems to be general to 2.x).

I am not sure of what ortbConverter should do, it's not trivial to enforce 2.5 - because anything provided directly by the pub is transferred almost verbatim into the request.

As an immediate fix I believe we should start with removing sua from the Improve Digital adapter. Long term I think the best option would be for your endpoint to tolerate extra fields, as we plan to adopt more 2.6 fields over time; however, we can think about building a version toggle into ortb2Converter itself.

[dgirardi]

Both of the other adapters that are currently behind the common logic - Prebid Server and OpenX - do not mind getting extra fields. It's hard to gauge now if this problem will be more common as more adapters convert.

If it is to be addressed by core, my preference would be to add another library utility that takes in an ortb2 request and strips out all fields that are not defined in the 2.5 spec.

[patmmccann]

My understanding from @SyntaxNode was that extra fields should be ignored, and that is part of the 2.x spec to not fail on unrecognized fields. I'm having trouble finding a reference on this.

[jbartek25]

@dgirardi @patmmccann thanks for your input. I'll discuss with our ad server team regarding relaxing the 2.5 schema. I'll get back to you.

[jbartek25]

As mentioned in #9251, a temporary fix to handle device.sue will be applied on our end so the immediate issue will be resolved.

Regarding the general point of unknown fields in ortb2 object:

The conclusion of an internal Improve Digital discussion is that we can relax our schema validation in our SSP but it will have to be thoroughly tested if all DSPs that we work with are ok with unknown fields. If not, then we would need to implement filter to remove extra fields from DSP requests. This will take same time and until then we will be vulnerable and will face the same issue of validation failing and requests rejected with any new ortb 2.6 field added. Since Improve Digital is the first and currently the only case, it's hard for me to propose what action should be taken in Prebid core.

What I find confusing though is that any FPD docs don't make it clear the fields can be from any 2.x version. If anything, there's a note in https://docs.prebid.org/features/firstPartyData.html saying:

Fields that are meant to be standard OpenRTB 2.5 should be in ortb2.site or ortb2.user.

The above is either not true (site and user can contain also fields from other ortb versions) or all objects inside ortb2 should adhere to 2.5, not just site and user.

[bretg]

@jbartek25 - FWIW, Prebid Server has a similar assumption -- that bidders will ignore unknown ORTB fields. We think this is a desirable approach for endpoints because the IAB plans to release updates to 2.6 frequently, which means new fields more often than before.

[dgirardi]

After discussing this offline, we agreed that it makes sense for Prebid to offer adapters an option to "downgrade" 2.6 to 2.5. This means a utility that removes or renames 2.6 fields (for example, renaming regs.gdpr to regs.ext.gdpr).

[jbartek25]

FWIW, Prebid Server has a similar assumption -- that bidders will ignore unknown ORTB fields

@bretg The PBS itself (at least the Java version I tested) ignores the unknown fields, but then the server-side SSP requests sent from PBS have device.sue (or any other orb 2.6 field I checked) removed so PBS must be doing some filtering. Why?

[bretg]

Hmm. You're right... PBS-Go and PBS-Java are handling this differently. When I add device.sua to a request, PBS-Java does not forward to rubicon, but PBS-Go does. Will check with the teams, but in any case, the PBS committee re-affirmed today that we would like bidder adapters and endpoints to be responsible for handling unknown fields in the way that seems best suited for them: either specifically remove or ignore.

[patmmccann]

@jbartek25 there is a very good chance you will start seeing regs.gpp if publishers start doing setConfigs to see it. You might want to handle that explicitly in your adapter or backend.

[jbartek25]

Thanks for the heads-up. Improve Digital SSP will make changes to handle unkown fields in all our endpoints (both PBJS and PBS) soon. In the meantime, we will probably just handle new fields as we see them as any adapter fix would be temporary and would have to be made in both PBS and PBJS adapters.