Native ortb

[FilipStamenkovic]

This pull request is based on initiative to change prebid native to support openRTB standard.

Changes are backwards compatible, meaning with these changes puc can work with openRTB assets and with prebid 'standard' assets.

Included in this PR:

• don't fire impression and click trackers until rendering ad is finished (rendering can fail, for instance loading script hosted under rendererUrl may fail, in that case we don't want to fire events)
• in case ortb format is received in assetResponse, handle impression trackers and click trackers from inside the PUC, no need to be sending post messages to the pbjs so that pbjs can trigger pixels
• added feature for handling clicktrackers defined on asset level (previously that was not possible with pbjs + puc).
• if renderer (loaded from rendererUrl) has defined window.postRenderAd function call it after ad is rendered.

These changes are related to the following pbjs PR: native ortb

[Fawke]

Hi @FilipStamenkovic,

Thank you for putting up this PR. I've gone through most of your change and I had a few questions/comments.

• Do we need to update the macros? Specifically, over here, when we are trying to gather all the placeholder values to request them from Prebid.js, I don't see the new ORTB 2.5 values/placeholders being forwarded.

• Will this work when the publisher defines the native template in the adUnit or in the Ad server?

• It'll really be helpful if you can provide a working test page. I know you had linked one here, but I don't think it'll run locally on my machine, because you're loading up a file used on your local machine, check here. Also, while testing locally, did you try to see if the old way, using Prebid.js Native Asset names, that still works. Again a test page using the old way would be highly appreciated.

• A unit test case for the support for ortb asset in nativeAssetManager.js would be good to have.

• We would need a docs PR.

[FilipStamenkovic]

Hi @Fawke

Thanks for your review.
To answer your questions:

1. it looks like this is related when there isn't pbNativeTagData.requestAllAssets = true;. I'm looking at the docs how puc should behave in this situation and I couldn't find any docs or examples. In all docs that I looked we always suggest using requestAllAssets = true. So, I'm not sure if this is some legacy stuff, or how is it used.
2. I don't think it will work. I forgot that use case, I'll update my code to support adTemplate
3. That link I attached was just to show changes for some random bidder how their bid adapter code would change when we switch to openRTB. I will create some test page to demonstrate openRTB changes.
4. Sure, I'll add tests for that file
5. I opened docs pr

[FilipStamenkovic]

Hi @Fawke I added example page how to test native ortb.

You can find page here

On the same branch there is also a renderer, which can render ortb native ads and 'legacy pbjs' native ads. It's located here.

To test PUC with pbjs changes I suggest using requestly to replace native-render.js that comes from GAM with script you build from this PR.

Also, I prepared python2 script that can be used as 'fake prebid server bidder' for native ads. You can find it bellow.
fake_pbs_python2.txt

[dgirardi]

If I understand this right, when we get ortb2 data in here we don't send any message back to the parent window. But it looks like on the other side we use the tracker message to signify "bid won":

https://github.com/prebid/Prebid.js/blob/7cff54d4ebf3c751c86248e9b2c14485aa41025f/src/secureCreatives.js#L111

I tried your test page to see what happens (I don't know why we would trigger BID_WON in such a way) but I am having trouble getting anything from GAM, the safeframes don't send out any request for me.

[FilipStamenkovic]

You are right, BID_WON event won't be fired in this case.
Thanks for noticing this issue.
My goal was to move firing impression trackers and click trackers from inside PUC.
I missed this effect: when pbjs is sending impression tracker, at the same time it's marking bid as winning.

The way I see it, there are two possible solutions:

• mark bid as winning bid (and send BID_WON) when responding to the allAssetRequest post message
• or, add one more post message when ad is rendered so that pbjs can mark bid as winning bid.

@dgirardi what do you think?

[dgirardi]

First option is probably cleaner especially if you try to figure out the general case; without safeframe we set winning bid at the beginning of rendering; if that makes sense, it also makes sense to do it on the first message we get - could be a single asset request, could be done for this case as well (but I did not check how modules are using BID_WON).

[FilipStamenkovic]

For native, PUC won't fallback to window.pbjs.renderAd even without safe frames.
Then, I'll go with 1st approach. For this I've updated pbjs, more specifically native ORTB PR.

[musikele]

Hi @dgirardi , is there anything else that has to be fixed on this PR ?

[dgirardi]

To test this, I am using the "fake" native response from the test server you gave me @musikele. Here's the test page; it expects PUC to live at localhost:9990.

I am getting Uncaught TypeError: ortb.assets.forEach is not a function, because the fake response packages assets in an object and not an array. I don't know if it's a problem with the response or if that's legitimate.

<html>

<head>
    <link rel="icon" type="image/png" href="/favicon.png">
    <script async src="//www.googletagservices.com/tag/js/gpt.js"></script>
    <script async src="../../build/dev/prebid.js"></script>


    <script>
        var googletag = googletag || {};
        googletag.cmd = googletag.cmd || [];

        var pbjs = pbjs || {};
        pbjs.que = pbjs.que || [];
        var PREBID_TIMEOUT = 10000;

        var date = new Date().getTime();

        function initAdserver() {
            if (pbjs.initAdserverSet) return;

            googletag.cmd.push(function () {
                pbjs.que.push(function () {
                    pbjs.setTargetingForGPTAsync();
                    googletag.pubads().refresh();
                });
            });

            pbjs.initAdserverSet = true;
        }

        // Load GPT when timeout is reached.
        // setTimeout(initAdserver, PREBID_TIMEOUT);

        pbjs.que.push(function () {
            pbjs.setConfig({
                debug: true,
                bidderTimeout: 10000,
                s2sConfig: {
                    accountId: '1',
                    enabled: true, //default value set to false
                    defaultVendor: 'appnexus',
                    bidders: ['appnexus'],
                    timeout: 10000, //default value is 1000
                    adapter: 'prebidServer', //if we have any other s2s adapter, default value is s2s
                },
            });

            var adUnits = [{
                code: 'div-3',
                // sizes: [[1, 1]],
                mediaTypes: {
                    native: {
                        sendTargetingKeys: false,
                        //adTemplate: "<div class=\"sponsored-post\">\r\n  <div class=\"thumbnail\" style=\"background-image: url(##hb_native_image##);\"><\/div>\r\n  <div class=\"content\">\r\n  <h1>\r\n    <a href=\"%%CLICK_URL_UNESC%%##hb_native_linkurl##\" target=\"_blank\" class=\"pb-click\">\r\n  ##hb_native_title##\r\n    <\/a>\r\n   <\/h1>\r\n    <p>##hb_native_body##<\/p>\r\n    \t<div class=\"attribution\">\r\n \t##hb_native_brand##\r\n           \t<\/div>\r\n\t<\/div>\r\n<\/div>",
                        //rendererUrl: window.rendererUrl,
                        title: {
                            required: true,
                            len: 800
                        },
                        image: {
                            required: true,
                            sizes: [989, 742],
                        },
                        sponsoredBy: {
                            required: true
                        }
                    }
                },
                bids: [{
                    bidder: 'appnexus',
                    params: {
                        placementId: '13232354'
                    }
                }]
            }];

            pbjs.setConfig({
                debugging: {
                    enabled: true,
                    intercept: [{
                        when: {
                            bidder: 'appnexus'
                        },
                        then: function() { return {
                            mediaType: 'NATIVE',
                            native: {
                                ortb: JSON.parse(`{"ver": "1.2", "eventtrackers": [{"url": "http://localhost:5500/event?type=impression&component=card", "event": 1, "method": 1}, {"url": "http://localhost:5500/event?type=v50&component=card", "event": 2, "method": 1}], "link": {"url": "https://rtk.io", "clicktrackers": ["http://localhost:5500/event?type=click1&component=card", "http://localhost:5500/event?type=click2&component=card"]}, "assets": [{"data": {"value": "Read More"}, "id": 6}, {"data": {"value": "This is a Prebid Native Creative2."}, "id": 5}, {"id": 1, "img": {"url": "https:\\/\\/vcdn.adnxs.com\\/p\\/creative-image\\/94\\/22\\/cd\\/0f\\/9422cd0f-f400-45d3-80f5-2b92629d9257.jpg", "h": 2250, "w": 3000}}, {"id": 2, "img": {"url": "https:\\/\\/vcdn.adnxs.com\\/p\\/creative-image\\/bd\\/59\\/a6\\/c6\\/bd59a6c6-0851-411d-a16d-031475a51312.png", "h": 83, "w": 127}}, {"id": 4, "title": {"text": "This is a Prebid Native Creative"}}, {"data": {"value": "Prebid.org"}, "id": 3}]}`)
                            }
                        }}
                    }]
                }
            })

            pbjs.addAdUnits(adUnits);

            pbjs.requestBids({
                bidsBackHandler: function (bidResponses) {
                    initAdserver();
                }
            });
        });

    </script>

    <script>
        // GPT setup
        googletag.cmd.push(function () {
            googletag.defineSlot('/41758329/dgirardi-test', ['fluid'], 'div-3').setTargeting('liselect', '4').addService(googletag.pubads());

            googletag.pubads().disableInitialLoad();
            googletag.pubads().enableSingleRequest();
            googletag.enableServices();
        });

    </script>
</head>

<body>


<h2>Native Normal</h2>
<div id='div-3'>
    <script>
        googletag.cmd.push(function() { googletag.display('div-3'); });
    </script>
</div>

</body>

</html>

[musikele]

@dgirardi
I can't observe your same problem. Here's some things I noted about your example:
0. I don't see any error in puc being logged in console.

1. Is debugging module really working? Because I see the call being made to appnexus prebid server; shouldn't it be mocked? Also, the response seens to be sent to the ad unit.
2. appnexus server call is failing because publisherId is a string instead of a number... And even if you change the placementId to be a number, you won't get any response
3. Were you testing the template-inside-adunit scenario ? Because I see rendererUrl and adTemplate keys being commented out. Also, I guess you were testing this scenario to check if it's retro-compatible?
4. You will not test prebid-universal-creative if you don't load an ad unit with the puc in it. If you want, I can provide you with one.

So, it would be very helpful to know what you wanted to exactly achieve and test.

[musikele]

Hi @dgirardi
In the end I found out why I was not seeing debugging working - I was building my pbjs without debugging module in it. Also, there is another debugging.js file that seems to be unrelated, but I was caught off guard.

Do you need anything else to move forward with this?

[dgirardi]

@musikele I'd like a working example, and for release probably also documentation about the new template format. (Or just the docs and I can try building an example myself).

[dgirardi]

with prebid/Prebid.js#8748, this should no longer be done here. I think that might be better in general since it doesn't force you to use the PUC; if we still want to do that here, the logic needs to be adjusted (see that PR) and we need to figure out a migration strategy so that trackers are not fired twice.

[musikele]

Tracker logic was moved to PrebidJS.

[musikele]

Hi @dgirardi . I changed this PR to send messages for tracking to PBJS. Instead of firing trackers, we're now sending a message to PBJS. In PBJS there's this companion PR that will help asset clicktrackers to be fired: prebid/Prebid.js#8905

[dgirardi]

this should default to '' (in light of #171)

[musikele]

If I understand well #171 this function should return an empty string when there's no value for that placeholder... correct me if I'm wrong.

[musikele]

I added some code after that will check for placeholders of assets that are never returned, so these are removed from the template. The scenario I'm thinking is: the request specifies some optional assets, but these assets are not returned in the response; the publisher wants to remove those from the template - so if the publisher is using a ##hb_native_asset_id_100 and asset with id 100 is not part of the response, it gets deleted.

[dgirardi]

How is this related to the switch to ortb - or does this mean that native inside safeframes is currently broken?

[musikele]

The following "if" block is to ensure the following use case: when the creative is not a GAM Native creative but "just" a simple GAM creative (the banner creative). In that case, the GAM creative does not resize itself the same way a GAM native creative would do.

Ideally this change, together with the one that separates the puc into different files, allows to render native ads into GAM banner creatives. But we need them to resize accordingly.

The only choice I did here is to say that the GAM banner creative must have a size of 1x1; in that scenario, the PUC will try to resize the iframe to the native dimentions.

[patmmccann]

@musikele merging 171 created a conflict here

[musikele]

@dgirardi I added some changes to native-trk.js . There's no documentation on how it should work, so my changes are just an educated guess. I think this file was reccomended with the very first version of native, the one that had the template in the ad server creative.

[jsnellbaker]

Can this file and the local.log be removed from the PR? I don't think they need to be in the project.

[musikele]

Thanks for catching that!

[jsnellbaker]

I'm not sure about the system files that were included here (eg. the .tool-versions, .vscode/, etc), but the rest of the code appears good to me from what I saw and tested with my few examples.

[musikele]

@jsnellbaker thanks for catching this ! I'm going to remove the editor files from the PR

[jsnellbaker]

@musikele Can you take a look at the conflicts here, additionally can you double-check the errors in the circleci test? There are some references to some unsupported functions that may be an issue?

Update This branch may just need to be rebased/synced with master. I saw there was another PR recently merged (#178) that updated the browsers.json file to not include IE11 as part of the test suite; the failures were largely in IE11, though there was a test failure that occurred in the other browsers related to:

no placeholders found but requests all assets flag set - adTemplate - openRTB
	TypeError: Object doesn't support property or method 'replaceAll'
	   at replaceORTBAssetsAndLinks (webpack:///src/nativeAssetManager.js:414:5 <- creative.js:18569:5)
	   at replace (webpack:///src/nativeAssetManager.js:435:7 <- creative.js:18592:7)
	   at replaceAssets (webpack:///src/nativeAssetManager.js:327:9 <- creative.js:18452:9)
	   at windowListener (webpack:///src/messaging.js:38:17 <- creative.js:17850:9)
	   at Anonymous function (webpack:///test/spec/nativeAssetManager_spec.js:58:5 <- creative.js:31860:5)
	   at sendMessage (webpack:///src/messaging.js:24:13 <- creative.js:17836:7)
	   at requestAllAssets (webpack:///src/nativeAssetManager.js:252:5 <- creative.js:18377:5)
	   at loadAssets (webpack:///src/nativeAssetManager.js:199:7 <- creative.js:18324:7)
	   at Anonymous function (webpack:///test/spec/nativeAssetManager_spec.js:396:7 <- creative.js:32146:7)

[musikele]

Hi @jsnellbaker, just completed the rebase and yes, the configuration for CircleCI was updated.