Stabilize circleci tests

[mkendall07]

Type of issue

21 of the last 30 builds failed on master. We need to pinpoint and fix the issues to make tests stable again.

https://circleci.com/gh/prebid/Prebid.js/tree/master

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[snapwich]

I've done some initial digging into the test issues; some of these issues I'm seeing could possibly be related to external requests being successfully made during run of the test suite, especially the ones that successfully load third-party code and run it.

I've categorized all the external requests I see being made by the test suite by the type of request. We should probably take efforts to see how we can stop these requests from being made as part of the test running.

I know I did some initial work by mocking request makers, such as loadExternalScript but still some are getting through; specifically one I looked at got through because loadExternalScript is only stubbed during a test (using before/after hooks) and this module is actually making external requests while the module is being loaded, not during a test.

A lot are ajax requests using our internal module that I thought we stubbed? ...but are getting through somehow?

The currency module is being loaded by some spec files (besides the currency module spec file) and so requests are being made to the currency file.

Some adapters are appending third-party code and running it outside our stubbed loadExternalScript which I've marked as custom script appending (without loadExternalScript).

The adapters loading and running external code seem the most problematic since we don't know what they're doing and they could technically be doing anything, including changing over time, and affecting our tests randomly.

loadExternalScript
	- modules/adagioBidAdapter.js
 		- http://script.4dex.io/localstore.js # from firing loadExternalScript outside of test
 		- https://script.4dex.io/adagio.js # third-party script loaded from localstore.js
 		- http://www3.smartadserver.com/ac?pgid=1177189&insid=9202901&tmstp=8180072780&out=js # request initiated by adagio.js script above
	- modules/djaxBidAdapter.js
		- https://acdn.adnxs.com/video/outstream/ANOutstreamVideo.js # installing custom renderer in interpretResponses
	- modules/vubleBidAdapter.js
		- http://localhost:9876/vuble_renderer.js # installing custom renderer from mocked responses in vubleBidAdapter_spec.js

ajax requests
	- modules/djaxBidAdapter.js
		- 200 xhr: https://demo.reviveadservermod.com/headerbidding_adminshare/www/admin/plugins/Prebid/tracking/track.php # firing tracking requests in interpretResponses, seems to be happening multiple times maybe asynchronously?
	- modules/digiTrustIdSystem.js
		- 200 xhr: https://prebid.digitru.st/id/v1 # request made from getUser calls originiating from tests in digiTrustIdSystem_spec.js 
	- modules/browsiRtdProvider.js
		- 404 xhr: http://testurl.com/prebid?sk=testKey&sw=1920&sh=1080&url=http%253A%252F%252Flocalhost%253A9876%252Fdebug.html&us=%7B%7D&r=http%3A%2F%2Flocalhost%3A9876%2Fdebug.html&at=Karma%20DEBUG%20RUNNER # requests initiated in getPredictionsFromServer from tests in realTimeModule_spec.js
	- modules/vrtcalBidAdapter.js
		- 404 https://vrtcal.com/faketracker # onBidWon "pixel"

currency module related ajax requests
	- modules/adformBidAdapter.js
		- 200 xhr: https://cdn.jsdelivr.net/gh/prebid/currency-file@1/latest.json?date=20191218 # initializing currency module and testing in adformBidAdapter_spec.js
	- modules/readpeakBidAdapter.js
		- 200 xhr: https://cdn.jsdelivr.net/gh/prebid/currency-file@1/latest.json?date=20191218 # initializing currency module and buildRequests tests in readpeakBidAdpter_spec.js
	- modules/smilewantedBidAdapter.js
		- 200 xhr: https://cdn.jsdelivr.net/gh/prebid/currency-file@1/latest.json?date=20191218 # initializing currency module in tests in smilewantedBidAdapter_spec.js

custom script appending (without loadExternalScript)
	- modules/browsiRtdProvider.js
		- 404 script: http://localhost:9876/scriptUrl.com # initiated by addBrowsiTag from tests in realTimeModule_spec.js
	- modules/showheroes-bsBidAdapter.js
		- 200 script: https://static.showheroes.com/publishertag.js # initiated by createOutstreamEmbedCode as part of custom renderer code in tests testing outstream in showheroes-bsBidAdapter_spec.js
		- 200 script: https://static.showheroes.com/pubtag.js?rnd=5274587599225995 # request initiated by publishertag.js script above
	- modules/videoNowBidAdapter.js
		- 404 script: https://localhost:8086/vn_init.js?profileId=1 # code loaded for custom renderer
		- 404 script: https://localhost:8086/vn_init.js?div=1&profileId=1 # code loaded for custom renderer

img pixel
	- modules/adponeBidAdapter.js
		- 502 text/html: https://rtb.adpone.com/prebid/analytics?q=e30= # firing pixel on spec.onBidWon({}) call in adponeBidAdapter_spec.js
	- modules/emoteevBidAdapter.js
		- (failed): https://prebid.emoteev.xyz/api/ad_event.json?eventName=on_bid_won&pubcId=null&bidId=2b8de6572e8193&cache_buster=10747bd32e34b349 # firing pixel in onBidWon for onBidWon test in emoteevBidAdapter_spec.js
	- modules/mgidBidAdapter.js
		- 404 text/plain: http://localhost:9876/nurl&s=0.66 # onBidWon pixels
		- 404 text/plain: http://localhost:9876/burl&s=0.66 # onBidWon pixels
	- modules/telariaBidAdapter.js
		- 200 gif: https://events.tremorhub.com/diag?srcPageUrl=http%3A%2F%2Flocalhost%3A9876%2Fdebug.html%23pubmaticTest%3Dtrue%23pubmaticTest%3Dtrue&adCode=ssp-!demo!-lufip&supplyCode=ssp-demo-rm6rh&mediaId=MyCoolVideo&hb=1&evt=TO # onTimeout pixel
	- modules/videoNowBidAdapter.js
		- (failed) https://fakedomain.nld/?price=10 # onBidWon pixel

[snapwich]

The issues I outlined above are all fixed with the merge of #4687. There are possibly other issues still present involving race conditions (there seem to be some issues in the userId tests with intermittent failures) as well as some issues with circle ci failing on time limit. The later could be because of occasionally slow running tests...? but probably more likely, something is occasionally causes the test suite to hang (open handlers, etc).

[bjorn-lw]

With every last PR I've made (not that many admittedly) I get failed circleci tests not related to my changes. This time it's the Rubicon Analytics adapter that fails.

[mkendall07]

hey @Fawke how is this going?

[Fawke]

hey @Fawke how is this going?

Hi @mkendall07, I just created a PR. #4840
Please have a look when you get some time.

[ChrisHuie]

closing in relation to pr #4840 #4840