Mashape Analytics URI bug

[subnetmarco]

Aug 11 16:29:59 91.202.65.74 Aug 11 23:29:59 8851f804c05e nginx: 2015/08/11 23:29:59 [error] 54#0: [lua] buffer.lua:164: [mashape-analytics] socket server refused the batch. Status: (400) Error: (["[1] har.log.entries.0.request.url must be uri format"]), context: ngx.timer, client: 188.196.56.12, server: 0.0.0.0:8000

[SGrondin]

I added better debugging, so now it'll say [1] har.log.entries.0.request.url must be uri format in [whatever] where [whatever] is the incorrect value.

[subnetmarco]

@SGrondin did you deploy your fix? Because the log message is still the same.

[thibaultcha]

As of today, the error still says:

[3] har.log.entries.0.request.url must be uri format

So there is no way to know what it really contains.

[subnetmarco]

Now I can see the URI error properly:

Aug 26 16:16:14 52.18.160.195 Aug 26 23:14:58 ca6a250858df nginx: 2015/08/26 23:14:58 [error] 37#0: [lua] buffer.lua:164: [mashape-analytics] socket server refused the batch. Status: (400) Error: (["[0] har.log.entries.0.request.url must be uri format in 'http://square-core-qa.elasticbeanstalk.com/api/v1/customer/1_0_0/ideapiu/registrationservice/Dimitri/De Franciscis/12345'","[1] har.log.entries.0.request.url must be uri format in 'http://square-core-qa.elasticbeanstalk.com/api/v1/customer/1_0_0/ideapiu/registrationservice/Dimitri/De Franciscis/12345'"]), context: ngx.timer, client: 5.157.103.239, server: 0.0.0.0:8000

and

Aug 26 16:16:14 52.18.160.195 Aug 26 23:14:58 ca6a250858df nginx: 2015/08/26 23:14:58 [error] 37#0: [lua] buffer.lua:164: [mashape-analytics] socket server refused the batch. Status: (400) Error: (["[0] har.log.entries.0.request.url must be uri format in 'http://cart.lmcloud.aws/cart/v1/remove/dKonDHcKn73Z1UFm9TSFRrnmQe2m8RZz/[object Object]'"]), context: ngx.timer, client: 93.33.106.147, server: 0.0.0.0:8000

and

4:16:14.000 PM  
Aug 26 16:16:14 91.202.66.112 Aug 26 23:16:14 bbeb37766863 nginx: 2015/08/26 23:16:14 [error] 52#0: [lua] buffer.lua:164: [mashape-analytics] socket server refused the batch. Status: (400) Error: (["[1] har.log.entries.0.request.url must be uri format in 'http://haproxy.marathon.mesos/spored/article/get/ekskluziv/tuja-scena/foto-ali-voscena-nicki-minaj-dobiva-kar-si-za sluzi.html'"]), context: ngx.timer, client: 91.185.217.224, server: 0.0.0.0:8000

to give a few.

[thibaultcha]

Here are several values reported as invalid:

http://haproxy.marathon.mesos/spored/article/get/ekskluziv/tuja-scena/foto-ali-voscena-nicki-minaj-dobiva-kar-si-za sluzi.html
http://square-core-qa.elasticbeanstalk.com/api/v1/customer/1_0_0/ideapiu/registrationservice/Dimitri/De Franciscis/12345
http://cart.lmcloud.aws/cart/v1/remove/dKonDHcKn73Z1UFm9TSFRrnmQe2m8RZz/[object Object]

Any idea why? Maybe the space? I will check the validation code later.

[thibaultcha]

It is visibly the space (see the regex). I will encode the URI but lua-nginx only offers escape_uri which is meant for URI components, and destroys the URI, so that will probably have to be custom.

[thibaultcha]

On a second thought, I am not sure I should touch the URI at all. After all, it is the client who requested it without escaping it (and using [] which are reserved characters). I feel like the socket should accept those values, as they might not be valid URIs, but that is what was requested and as an API provider, I want to see those erroneous requests (and potentially reach out to my users to point the issue to them). @SGrondin @ahmadnassri thoughts as mashape analytics maintainers?

[ahmadnassri]

this is more of a HAR format / JSON validation issue than it is Mashape analytics ... we could expand the allowed character set ... maybe

actually surprised nginx even allows it?

[subnetmarco]

So does this issue bounce back to @SGrondin (or @kennethklee)?

[ahmadnassri]

bounce back to you first, do you want to follow the RFC specs or not?

[thibaultcha]

It's not so much about the specs. The client did not respect the specs, we have to reflect it in the analytics logs.

[ahmadnassri]

that I agree with, and I believe latest build has error logging output in
detail on the server

[thibaultcha]

No, I meant the logs stored by analytics. I mean on analytics' dashboard.

[ahmadnassri]

but we are discarding the input if its invalid ... not storing anywhere, so no way to message the user about it ... is that what you mean? otherwise I'm not following

[thibaultcha]

I mean you should not discard it, and you should store it.

[ahmadnassri]

that could become very dangerous since clients are paying for analytics data retention and storage, if somebody is sending malformed requests they could easily take over a user's quota and flood the system ..

that said, this conversation is about Analytics then, so lets take the discussion there, unless there's something specific to Kong, otherwise, lets close this issue,

[thibaultcha]

They would not take more space because of accepting invalid URIs. They could as well flood the system with valid URIs?

[thibaultcha]

Moved to Mashape Analytics.

[ahmadnassri]

actually, all of the examples provided above of the errors are because of the space in the URL, not the [] characters ... seems like the analytics plugin in Kong is not properly parsing & encoding the URI before setting it in the JSON object.

[thibaultcha]

Damn it, ngx_lua returns those values unescaped. And the only way to escape them back is with escape_uri which break the URI (also escaping / separating components)

[ahmadnassri]

Damn it, ngx_lua returns those values unescaped. And the only way to escape them back is with escape_uri which break the URI (also escaping / separating components)

I had the same problem with Node / PHP ... I end up parsing and re-constructing the URI every time to be safe.

[thibaultcha]

So while writing the serializer I respected the ALF schema (where request.url does not contain the querystring) but the HAR specs do contain the querystring.

Hence I used ngx.var.uri -> without the querystring (but unescaped). If we just use request_uri, it contains the querystring but it is the raw URI, as made by the client.