[mashape-analytics] Why are chunked request over 10kb not being sent?

[therebelrobot]

I made sure I am sending the bodies via config.log_body. It sends chunked request bodies for the exact same file/endpoint when it's below 10kb, then add a line and it sends nothing. Doesn't matter what mimetype is being sent, I've seen it fail on javascript, csv, jpeg, and png bodies. I'll see what I can do with it.

[thibaultcha]

You are hitting the limit set by the default proxy_buffer_size directive.

When a request body exceeds this limit, it is written to a temporary file and flushed from the Nginx buffers. The plugin does not support reading this value from the file when the body exceeds this limit.

By increasing this value:

proxy_buffer_size 1m;

You will allow bigger bodies to be stored in the buffers at the same time, and the plugin will be able to read them. Please stay aware of the plugin's config.max_sending_queue_size limit (10MB by default) which will also greatly influence wether ALFs will be buffered or not.

Hence, I wouldn't consider this as a bug, eventually an enhancement to be made by, for example, adding a note in the plugin's documentation. Secondly, this error (should probably be downgraded to a warning) is logged when the plugin has to read from a tmp file:

[mashape-analytics] cannot read request body from temporary file. Try increasing the client_body_buffer_size directive.

Unfortunately, it will only be logged when the content-type is form-encoded. Another improvement would be to log this on a failure of ngx.req.get_body_data() but sadly it seems there is no error returned by this function, so no certain way to know if the plugin couldn't read the body because it was in a tmp file.

---

Side note: thank you for not labelling issues as bugs until confirmed to be so. At the very best this is an enhancement over the documentation/logging.

[therebelrobot]

Thanks for the update. @ahmadnassri had added the tag, I had just assigned it to myself to work on :)

[therebelrobot]

I received the following error when starting kong after adjusting proxy_buffer_size, preventing startup:

nginx: [emerg] "proxy_busy_buffers_size" must be equal to or greater than the maximum of the value of "proxy_buffer_size" and one of the "proxy_buffers" in /Users/therebelrobot/git/mashape/kong/nginx_tmp/nginx.conf:152

Since both proxy_buffers and proxy_busy_buffers_size was listing double what proxy_buffer_size was (256k vs 128k respectively), I'll set those to 2m and see if it works.

EDIT: that adjustment allowed Kong to start. Will test the bodies with that change, and will update with more info.

[therebelrobot]

I tried several different settings for proxy_buffer_size, starting with 1m, and even got as high as 256m, with proxy_buffers and proxy_busy_buffers_size scaling accordingly, but still, nothing over 10kb is getting through. Here's the current setting I've put in ./kong_DEVELOPMENT.yml:95-99:

    # Proxy Settings
    proxy_buffer_size 256m;
    proxy_buffers 4 512m;
    proxy_busy_buffers_size 512m;
    proxy_ssl_server_name on;

[thibaultcha]

Oops sorry, no idea how I could have linked to the wrong directive, but the correct one here is client_body_buffer_size, not proxy_buffer_* which has to do with the response buffers.

[therebelrobot]

Ooooooh, ok, that makes sense then. I tweaked with that, adding the following line to # Other Settings in the conf:

client_body_buffer_size 200m;

And added the following settings to the plugin itself:

I realized that that buffer size limit was being set not only by client_body_buffer_size but by the number of ALFs I was allowing to be cached (so if I set that to 100m and I had config.batch_size set to 100, then each alf would have a max of 1m each available for content.) I also adjusted the config.max_sending_queue_size while I was at it, since the documentation seemed to suggest it has a role to play in it. I fiddled around with a few different numbers to get to what I needed.

tl;dr I'm now able to see the data come through as expected.

Is it possible to set that nginx setting via plugin config? From what I saw it seems that the config only affects the lua script for the plugin, rather than the nginx configuration itself. Is there any easy way of implementing customization of that setting?

[thibaultcha]

Is it possible to set that nginx setting via plugin config?

Nope. And no, no easier way than to edit the nginx configuration.

[subnetmarco]

Possibly related to #701

[thibaultcha]

Considering this answered.

[rca]

For anyone else landing here, tl;dr: check your disk space.

I started searching for reasons why large-ish JSON responses were being truncated via curl with the error (18) transfer closed with outstanding read data remaining.

Searching Google for this led me here.

The answer for my case ended up being a full disk and it appears the large payload was causing buffering, but there was no space to buffer, so 💣 .