transfer closed with 16467 bytes remaining to read #44
Comments
|
same issue here, anybody got a workaround for this? |
|
I don't have a solution just hypothesizing at the moment. The file sizes you mentioned strike me as suspiciously close to the default value for postpone_output. If you lower that setting bellow the file size, do you observe a difference? Having just completed some testing of my own (latest version of mod_zip from git here and nginx 1.9.2), I don't seem to be able to reproduce this issue. |
|
We implemented a workaround for this issue where we create the ZIP archive in the application (instead of through mod_zip) when files within the problematic size range are requested. However, the very same issue now occurs with different, larger file sizes using nginx 1.9.5 and the latest mod_zip from this repo. We could not nail it down to a size range as before but start to suspect it being a timing/buffer issue again at another level. The change of postpone_output did not help, in fact when we disable it with "postpone_output 0;", nothing works but a change to the value definitely has an impact. Anyone another solution here? Or an alternative module for creating zip archives on the fly? Our application delivers up to 3'000 files per request and creating zip archives in the backend is a huge performance and timing issue. |
|
Some more information from the debug log (only related to pieces of mod_zip) Afterwards, the response is "stalled" - no more data is sent to the client who suffers from a timeout. Here is part of the debug log (github refuses this data as file attachment...): |
|
Just in case anyone has the same issues... It seems to be related to SSL - when using plain HTTP no such issues arise. We suspect a buffering issue in combination with SSL. Our current workaround is to have another server section without SSL in nginx listening only to localhost for this purpose. The outward facing server section will proxy the zip archive requests there. This creates a double proxy situation (our backend application creates the mod_zip special response body) but at least we have a working mod_zip. |
avodaniel
pushed a commit
to avodaniel/mod_zip
that referenced
this issue
Jan 17, 2016
evanmiller
added a commit
that referenced
this issue
Jan 27, 2016
Flush file header under SSL. Fixes #44
|
Got same issue here with file size in 1444-1459 bytes and without CRC in file list. It seems that nginx waits for data to fit postpone size, but this module wait nginx to send it to the client. Fixed this with b->flush=1 in ngx_http_zip_data_descriptor_chain_link. But it's not good solution. |
|
I have debugged issue with 1444-1459 file size via SSL in nginx-1.22.1 and found that this size is lower than postpone_output(1460) so it is not send by nginx and becomes postponed. Function ngx_http_write_filter(src/http/ngx_http_write_filter_module.c:220) returns NGX_OK to mod_zip and module try to send next piece - trailer_piece(16bytes), now buffer is not lower than postpone_output(1444+16) and ngx_http_write_filter starts sending it in "chain = c->send_chain(c, r->out, limit);" (src/http/ngx_http_write_filter_module.c:294). But this size is still cannot be send, because it is lower than SSL_buffer (16 kbytes) and connection became buffered: "c->buffered |= NGX_SSL_BUFFERED;"(src/event/ngx_event_openssl.c:2705). Call returns in ngx_http_write_filter (src/http/ngx_http_write_filter_module.c:348), but trailer_piece request is not postponed, unlike file_piece, so ngx_http_write_filter returns NGX_AGAIN to mod_zip. Module ends its run and returns NGX_AGAIN to upper level, but with empty chain. Nginx process returns to waiting for events, but there is no events in the chain and connection becomes stalled. |
|
@evanmiller could you help me with that issue or you need more technical information? |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
We use mod_zip with nginx 1.6.3 serving zip files with HTTPS (openssl 1.0.1k).
For a list of 3 files, we can reproduce an error where the client does not receive the full zip archive but experiences a timeout with the error above as not all data announced in "Content-Length" is sent by the server.
The response body (as generated by a rails application is)
The response header is (curl output)
The transfer is stalled after 15437 bytes.
The most strange thing is that the size of the second file matters (the 1386 bytes) but not the content. If we reorder the body lines for mod_zip to have "Media/foo.png" at the end, it works without any issues.
When we change the size of the file (by adding random bytes), it works again. When the file size (random data) is between 1285 and 1386 bytes, the transfer is always stalled.
The error log of nginx (even with debugging enabled) does not show errors but in turn complains about the client having timed out (after 60 seconds, the keep-alive timeout) with "client timed out (110: Connection timed out) while sending response to client"
So both client and server seem to wait for each other and then timeout.
The text was updated successfully, but these errors were encountered: