Requests stuck in certain cases #226

FlorinAsavoaie · 2015-09-29T13:49:57Z

I raised nginx bug 801 for this but actually it seems to be an issue with Naxsi.

Long story short, here's the comment from the nginx developers:

"[...] it looks like a 3rd party module have read a request body in the rewrite phase (not something
normally done, the body is expected to be read only at content phase), and failed to properly restore
request handlers after this. That is, it's a naxsi bug."

buixor · 2015-09-30T09:47:57Z

Hi,

Thanks for the bug-report.
In the configuration you pasted in your nginx ticket, I do not see any naxsi configuration, did you strip it on purpose, or does it means that it happens even if naxsi's code is not explicitly enabled ?

regards,

FlorinAsavoaie · 2015-09-30T09:57:41Z

I just stripped it but it happens if I enable it for the location only. If naxsi is not enabled for the location but just built into nginx, the bug does not occur. However, it doesn't matter if it is in learning mode or not.

buixor · 2015-09-30T09:58:24Z

Can you please provide a full non strippped config so I can reproduce it ?

FlorinAsavoaie · 2015-09-30T10:03:41Z

location /myapp {
  # Naxsi config
  LearningMode;
  SecRulesEnabled;

  DeniedUrl "/RequestDenied";

  CheckRule "$SQL >= 8" BLOCK;
  CheckRule "$RFI >= 8" BLOCK;
  CheckRule "$TRAVERSAL >= 4" BLOCK;
  CheckRule "$EVADE >= 4" BLOCK;
  CheckRule "$XSS >= 8" BLOCK;

  auth_request /auth;
  proxy_pass http://some-upstream;
}

location = /auth {
  proxy_method GET;
  proxy_pass_request_body off;
  proxy_pass_request_headers off;
  if ($cookie_auth) {
    set $app_auth "Token $cookie_auth";
  }
  if ($http_authorize) {
    set $app_auth $http_authorize;
  }
  proxy_set_header Authorize $app_auth;
  proxy_set_header Host $host;
  proxy_set_header Content-Length "";
  proxy_cache proxyCacheAuth;
  proxy_cache_key $remote_addr$host$app_auth;
  proxy_cache_valid any 15m;
  proxy_pass http://authupstream/random/url;
}

buixor · 2015-09-30T10:53:05Z

thanks for the details, I will look into it

buixor · 2015-10-09T10:38:58Z

Hello,

Sorry, but I did not manage to reproduce the issue, can you give me a hand ?
(I use naxsi dev's env, so we have something reproducible)

cp -R .../nginx/nginx-1.9.1 /tmp/
cd naxsi_src
make re deploy

(note: this deploys a dummy nginx in /tmp/naxsi_ut)

Then I edit nginx.conf to make it look like yours (/tmp/naxsi_ut/nginx.conf)

server {
  listen 4242;
  server_name  localhost;

location /myapp {
  # Naxsi config
  LearningMode;
  SecRulesEnabled;

  DeniedUrl "/RequestDenied";

  CheckRule "$SQL >= 8" BLOCK;
  CheckRule "$RFI >= 8" BLOCK;
  CheckRule "$TRAVERSAL >= 4" BLOCK;
  CheckRule "$EVADE >= 4" BLOCK;
  CheckRule "$XSS >= 8" BLOCK;

  auth_request /auth;
  proxy_pass http://77.153.128.117/;
  proxy_set_header Host www.google.fr;
#  root html;
}
location = /auth {
  proxy_method GET;
  proxy_pass_request_body off;
  proxy_pass_request_headers off;
  if ($cookie_auth) {
    set $app_auth "Token $cookie_auth";
  }
  if ($http_authorize) {
    set $app_auth $http_authorize;
  }
  proxy_set_header Authorize $app_auth;
  proxy_set_header Host $host;
  proxy_set_header Content-Length "";
  proxy_pass http://localhost:4241/random/url;
}

(localhost:4241 is just a netcat so I can answer 200 OK by hand)

GET request : I get the answer without problem, as you described.

POST request :

$ curl -d "test" http://127.0.0.1:4242/myapp
<!DOCTYPE html>
<html lang=en>
  <meta charset=utf-8>
  <meta name=viewport content="initial-scale=1, minimum-scale=1, width=device-width">
  <title>Error 405 (Method Not Allowed)!!1</title>
  <style>
    *{margin:0;padding:0}html,code{font:15px/22px arial,sans-serif}ht
....

(the scrapped HTML is google's answer)

in the meanwhile in the netcat :

$ nc  -vlp 4241
listening on [any] 4241 ...
connect to [127.0.0.1] from localhost [127.0.0.1] 60381
GET /random/url HTTP/1.0
Host: localhost
Connection: close

HTTP/1.1 200 OK

And I get the answer here.

Please let me know what I missed to reproduce your issue !

Regards,

andreidotnet · 2015-10-14T12:36:04Z

Hello,
I will provide soon a full configuration that is confirmed from our side to have this issue.
This has not been abandoned from our side :)

andreidotnet · 2015-10-15T14:14:47Z

After investigating further we've tracked the bug to the size of the request, it seems to be working until up to 1024.

The following should not work with your config :
dd if=/dev/zero bs=1 count=1024 | curl -vvk 'http://127.0.0.1:4242/myapp' -X PUT --data-binary @-

The following should work, with your config :
dd if=/dev/zero bs=1 count=10 | curl -vvk 'http://127.0.0.1:4242/myapp' -X PUT --data-binary @-

Note that if you comment #SecRulesEnabled; and retry the request with a content length greater than 1024, it will work.

Please let us know if you were able to reproduce the issue with the settings you have posted above.

buixor · 2015-10-19T09:37:49Z

thank you, I'm trying to give it a look soon :)

andreidotnet · 2015-10-23T08:07:01Z

Just noticed, we're using naxsi-0.54rc3. Will test the latest release soon and confirm if we see the same problem there.

buixor · 2015-10-23T08:17:01Z

Sadly, I'm fraid the latest won't fix the issue, none of the mechanics
where changed :/

On Fri, Oct 23, 2015 at 10:07 AM, Andrei S. notifications@github.com
wrote:

Just noticed, we're using naxsi-0.54rc3. Will test the latest release soon
and confirm if we see the same problem there.

—
Reply to this email directly or view it on GitHub
#226 (comment).

andreidotnet · 2015-10-23T08:28:28Z

So we can take this as a confirmation that you were able to reproduce the issue ? :)

buixor · 2015-10-23T08:37:05Z

Not yet, but you seem to have provided some quite precise things for me to
do so.
I'll try not to be too lazy this week-end and give it a more serious look ;)

On Fri, Oct 23, 2015 at 10:28 AM, Andrei S. notifications@github.com
wrote:

So we can take this as a confirmation that you were able to reproduce the
issue ? :)

—
Reply to this email directly or view it on GitHub
#226 (comment).

buixor · 2015-10-26T10:14:01Z

Sorry, I failed you and had a lazy week-end :p
But I'll check it asap, just that it might requires some serious dive-in :)

FlorinAsavoaie · 2015-10-26T13:00:27Z

Yeah, it probably will. I wasn't sure if it is really a pretty nasty problem or it is just me not having enough skills :)...

buixor · 2015-10-26T16:12:27Z

Hi,

Ok, just tested it :)

At least (good for you, bad for me), I can confirm the bug exists.

buixor · 2015-11-12T15:15:32Z

Hello,

I didn't gave up, but I still didn't understand what is happening, as it's something I never see.
I guess I will have to turn myself to the developper's ML for help :)

andreidotnet · 2015-12-16T09:23:58Z

Hello,
Any luck so far with the discovered issue ?

buixor · 2015-12-16T09:38:05Z

Hi,

I worked a bit on it this week-end and I'm getting closer to it I guess, but I don't have yet a fix to submit, I still need to play around a bit !

I didn't give up and I'll keep you posted as soon as I have something that seems plausible :)

buixor · 2016-01-06T18:23:08Z

Hi & happy new year !

Some further testing have provided some interesting things (no, i didn't solve the issue yet):

form-input-nginx-module (by agentzh) is often cited as a reference (on nginx-dev ML) for modules reading the request body in the REWRITE phase.
if I produce a setup combining form-input-nginx-module + mod_auth without naxsi, I still get the bug (based on the provided test-case)

So far, I think that either nginx changed something in the internals (since nginx 1.7.4), or I've been wrong the whole time, or mod_auth as an issue. Just to be sure I'm not brain-fried, can you confirm that the following setup still gives you the same issue

cd .../naxsi_src/
git clone http://github.com/simpl/ngx_devel_kit.git
git clone http://github.com/calio/form-input-nginx-module.git

Remove naxsi from ./configure, add form-input :

 --add-module=$(MOD_PATH)/ngx_devel_kit \
 --add-module=$(MOD_PATH)/form-input-nginx-module \
 --with-http_auth_request_module \
 --with-debug

I used the following configuration :

location /myapp {
# ensure client_max_body_size == client_body_buffer_size
client_max_body_size 100k;
client_body_buffer_size 100k;
set_form_input $data;    # read "data" field into $data
set_form_input $foo foo; # read "foo" field into $foo
auth_request /auth;
proxy_pass http://77.153.128.117/;
proxy_set_header Host www.google.fr;
}

Cheers, and thanks for your patience !

ps: that's a really sneaky tricky *** bug :)

FlorinAsavoaie · 2016-01-06T20:51:13Z

I rather think that both naxsi and the other module do something that they are not supposed to do in the wrong phase. It is possible that this changed at some point in nginx but it doesn't change the fact that the nginx devs do not recommend doing that. We will give it a try as soon as we can and confirm.

buixor · 2016-01-07T08:45:54Z

Hi !

I didn't mean that mod_auth is wrong, I definitely trust maxim on this kind of topic ^^

If you confirm the issue, I'm going to turn to nginx-dev's ml.

Using form-input-nginx-module as a use case is definitely easier than naxsi, as the code is way shorter and cleaner !

cheers,

andreidotnet · 2016-01-14T13:27:31Z

Hello,
Sorry for the late reply, got caught up with other stuff.
Tested today with nginx 1.9.9 with the requested configuration mentioned previously.

/nginx/usr/sbin/nginx -V
nginx version: nginx/1.9.9
built by gcc 4.4.7 20120313 (Red Hat 4.4.7-16) (GCC)
built with OpenSSL 1.0.1e-fips 11 Feb 2013
TLS SNI support enabled
configure arguments: --add-module=../ngx_devel_kit --add-module=../form-input-nginx-module --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock --http-client-body-temp-path=/var/cache/nginx/client_temp --http-proxy-temp-path=/var/cache/nginx/proxy_temp --user=nginx --group=nginx --with-http_ssl_module --with-http_realip_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_stub_status_module --with-http_auth_request_module --with-stream --with-file-aio --with-http_v2_module --without-http_fastcgi_module --without-http_scgi_module --without-http_uwsgi_module --with-cc-opt='-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic'

The nginx config

location /myapp {

ensure client_max_body_size == client_body_buffer_size

client_max_body_size 100k;
client_body_buffer_size 100k;
set_form_input $data; # read "data" field into $data
set_form_input $foo foo; # read "foo" field into $foo

auth_request /auth;
proxy_pass http://77.153.128.117/;
proxy_set_header Host www.google.fr;
}

With

dd if=/dev/zero bs=1 count=10 | curl -vvk 'http://127.0.0.1:4242/myapp' -X PUT --data-binary @-

we get an ok response

With

dd if=/dev/zero bs=1 count=1024 | curl -vvk 'http://127.0.0.1:4242/myapp' -X PUT --data-binary @-

the request gets stuck

1024+0 records in
1024+0 records out
1024 bytes (1.0 kB) copied, 0.00100387 s, 1.0 MB/s

About to connect() to 127.0.0.1 port 4242 (#0)

Trying 127.0.0.1... connected

Connected to 127.0.0.1 (127.0.0.1) port 4242 (#0)

PUT /myapp HTTP/1.1
User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.19.1 Basic ECC zlib/1.2.3 libidn/1.18 libssh2/1.4.2
Host: 127.0.0.1:4242
Accept: /
Content-Length: 1024
Content-Type: application/x-www-form-urlencoded

^C

When i get some time i'll try and test it out with previous nginx version to try and trace down the last version this worked ok .. if that helps.

buixor · 2016-01-14T14:36:27Z

I'm going to shoot an email to nginx-dev's ML to check. I'll mostly ask if form-input-nginx-module is doing things correctly :)

Keep me posted as well,

cheers !

andreidotnet · 2016-01-14T15:32:50Z

Did some more testing with multiple nginx version. We can reproduce the issue describe in the previous post even in nginx 1.5.4 (first version with http_auth_request_module
Without http_auth_request_module, the request works as expected even in the latest nginx builds.
It's starting to look like http_auth_request_module might be the issue.

buixor · 2016-01-22T15:57:55Z

Hi,

I posted on nginx-dev ML : http://mailman.nginx.org/pipermail/nginx-devel/2016-January/007762.html
So far, I didn't get the perfect answer but some good clues. Maxim suggested something I will need to try (first attempt failed), while agentzh tends to point at auth module ;)

I didn't give up ;)

andreidotnet · 2016-05-25T20:30:28Z

Heya buixor,
Did you manage to find any news regarding this issue ? Have not noticed any activity on the ML regarding this topic.

buixor · 2016-07-06T08:42:37Z

Hello,

Sorry for delay :)
I tried Maxim suggestion, but without success. Tbh I think agentzh might be right, as I didn't manage to reproduce the bug with any other module except this one. I will try to get back on it after we release 0.55, I didn't had much chance to work on this issue :(

FlorinAsavoaie · 2017-02-27T15:07:12Z

Hi, how are you doing?

0.55 is out 💃 Time for party!

But we'd also appreciate some more input on this one.

Thanks,

defanator · 2018-10-30T15:33:50Z

Any success with this one so far?

FlorinAsavoaie · 2018-10-30T15:40:50Z

We switched to mod_security :(

jvoisin · 2018-10-30T17:06:36Z

Aw </3

defanator · 2018-11-01T09:46:48Z

Well, don't be surprised if you'll see the same behavior with (lib)modsecurity. :)

owasp-modsecurity/ModSecurity-nginx#130
owasp-modsecurity/ModSecurity-nginx#131

dani · 2018-11-08T12:04:01Z

Guess I have somehow the same, or a similar/related issue. Running nginx 1.15.6 and naxsi 0.56. When naxsi is enabled, even in learning mode, some POST requests just hang. I haven't isolated exactly which ones, but apparently, it can affect requests < 1024. I'll try to get some more info and a simple reproducer

dani · 2018-11-08T14:38:04Z

For now, I can confirm the issue is only present when I protect the location with auth_request. I'm using Lemonldap::NG handler (see configuration example here: https://lemonldap-ng.org/documentation/latest/configvhost#reverse_proxy1)

As soon as I remove the auth_request part, everything is working again.

buixor · 2018-11-12T10:33:16Z

@FlorinAsavoaie Can you please notify us if you encounter the same issue with mod_sec and auth_request ? If it's the case, I should document the known incompatibility between those two modules.

dani · 2018-11-12T10:39:53Z

Has something like owasp-modsecurity/ModSecurity-nginx@1cd5484 been applied on NAXSI ?

buixor · 2018-11-12T10:55:58Z

@dani This is what (I think but ofc I didn't keep proper track of it) I tried after maxim's suggestion when I posted, but it's worth giving a try again :)

dani · 2018-11-12T10:59:02Z

I'd be happy to give it a try, if you can show me a commit. I have a setup where I can easily reproduce the issue.

buixor · 2018-11-12T11:15:51Z

@dani in our case, the changes should be made in ngx_http_dummy_access_handler of naxsi_skeleton.c around this part.

Unfortunately, it seems I didn't keep track of my previous attempts (no commit), so I guess I'll have to first spend some time to do the restoration of event_handlers in a branch for you to properly test :/

buixor added the bug label Jun 13, 2017

buixor mentioned this issue Mar 21, 2019

Issue with Graphql file uploads using nginx proxy #459

Closed

mremande closed this as completed Nov 8, 2023

Requests stuck in certain cases #226

Requests stuck in certain cases #226

Comments

FlorinAsavoaie commented Sep 29, 2015 • edited by jvoisin Loading

buixor commented Sep 30, 2015

FlorinAsavoaie commented Sep 30, 2015

buixor commented Sep 30, 2015

FlorinAsavoaie commented Sep 30, 2015 • edited by jvoisin Loading

buixor commented Sep 30, 2015

buixor commented Oct 9, 2015 • edited by jvoisin Loading

andreidotnet commented Oct 14, 2015

andreidotnet commented Oct 15, 2015

buixor commented Oct 19, 2015

andreidotnet commented Oct 23, 2015

buixor commented Oct 23, 2015

andreidotnet commented Oct 23, 2015

buixor commented Oct 23, 2015

buixor commented Oct 26, 2015

FlorinAsavoaie commented Oct 26, 2015

buixor commented Oct 26, 2015

buixor commented Nov 12, 2015

andreidotnet commented Dec 16, 2015

buixor commented Dec 16, 2015

buixor commented Jan 6, 2016

FlorinAsavoaie commented Jan 6, 2016

buixor commented Jan 7, 2016

andreidotnet commented Jan 14, 2016

ensure client_max_body_size == client_body_buffer_size

buixor commented Jan 14, 2016

andreidotnet commented Jan 14, 2016

buixor commented Jan 22, 2016

andreidotnet commented May 25, 2016

buixor commented Jul 6, 2016

FlorinAsavoaie commented Feb 27, 2017

defanator commented Oct 30, 2018

FlorinAsavoaie commented Oct 30, 2018

jvoisin commented Oct 30, 2018

defanator commented Nov 1, 2018

dani commented Nov 8, 2018

dani commented Nov 8, 2018

buixor commented Nov 12, 2018

dani commented Nov 12, 2018

buixor commented Nov 12, 2018

dani commented Nov 12, 2018

buixor commented Nov 12, 2018

FlorinAsavoaie commented Sep 29, 2015 •

edited by jvoisin

Loading

FlorinAsavoaie commented Sep 30, 2015 •

edited by jvoisin

Loading

buixor commented Oct 9, 2015 •

edited by jvoisin

Loading