Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Regression: Caddy 2.9.0 returns 411 Length Required for some sites #6757

Closed
spurgeonbj opened this issue Jan 1, 2025 · 20 comments · Fixed by #6759
Closed

Regression: Caddy 2.9.0 returns 411 Length Required for some sites #6757

spurgeonbj opened this issue Jan 1, 2025 · 20 comments · Fixed by #6759
Labels
bug 🐞 Something isn't working
Milestone

Comments

@spurgeonbj
Copy link

After upgrading from Caddy 2.8.4 to 2.9.0, Joomla sites started returning HTTP 411 ("Length Required") errors for certain requests. Downgrading to Caddy 2.8.4 resolves the issue, indicating a regression or stricter enforcement of request header handling in 2.9.0.

The problem seems related to:

  1. Handling of requests without a Content-Length header.

. Requests that were previously allowed in Caddy 2.8.4 now fail in 2.9.0.

  1. Interaction with Joomla's rewrites.

. Joomla relies on rewrites for dynamic routing (e.g., /index.php?{query}). The rewrites still work, but certain requests (possibly POST or other non-GET methods) fail with 411.

Steps to Reproduce:

Configure a Joomla site with the following minimal Caddyfile:

example.com {
    root * /var/www/samplesite
    php_fastcgi unix//run/php/php7.4-fpm.sock
    file_server

    @joomlaNotFound {
        not file {path}
        not file {path}/index.php
    }
    rewrite @joomlaNotFound /index.php?{query}
}

Make a POST request without a Content-Length header:

curl -X POST https://dev.apcwo.org/path -d '' -v

Observe a 411 response in Caddy 2.9.0. The same configuration works in Caddy 2.8.4.

Expected Behavior: Caddy should handle requests without a Content-Length header the same way as in version 2.8.4 or provide clear guidance on how to adapt the configuration.

Observed Behavior: Caddy 2.9.0 returns 411 errors for requests that worked in 2.8.4, breaking Joomla functionality.

Additional Details:

Joomla uses rewrites heavily, and the issue seems tied to how Caddy processes requests during rewrites.
The problem was tested with PHP-FPM and occurs even with minimal configurations.
Caddy logs show no additional details besides the 411 response.
Downgrading to Caddy 2.8.4 immediately resolves the issue.

Environment:

Caddy Version: 2.9.0
PHP Version: 7.4 (PHP-FPM)
Joomla Version: (3.10)
OS: (Ubuntu 20.04)

Suggestions for Debugging:

Review changes in Caddy 2.9.0 related to request handling or HTTP compliance.
Investigate interactions between the rewrite directive and PHP-FPM.

@FatihKoz
Copy link

FatihKoz commented Jan 1, 2025

Issue also affects Laravel applications running with Caddy.

@devzkhalil
Copy link

I cannot setup my laravel project using Caddy 2.9.0

@Brawl345
Copy link

Brawl345 commented Jan 1, 2025

Same thing happening to my WordPress site. I have only observed this in Firefox though. Downgrading fixed it.

@teddysun
Copy link

teddysun commented Jan 1, 2025

Same issue.
I think I found the reason.
The following two PRs are the reasons.

#6638
#6661

I revoked these two PRs and compiled manually again, and the problem was solved.

@WeidiDeng
Copy link
Member

Add request_buffers to your reverse proxy configuration and see if the problem is fixed.

@teddysun 6638 is not the cause, that actually fixes the buffering problem. Revert 6661 and see if it will happen.

These two patches are implemented to solve the ddos attacks reported. However, request buffering will be needed if content length is not specified.

Any access logs will be appreciated to see if any alternative approaches without enabling explicit request buffering can be done.

@spurgeonbj spurgeonbj changed the title Regression: Caddy 2.9.0 returns 411 Length Required for Joomla Sites Regression: Caddy 2.9.0 returns 411 Length Required for some sites Jan 1, 2025
@teddysun
Copy link

teddysun commented Jan 1, 2025

@WeidiDeng

Yes, revert #6661 and problem was solved.

@dzervelce
Copy link

Ubuntu 22 (ARM), PHP 8.3.
Same problem - after upgrading to 2.9.0 from 2.8.4 all websites return 411 error, also when making GET request from Chrome 132.
Downgrading back to 2.8.4 solved this issue.

@WeidiDeng
Copy link
Member

@teddysun Can you test if xcaddy build fastcgi_buffer fixes this issue?

@Telesphoreo
Copy link

I am having this issue, but oddly only on a few select servers. Downgrading has not fixed it for me. Tried Firefox and the first request went through, but subsequent ones did not. This is very odd

@rwinkhart
Copy link

Encountered this issue on my basic HTML+CSS+PHP sites. Was fixed by reverting #6661.

@WeidiDeng
Copy link
Member

@rwinkhart Can you test if xcaddy build fastcgi_buffer fixes this issue?

@mholt
Copy link
Member

mholt commented Jan 2, 2025

@teddysun @Telesphoreo @dzervelce @Brawl345 @spurgeonbj We know that downgrading fixes the issue; it's already an established regression -- but please notice @WeidiDeng is asking if others can verify whether #6759 fixes it without having to revert and downgrade. If no one experiencing the issue can verify this then we cannot release a patch. Weidi has even provided instructions multiple times now for an easy way to compile Caddy with the proposed patch: xcaddy build fastcgi_buffer -- we would greatly appreciate that you try it so we can release a proper patch. 🙂 Thank you!

@Telesphoreo
Copy link

Telesphoreo commented Jan 2, 2025

Yes, I added request_buffers 100m inside my php_fastcgi block and it did fix it. Very weird on why this only happened on one server (regardless of PHP being used for the actual subdomain). I have one local VM that builds caddy with xcaddy and deploying it to all of those went fine. On a few servers that I just update with apt, some had issues and some didn't. They all have the same Caddy configuration. Very weird that downgrading didn't fix it for me, but adding that block did fix it.

Edit: The fix did work. You can ignore everything else. I am definitely very tired and have everything mixed up. I did not realize that the caddy add-package command upgraded you to 2.9.0 even though I had downgraded with apt. Basically, all the servers I deployed a self built caddy binary worked. Some upgraded fine with apt, some did not. I realize now that all the inconsistencies were my use of the add-package command on the servers that install caddy from apt.

@rwinkhart
Copy link

rwinkhart commented Jan 2, 2025

@WeidiDeng The build seems to be failing due to an import cycle when I try that. Haven't used xcaddy before (I've just been building with go build). All I did was install it and run your suggested command, so I'm not sure if I did something wrong. Tried on two different computers using xcaddy from the Alpine repos and from the AUR.

2025/01/02 01:47:43 [INFO] exec (timeout=0s): /usr/bin/go build -o /home/cuan/caddy/caddy -ldflags -w -s -trimpath -tags nobadger,nomysql,nopgx 
package caddy
        imports github.com/caddyserver/caddy/v2/modules/standard
        imports github.com/caddyserver/caddy/v2/modules/caddyhttp/standard
        imports github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy
        imports github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy/fastcgi
        imports github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy: import cycle not allowed

Edit: Though this could probably be assumed, I thought I should specify that it does build fine without specifying fastcgi_buffer.

@zwieratko
Copy link

I have the same problem. After upgrading from Caddy 2.8.4 -> 2.9.0 the site was broken/unresponsive. I can confirm that after adding request_buffers 4k to the php_fastcgi directive everything is fine.
Sorry, I couldn't do the build.

zwerimex.com {
    root * /var/www/zw_test01
    encode zstd gzip
    try_files {path} {path}.php {path}.html

    file_server

    php_fastcgi unix//run/php/php8.3-fpm.sock {
        request_buffers 4k
    }

    header {
        Permissions-Policy "accelerometer=(), autoplay=(), camera=(), geolocation=(self), gyroscope=(), magnetometer=(), microphone=(), payment=(), serial=(), usb=()"
        Strict-Transport-Security "max-age=31536000; includeSubDomains; preload"
        X-Frame-Options "deny"
        X-Content-Type-Options "nosniff"
        Referrer-Policy "no-referrer-when-downgrade"
    }
}

@teddysun
Copy link

teddysun commented Jan 2, 2025

@teddysun @Telesphoreo @dzervelce @Brawl345 @spurgeonbj We know that downgrading fixes the issue; it's already an established regression -- but please notice @WeidiDeng is asking if others can verify whether #6759 fixes it without having to revert and downgrade. If no one experiencing the issue can verify this then we cannot release a patch. Weidi has even provided instructions multiple times now for an easy way to compile Caddy with the proposed patch: xcaddy build fastcgi_buffer -- we would greatly appreciate that you try it so we can release a proper patch. 🙂 Thank you!

After #6759 commit 3ba88fe
I tested again.
All the issues I found were resolved. LGTM.
I hope there will be more tests and merge into the master as soon as possible.

@spurgeonbj
Copy link
Author

Thanks a ton! #6759 LGTM too! God bless you abundantly in all that concerns you!

@jum
Copy link
Contributor

jum commented Jan 2, 2025

Happy to report that fastcgi_buffer fixes HTTP/3 error 411 problems from nextcloud via HTTP/3 for me.

@skrlance
Copy link

skrlance commented Jan 2, 2025

Happy to report that fastcgi_buffer fixes HTTP/3 error 411 problems from nextcloud via HTTP/3 for me.

I am unable to use xCaddy on my Almalinux 9 server to build with fastcgi_buffer. Instead I tried adding the syntax request_buffers 100m or request_buffers 4k both didn't solved the issue.

        php_fastcgi unix//run/php-fpm/www.sock {
        request_buffers 100m
        }

All images stopped displaying in WordPress admin before or after I used request_buffers syntax. Difference is that previously it wasn't displaying and after it didn't loaded.

@mholt
Copy link
Member

mholt commented Jan 2, 2025

Thank you to everyone who tested @WeidiDeng's hotfix! (And huge thanks to Weidi for the quick PR.) We will probably rework the details of the patch later but for now it seems like it does the job.

@mholt mholt added the bug 🐞 Something isn't working label Jan 2, 2025
@mholt mholt added this to the v2.9.1 milestone Jan 2, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug 🐞 Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.