Thread/workers questions

[bogdan-dubyk]

Hello, a bunch of noob questions I have :)

1. Is there any page or something that explains what is the difference in terms of Franken between worker numbers, and num_threads, If I have a single application it should be the same value or different values?

2. And is there any way to understand/measure/calculate the proper number of threads/workers your setup can handle? From documentation I only see that default value for both threads and workers is 2x the number of available CPUs, but how it calculate number of available CPUs? If app is spininig in K8s, does it take number of CPUs on VM, or it take to the account limits we set for pod?

3. Can I somehow see current number of threads and workers (maybe with some command) when I use default config?

[withinboredom]

For (1): num_threads must be greater than or equal to sum(num_workers), as you can define more than one worker script. If they are equal to each other, then only a single additional thread is used for nonworker requests.

For (2): the defaults are fine for most cases. Tuning these in conjunction with MAXPROCS, GOGC and things aren't well documented yet. Mostly because there hasn't really been a reason to tune them. It gets the number of cores from the operating system, not k8s. Also, don't set CPU limits in k8s... https://www.youtube.com/watch?v=KCFFZ_qfKXk, unless you know what you are doing.

As a side-note, if you have memory limits in k8s, take care to make sure you set GOMEMLIMIT environment variable and max_memory ini setting so that you don't use more memory than available (k8s doesn't tell the process how much memory it has, so it assumes it has ALL the memory). e.g.: if you have 2 workers with max_memory set to 512 mb and GOMEMLIMIT set to 1gb, you need at least 2gb of memory for the pod.

For (3): you can configure these with the FRANKENPHP_CONFIG environment variable.

[dunglas]

Actually, FrankenPHP does get the number of CPUs from K8S, because it uses https://github.com/uber-go/automaxprocs (to be exact, Caddy uses this package).

We definitely need to document all of this.

[bogdan-dubyk]

to not create any new discussion, want to share my experience of comparison between Franken in worker mode with default setting and NGNIX + PHP-FPM

For Nginx + FPM I have 2 pods in k8s, one for nginx and another for fpm, CPU requests/limits are: limits 0.5, request 0.2.
For Franken single pod in k8s with the same limits are limits 0.5, request 0.2.

The endpoint is to get a record from Mongo DB by id and return.
To benchmark I used wrk

Franken:

Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    29.68ms   28.32ms 136.26ms   79.48%
    Req/Sec    41.54     14.94   158.00     77.30%
  Latency Distribution
     50%   10.66ms
     75%   53.31ms
     90%   77.16ms
     99%   88.95ms
  2354 requests in 1.00m, 25.94MB read
  Socket errors: connect 0, read 0, write 0, timeout 1
Requests/sec:     39.18
Transfer/sec:    442.15KB

FPM

 Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    25.79ms   20.86ms 182.09ms   80.74%
    Req/Sec    44.26     10.51   101.00     72.12%
  Latency Distribution
     50%   13.18ms
     75%   39.05ms
     90%   60.25ms
     99%   73.28ms
  2653 requests in 1.00m, 29.67MB read
Requests/sec:     44.17
Transfer/sec:    505.93KB

Maybe I'm doing something wrong, but I do not see any performance boosts at all, I run it many times and always looks like Franken is 5% slower and always with timeout on the first request.

[bogdan-dubyk]

BTW if I increase k8s resources from limits 0.5, request 0.2. into limits 2.5, request 2.0 all good, but we do not want to set such limit but instead spin more pods with HPA if needed

[withinboredom]

I don’t mean to sound critical, but I’m struggling to see the logic here. It seems you’ve configured FrankenPHP to be (which is multithreaded) on less than a single core. It’s unlikely to achieve better performance this way compared to a single-threaded instance like FPM on less than one core.

FPM is designed for single-threaded operation, whereas FrankenPHP is not. It seems like contention might be severely impacting your RPS (requests per second). Trying to have 64 threads competing for less than a core of CPU time isn’t likely to end well. With all due respect, what outcome are you expecting here?

Perhaps consider using just 2 threads?

Align GOMEMLIMIT with memory_limit and Kubernetes memory limits

The pod memory limit should be calculated as num_threads * memory_limit + GOMEMLIMIT. Requests can likely just be GONEMLIMIT

[bogdan-dubyk]

64 I set just for the sake of the test, as I do not know what the default value. I already tried with 2 threads, and it is still slower than FPM, and again the first request failed with a timeout. I think in my setup with a small amount of resources it makes no sense to use more than 1 thread or even worker mode as I see no benefits over FPM.

In general what we expected to see with Franken is the ability to handle a bit more request requests per second which should be like that by keeping the app in memory, right? Even not a big boost should be there?

Tomorrow I'll remove limits from both FPM and Franken and run the same tests, I already know that without limits Franken can handle 308 RPS when it uses 2 cores and 500Mb, will try the same over FPM

[withinboredom]

In general what we expected to see with Franken is the ability to handle a bit more request requests per second which should be like that by keeping the app in memory, right? Even not a big boost should be there?

I suspect you are running into most contention issues. For example, FrankenPHP gets some output from PHP and sends the output to another thread running caddy. In your case, the php thread must pause/yield its execution to actually send that data back. Having only half core worth of CPU time means this might be awhile. I'll have to check, but I'm pretty sure the response writer in FrankenPHP is unbuffered, so we have to yield the thread immediately after writing it out.

In fpm, the output is sent immediately to a shared buffer with nginx and can resume the php application until thread execution is paused. This is faster in <= single-cores because we don't have to switch threads. In FrankenPHP, we basically do every time we send some output.

I will do an experiment and set a buffer on the response writer in <2 core situations. This may (most likely) improve performance significantly in low-resource situations.

I do not know what the default value

For 0.5 cores, it is theoretically 1.

[withinboredom]

FWIW, ResponseWriter is buffered already.

I ran some tests using taskset to limit the number of cores available for FrankenPHP. You should probably do something similar on your machine. Here are the number of cores to RPS:

As you can see from the graph, there is a point of diminishing returns where adding more CPUs doesn't really give more benefits.

Finally, I experimented with nginx in front of several (or less) FrankenPHP servers. This is on dedicated server hardware instead of my development computer, with a dedicated link between them. This is where things start getting interesting...

In this case, as the number of replicas goes down and cores go up, performance starts to drop dramatically, which is unexpected. If I connect FrankenPHP directly to the internet using all 12 cores, I get >33k RPS...

So, what about if we put caddy in front of it? In this case, we get around 19k RPS (or about 50%). So, let's rerun the tests again...

As you can see, caddy beats the pants off on nginx (surprisingly not surprised). Here's everything put together for this hardware:

So, to answer your question, you probably want more than one cpu to be able to efficiently handle requests and take full advantage of your hardware. Otherwise, much of it will be "wasted" on contention. But if you have a loadbalancer, you probably also don't want a big "beefy" frankenphp, but several smaller ones. How many will be up to your load and you'll likely have to test it out for yourself.

[bogdan-dubyk]

One more question I do not understand, I did 2 tests:

• one is with the wrk command like this wrk -t10 -c10 -d1m --timeout 10s -s script.lua {domain}
• and another is with real traffic to the same endpoint as in wrk

I run both tests on the same k8 pod with 3 CPU requests and 4 CPU limits, with 1.18GB memory request, and 1.6GB memory limit

And in case of real usage it use only 1.5 CPU but 3.5GB of memory and when I run wrk it use all CPU but only 500Mb of memory

Again both are requests to the same endpoint

[withinboredom]

I can explain the memory difference off the top of my head: keepalives.

When you run wrk, it keeps the connection open and reuses it on the next request. This is very memory efficient.

When you have real users, each request must be tracked independently (tcp state, ports, etc) and that costs memory. Once the connection is done, go/caddy/frankenphp can allow that memory to be released. However, go tries to avoid reclaiming it until it needs to (see go's documentation on garbage collection) and further, tries to keep memory collection at a minimum until there is actual memory pressure (use GOMEMLIMIT to artificially create memory pressure and GOGC to control how aggressive it is).

You may be spending quite a bit of time doing gc in go (you can set the environment var: GODEBUG=gctrace=1 to get an output on gc usage, but there may be a better way via caddy admin/debug/prometheus api) with so low memory limits.

My personal opinion: <8gb of memory or <2cpu for an instance is raspberry pi level commodity hardware and expectations should be set as such. <32gb of memory or <12cpu is personal computer level commodity hardware, and expectations should be set as such. Anything higher than that is production servers. That being said, the things we used to run on with less than a rpi gives today should blow your mind... but those were the days when people gave up code readability for speed. Those days are long gone...

[AlliBalliBaba]

3.5 GB seems like a lot, is the endpoint doing a lot of memory intensive work?
For comparison: we've been using Frankenphp on a production AWS fargate service with multiple tasks sized at 1 CPU core for a month now. Memory utilization is pretty constant around 200MB. This is while using worker mode with laravel though (5 workers) and caching a lot of configuration in memory.
If you are not using using worker mode, you can also try upgrading to v1.2.3, which has some performance improvements for non-worker mode.

[bogdan-dubyk]

@AlliBalliBaba that endpoint queries Mongo by ID (and optionally some extra parameters) and returns the document back. The document itself is pretty big, sometimes PHP uses 340Mb (memory limit is 350Mb). But I found a way to hold memory on 1.5Gb by reducing the number of workers from 8 to 3 and by setting GOGC to 20.

Currently, it's only one endpoint for testing, and we have a heavier endpoint, some endpoints can take 10-40 seconds to be executed and in terms of memory are the same, and I'm not sure with 3 workers I'll be able to handle all the traffic.

Nginx-FPM combination handles this on easy and memory was never more than 600Mb, of course, RPS is smaller.

[sneycampos]

@bogdan-dubyk with my experience in trying to use frankenphp or octane, using a few workers may be a problem sometimes when you have slower endpoints like this that take seconds to respond since each worker may respond only 1 request per time.
i'm trying to figure out too how to handle with these cases when you have some "slow endpoints". Maybe only using a powerful machine with more cores/ram to have more workers. Meanwhile i'm using the loadbalancer to transfer all these "slow requests" to a server running fpm instead frankenphp/octane.

Any tip guys? 🥹

[sneycampos]

Hello guys, my question is not related to the original author question but almost the same. Here is:

How can i use frankenphp (non worker mode) in an application that may handle slow requests even using a machine with a few cpus, like 2 cpus? frankenphp is using 2 cores and 4 threads, so frankenphp will run 4 workers, right?

That said, how many requests is supposed to handle? In my example:
i have an endpoint with a slow operation (30s +- to respond), my application will handle only 4 slow operations at same time and if i receive any other requests, even not slow requests, will not be handled while my application is busy with the first slow operations?

Any tip to handle with this situation? with FPM is easier to increase RAM to increase the amount of fpm processes than increasing 10x the number of cores, if i'm not wrong.

[sneycampos]

Thank you @AlliBalliBaba. Appreciate your explanation!

[bogdan-dubyk]

but what if the bottleneck is an old legacy database with bad design and the same request can take for one set of criteria 200ms but the same request for another set of criteria can take 60 seconds? I know your answer will be to fix the database issue, and that is in progress, but it'll take time. So I have a feeling that Franken is not an option until we fix all the endpoints, or maybe Franken on CGI mode, that is something I need to test and compare with NGINX+FPM, well, of course big server and many workers

[sneycampos]

@bogdan-dubyk for this case is almost same as NGINX+FPM atleast for me is the same thing but with performance improvements (decreased some ms). Since you can increase the numbers of workers in CGI mode to deal with more requests like fpm but better.
I have a "small" machine with 2 cores 2gb ram (t3a.small) running FrankenPHP in CGI mode configured with same amount of FPM processes

[bogdan-dubyk]

@sneycampos

increase the number of workers in CGI

Do you mean num_treads ?

can you share you CGI config

[sneycampos]

Do you mean num_treads?
can you share you CGI config

Yes, i'm sorry.

I'm using the following env but you can adjust the num_threads according to your RAM

ENV FRANKENPHP_CONFIG="num_threads 40"

[bogdandubyk]

I have another foolish question, I saw the new release that promises an increase of performance on non-worker mode, but what is non-worker mode? it's when I set

	frankenphp {
		worker index.php 1
	}

? or there are some other config? I can't find anything related to non-worker mode in documentation

[sneycampos]

i have the same question maybe and i don't think its non-worker mode, in non-worker mode you shouldn't use the worker configuration in caddyfile

should be something like (I THINK trademark)

frankenphp {
    num_threads 10
}

[dunglas]

"non-worker mode" is just the default mode, without a worker, no special config is needed. Worker mode is faster than the default mode, so you should use it if possible.