Docker image memory: Ridiculous ll_alloc Block size

[pichi]

Hi!
I'm running a LiveBook using the latest docker image on Rocky Linux release 9.3 (Blue Onyx).

REPOSITORY                      TAG                   IMAGE ID       CREATED         SIZE
ghcr.io/livebook-dev/livebook   latest                5df3006b7837   5 months ago    604MB

I don't know if it's a problem with this particular version of the image or version of OS, docker, size of memory, or whatever.
The issue is that in this image running BEAM set its ll_alloc (/dashboard/memory_allocators) Block size to 2.0GB.

On other virtual machines, I have observed sizes 26MB, 28MB, 32MB, and so on.

So my question is how can I fix this particular instance and how can I prevent this issue in the future?

[josevalim]

Hi @pichi, nice to see you here!

I don't think we do anything in particular about allocators: https://github.com/livebook-dev/livebook/blob/main/rel/server/vm.args.eex

So it would have to be the Erlang VM somehow inferring that it needs such large blocks? Would you know why that could be?

[starbelly]

Hi, @pichi it's great to see you in such a long time! 🤗

I only looked at the code in play for this dashboard (and related utility modules) briefly, but it looks like it will tally up all the blocks (I could be wrong, just looked like that at a glance). Did you try running :erlang.system_info({:allocator_sizes, [:ll_alloc]}) to see what the raw data shows?

[josevalim]

Here is the source code for computing the allocators. Our logic may be wrong indeed: https://github.com/phoenixframework/phoenix_live_dashboard/blob/f87a1bd115a94ac72d75bd173c6ff0b76db44584/lib/phoenix/live_dashboard/system_info.ex#L202

[starbelly]

Oh, I'm not so sure it's wrong, though there may be an edge case or two in there. I think it depends on what the intention is, and if it's to show total size in terms of blocks allocated, then it seems correct to me, but I can look further. The more interesting part to me is how did Pichi end up with 2GB of ll_alloc blocks :)

[starbelly]

Note for @pichi this was bugging me to no end; I did some searching and found where people would OOM in containers on some platforms because the number of file descriptors allowed was very high (specifically related to containers and cgroups). Quoting erl documentation :

+Q Number
Sets the maximum number of simultaneously existing ports for this system if a Number is passed as value. Valid range for Number is [1024-134217727]
NOTE: The actual maximum chosen may be much larger than the actual Number passed. Currently the runtime system often, but not always, chooses a value that is a power of 2. This might, however, be changed in the future. The actual value chosen can be checked by calling erlang:system_info(port_limit).
The default value used is normally 65536. However, if the runtime system is able to determine maximum amount of file descriptors that it is allowed to open and this value is larger than 65536, the chosen value will increased to a value larger or equal to the maximum amount of file descriptors that can be opened.
On Windows the default value is set to 8196 because the normal OS limitations are set higher than most machines can handle.

This makes sense, as if erts creates a LOT of entries in the port table on boot, it would explain an incredible number of blocks allocated via ll alloc.

You can test this hypothesis by queries the number of ports possible, check the fd limit on the os, or limit with the +Q flag.

[pichi]

Sorry, I forgot to mention the reason I stumbled upon this problem is that it crashes for insufficient memory but you deduced it right.

(livebook_server@aff10f738ce7)1> erlang:system_info({allocator_sizes, [ll_alloc]}).
[{ll_alloc,[{instance,0,
                      [{mbcs,[{blocks,[{ll_alloc,[{size,2153885504,2153885504,
                                                        2154034728}]}]},
                              {carriers_size,4304404480,4304404480,4304404480}]},
                       {sbcs,[{blocks,[]},{carriers_size,0,0,0}]}]},
            {instance,1,
                      [{mbcs,[{blocks,[{ll_alloc,[{size,4780872,4780872,
                                                        4780872}]}]},
                              {carriers_size,6815744,6815744,6815744}]},
                       {mbcs_pool,[{blocks,[]},{carriers_size,0}]},
                       {sbcs,[{blocks,[]},{carriers_size,0,0,0}]}]},
            {instance,2,
                      [{mbcs,[{blocks,[{ll_alloc,[{size,1370376,1370376,
                                                        1391928}]}]},
                              {carriers_size,1572864,1572864,1572864}]},
                       {mbcs_pool,[{blocks,[]},{carriers_size,0}]},
                       {sbcs,[{blocks,[]},{carriers_size,0,0,0}]}]}]}]
(livebook_server@aff10f738ce7)2> erlang:system_info(port_limit).
134217727

So you are right the problem is the number of file descriptors. On the node that works well

(livebook_server@2310528d9881)1> erlang:system_info(port_limit).
1048576

It's a shame that it didn't hit me. We already had a similar problem when we went from Rocky Linux 8 to Rocky Linux 9. In Rocky Linux 9, there are ridiculous default limits.

Thanks, both of you @josevalim and @starbelly for help.