memory consumption unpredictible

[soloturn]

when compiling software like cosmic-epoch:
https://github.com/pop-os/cosmic-epoch

on a laptop with 8GB and default settings, rust toolchain fails when linking the binaries, as no memory is left. there is an option to only use 1 of the 4 cores, but this is a setting for both, compiling and linking.

would be cool if by default, the rust toolchain would be aware how much memory there is and use all the threads for compile, but restrict itself to a max memory for linking.

[saethlin]

The title says memory consumption is unpredictable, but the issue says memory consumption is quite predictable. Which is it? I adjusted the labels anticipating the answer is something like "Build parallelism produces shocking increase in memory usage" which is a statement I would certainly agree with.

I'd like to reproduce the memory usage in question. How would I do that? I tried to run just in a clone of the linked repo and I see this:

mkdir -p build
/home/ben/.cargo/bin/just cosmic-settings/build-release
error: Justfile does not contain recipe `build-release`.
error: Recipe `build` failed on line 7 with exit code 1

[soloturn]

@saethlin, with unpredictable i ment: cannot say in advance that it uses so much memory, and does not restrict itself to the memory in the system. but you are absolutely right, running it a second time uses again more than 8GB.

would what is mentioned in the readme work for you?

git clone --recurse-submodules https://github.com/pop-os/cosmic-epoch
cd cosmic-epoch
just sysext

on arch linux the following works for me, sometimes:

$ git clone https://aur.archlinux.org/package_name.git
CARGO_BUILD_JOBS=1 CARGO_REgistries_CRATES_IO_PROTOCOL=sparse makepkg

the PKGBUILD file specifies how this is run, https://aur.archlinux.org/cgit/aur.git/tree/PKGBUILD?h=cosmic-epoch-git. when using 4 build jobs as the processor would allow by default, linking uses all the time more than 8gb.

[saethlin]

Hm, I can't reproduce 8 GB peak memory usage, I consistently only see 2.8 GB. I'm on Arch, running /usr/bin/time -v makepkg for the PKGBUILD you linked.

[soloturn]

at some point it arrives at building 3 binaries at once:

$ CARGO_BUILD_JOBS=3 CARGO_REGISTRIES_CRATES_IO_PROTOCOL=sparse makepkg

   Compiling cosmic-applet-notifications v0.1.0 (/home/src/tmp/cosmic-epoch-git/src/cosmic-epoch/cosmic-applets/cosmic-applet-notifications)
   Compiling cosmic-app-list v0.1.0 (/home/src/tmp/cosmic-epoch-git/src/cosmic-epoch/cosmic-applets/cosmic-app-list)
   Compiling cosmic-applet-network v0.1.0 (/home/src/tmp/cosmic-epoch-git/src/cosmic-epoch/cosmic-applets/cosmic-applet-network)
    Building [=======================> ] 619/630: cosmic-applet-network(bin), cosmic-app-list(bin), cosmic-applet-notifications(bin)                   

top - 01:06:33 up 36 min,  2 users,  load average: 3.00, 3.06, 2.82
Tasks: 191 total,   1 running, 190 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.4 us,  1.3 sy, 73.8 ni, 23.8 id,  0.0 wa,  0.6 hi,  0.0 si,  0.0 st 
MiB Mem :   7813.8 total,    357.8 free,   3013.1 used,   4891.8 buff/cache     
MiB Swap:      0.0 total,      0.0 free,      0.0 used.   4800.7 avail Mem 

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND   
  58112 src        30  10  843244 524252  90980 S  99.0   6.6   1:07.90 rustc     
  57024 src        30  10  795412 481152  83732 S  98.7   6.0   1:17.66 rustc     
  51120 src        30  10 1018644 663524 108436 S  98.0   8.3   5:53.65 rustc     

and - very cool - it now does not run out of memory any more.

[soloturn]

@saethlin cosmic desktop now got split up into single components, entry point is:
https://aur.archlinux.org/packages/cosmic-session-git

for me it fell over with out of memory when building the dependent cosmic-applets, on 2024.07.21. am leaving the ticket closed, as it was one time.