BIIGLE
-
|
I have set-up GPU branch on my biigle server. When I try MAIA novelty detection, I got the following errors (ran out of memory) in logs. It seems simliar to biigle/maia#64 Logs: |
Beta Was this translation helpful? Give feedback.
Answered by
mzur
Nov 10, 2022
Replies: 1 comment 5 replies
-
|
By default MAIA in the GPU branch is configured to expect 16 GB of GPU memory (the number in the .env file is actually 15 GB to avoid biigle/maia#64 most of the time). Do you have this amount of GPU memory? If not, try to lower this number, rebuild and restart your BIIGLE instance. If you do have this amount of GPU memory, you are probably affected by biigle/maia#64. In this case you can still try to lower the number and see when/if it works. We likely won't address biigle/maia#64 any more but rather port all the code to PyTorch/OpenMMLab. |
Beta Was this translation helpful? Give feedback.
5 replies
-
|
Indeed I have 8G, and changed to MAIA_AVAILABLE_BYTES=7E+9. No luck and get similar errors. I do watch nvidia-smi, saw top memory usage is about 4.8G, and then the maia job failed. I'm curious about the log said: It ended differently at differnt op & memory run-out vs. previous results (as I misused MAIA_AVAILABLE_BYTES=15E+9) But for both situations or at run-time (to 4.8G), I do have larger memory 8G, but it still ran out out of memory (at 2.61G, 5.59G) or in the docker-compose.yaml? (But either changed MAIA_MAX_WORKERS=8 or changed memory_limit, It still got errors and failed)
Is that means: I could try to install the "mmdet" branch for MAIA module for this problem? Any suggestions would be appreciated. Thanks a lot. |
Beta Was this translation helpful? Give feedback.
-
|
I assume that you rebuilt and restarted your instance after each change in .env? You are probably affected by biigle/maia#64. Please try to lower the number even further, maybe it works at some point. If you want to investigate this further, here is the part of the code that tries to estimate the memory cunsumption. There is already a badly estimated magic number in there and it's obvious that the estimation still doesn't work correctly.
No, sorry. This branch only contains a Dockerfile for PyTorch. We haven't ported any actual code yet. |
Beta Was this translation helpful? Give feedback.
-
Yes!! Finially work when It down to 3.2G (I have 8G, I tried 7, 6.4, 6, 5.6 (failed) -> 3.2). It's so "magic"^^. Thanks!! |
Beta Was this translation helpful? Give feedback.
-
|
Sorry for the bad implementation there. This will hopefully get better with the PyTorch implementation. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for the hard work to maintain such a big OSS. It's tough, not "bad", and certainly would be better and greater. |
Beta Was this translation helpful? Give feedback.
Answer selected by
cywhale
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
By default MAIA in the GPU branch is configured to expect 16 GB of GPU memory (the number in the .env file is actually 15 GB to avoid biigle/maia#64 most of the time). Do you have this amount of GPU memory? If not, try to lower this number, rebuild and restart your BIIGLE instance.
If you do have this amount of GPU memory, you are probably affected by biigle/maia#64. In this case you can still try to lower the number and see when/if it works.
We likely won't address biigle/maia#64 any more but rather port all the code to PyTorch/OpenMMLab.