Reduce memory usage of board representation

[Trevoke]

Hello!
I have just built the new version of the tool with the comment markers and it's exactly what I needed, thank you so much!

When doing batch analysis, I have the program running for hours, and that means the GTP logs equally build up, and eventually take a lot of RAM. I would like to have a max size for the GTP logs, such that q5go would only keep some amount of the most recent messages, so I don't have to worry about running out of RAM on my local machine.

Thanks!

[bernds]

It's already doing that (because the Qt text widget eventually crashes if you just keep appending to it). If you look in qgtp.cpp, append_text, it has a limit of 200 lines. Once that is reached, it rotates out an early line for each now one.

Maybe that isn't working, but are you sure the GTP log is what's eating the memory (as opposed to, for example, KataGo caching lots of positions)?

[Trevoke]

Well, I am running katago from a Google cloud engine box and connecting q5go to it through ssh, so I don't think that's the problem. I'll run q5go batch analysis for a bit tonight and pay attention to memory usage both on local and remote, to be sure. I also understand that RAM is not necessarily released to the OS even if it is freed, so I'll take what I see with a grain of salt.

…

-- Aldric. Sent from a mobile device.

On Sat, May 14, 2022, 10:24 Bernd Schmidt ***@***.***> wrote: It's already doing that (because the Qt text widget eventually crashes if you just keep appending to it). If you look in qgtp.cpp, append_text, it has a limit of 200 lines. Once that is reached, it rotates out an early line for each now one. Maybe that isn't working, but are you sure the GTP log is what's eating the memory (as opposed to, for example, KataGo caching lots of positions)? — Reply to this email directly, view it on GitHub <#61 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAQSSLOOC53TF5A6HSVLTDVJ6ZTFANCNFSM5V5TLTMQ> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[Trevoke]

Alright, so I left my computer running overnight with top on both my local box and the box running katago.

In the morning, the box with q5go was frozen, and top showed it was using all of the RAM. The box running katago only was using about 20% of its RAM.

So ... It may not be the GTP logs, but something is eating all the RAM without ever releasing it.

[Trevoke]

Oh! I should mention, in case it makes a difference, that I'm on Linux.

[bernds]

Hmm. I ran it for a while with the valgrind leak checker and it didn't have very much to complain about. You could try doing the same with "valgrind --leak-check=full q5go", ideally on a build with debug symbols.

How exactly did you run it? A batch analysis, or just live analysis in a board window?

[Trevoke]

Happy to run it with valgrind, though I'll request you help me build q5go with debug symbols, I don't know how to do that :) This was a batch analysis, so it was churning through multiple games, about 6 seconds per board position, adding max of 10 variations per board position.

…

On Sun, May 15, 2022 at 11:46 AM Bernd Schmidt ***@***.***> wrote: Hmm. I ran it for a while with the valgrind leak checker and it didn't have very much to complain about. You could try doing the same with "valgrind --leak-check=full q5go", ideally on a build with debug symbols. How exactly did you run it? A batch analysis, or just live analysis in a board window? — Reply to this email directly, view it on GitHub <#61 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAQSSMPWZAOI3QLOYZ4GULVKEL3TANCNFSM5V5TLTMQ> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[bernds]

One way of doing that is to find the CONFIG line at the top of q5go.pro and replace "release" with "debug". There's probably some command line way of doing it but I'm not actually all that familiar with qmake.

How much analysisPVLen were you giving KataGo? Over enough time the number of positions might add up eventually. Each one probably takes a few KB, and if you're computing millions of them... How much is "all the RAM"?

Perhaps it needs an "autosave and close when finished" checkbox.

[Trevoke]

The impression that I had was that the RAM kept on building up even if I saved and closed the files. (but I will look at this now, just finished compiling the debug build).

I don't know if it matters how katago was configured since katago is running on a different box?

[Trevoke]

All the RAM is 10+GB (total RAM is 16GB).

Also, valgrind experiment is on hold for me at the moment because this box is broken (long story) so I can't install the glibc debug symbols and I need to troubleshoot that first. Sigh, sigh, sigh.

[bernds]

It would be helpful to know if the RAM does go down if you close files.

[bernds]

I've made an experimental branch called test-lower-memory. This tries to discard state from board positions and recompute it as necessary. If you could test this to see if it helps with your memory consumption problems, we'll know if this is the right direction to investigate.

It's not recommended for general use, there are very probably things that break.

[Trevoke]

I was unable to compile it, I came across this problem:

../src/goboard.cc: In member function ‘void go_board::add_stone(int, int, stone_color, bool)’:
../src/goboard.cc:886:12: error: missing template arguments before ‘dw’
  886 |  discarder dw (m_units_w, !m_store_units);
      |            ^~
../src/goboard.cc:887:12: error: missing template arguments before ‘db’
  887 |  discarder db (m_units_b, !m_store_units);
      |            ^~
../src/goboard.cc: In member function ‘bool go_board::valid_move_p(int, int, stone_color)’:
../src/goboard.cc:996:12: error: missing template arguments before ‘dw’
  996 |  discarder dw (m_units_w, !m_store_units);
      |            ^~
../src/goboard.cc:997:12: error: missing template arguments before ‘db’
  997 |  discarder db (m_units_b, !m_store_units);
      |            ^~
make: *** [Makefile:1839: temp_/goboard.o] Error 1

[Trevoke]

Also, I am unable to launch valgrind against q5go at the moment, I get the following error:

q5go-debug/bin/q5go: /lib/x86_64-linux-gnu/libQt5Core.so.5: version `Qt_5.14' not found (required by q5go-debug/bin/q5go)

I am looking for what I need to install to make this error go away but I have very little experience with C++ / qt5 development so I'm not totally sure how to find the ubuntu packages I need (or what I need to install from source).

[Trevoke]

Okay -- got the master branch compiled with debug, and valgrind running on it. I'll run batch analysis with valgrind running for, well, a few hours, and we'll see what that comes out to be.

The lower-memory branch is in error at the moment, I'm not sure I know enough to fix that on my own unfortunately.

I will also let you know if I am able to notice a memory change when a game that went through batch analysis is saved and closed.

[Trevoke]

Okay so on the master branch, built with debug, when I ran batch analysis (two files selected, just enough to start he batch work, really), here's the end of the valgrind output.

When I started this, memory usage was around 2.97GB. It's worth noting that when I saved and closed the first file after analysis was complete, memory usage went DOWN ... But not a lot. It went from 4.28GB to 4.26GB. So analysis for a single game took a little over one GB of RAM, which remained in use after I saved and closed the game.

==38850== HEAP SUMMARY:
==38850==     in use at exit: 276,395,321 bytes in 2,874,233 blocks
==38850==   total heap usage: 47,968,532 allocs, 45,094,299 frees, 29,473,943,274 bytes allocated

==38850== 49,152 bytes in 1 blocks are possibly lost in loss record 16,695 of 16,726
==38850==    at 0x483DD99: calloc (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==38850==    by 0xB77407A: ??? (in /usr/lib/x86_64-linux-gnu/libcairo.so.2.11600.0)
==38850==    by 0xB7DA084: ??? (in /usr/lib/x86_64-linux-gnu/libcairo.so.2.11600.0)
==38850==    by 0xB7DAA07: ??? (in /usr/lib/x86_64-linux-gnu/libcairo.so.2.11600.0)
==38850==    by 0xB7DAAA0: ??? (in /usr/lib/x86_64-linux-gnu/libcairo.so.2.11600.0)
==38850==    by 0xB7ADF6A: cairo_surface_create_similar_image (in /usr/lib/x86_64-linux-gnu/libcairo.so.2.11600.0)
==38850==    by 0xB5A46D0: gdk_window_create_similar_image_surface (in /usr/lib/x86_64-linux-gnu/libgdk-3.so.0.2404.16)
==38850==    by 0xB57C447: gdk_cairo_surface_create_from_pixbuf (in /usr/lib/x86_64-linux-gnu/libgdk-3.so.0.2404.16)
==38850==    by 0xAFACB32: ??? (in /usr/lib/x86_64-linux-gnu/libgtk-3.so.0.2404.16)
==38850==    by 0xAFAD277: ??? (in /usr/lib/x86_64-linux-gnu/libgtk-3.so.0.2404.16)
==38850==    by 0xAFAD393: ??? (in /usr/lib/x86_64-linux-gnu/libgtk-3.so.0.2404.16)
==38850==    by 0xAFAD4DF: ??? (in /usr/lib/x86_64-linux-gnu/libgtk-3.so.0.2404.16)
==38850== 
==38850== 265,617 (176 direct, 265,441 indirect) bytes in 11 blocks are definitely lost in loss record 16,710 of 16,726
==38850==    at 0x483BE63: operator new(unsigned long) (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==38850==    by 0x3890D6: load_sgf(IODeviceAdapter const&) (sgfload.cc:121)
==38850==    by 0x38A840: SGFPreview::setPath(QString) (sgfpreview.cpp:76)
==38850==    by 0x38E612: QtPrivate::FunctorCall<QtPrivate::IndexesList<0>, QtPrivate::List<QString const&>, void, void (SGFPreview::*)(QString)>::call(void (SGFPreview::*)(QString), SGFPreview*, void**) (qobjectdefs_impl.h:152)
==38850==    by 0x38E4C1: void QtPrivate::FunctionPointer<void (SGFPreview::*)(QString)>::call<QtPrivate::List<QString const&>, void>(void (SGFPreview::*)(QString), SGFPreview*, void**) (qobjectdefs_impl.h:185)
==38850==    by 0x38E3DE: QtPrivate::QSlotObject<void (SGFPreview::*)(QString), QtPrivate::List<QString const&>, void>::impl(int, QtPrivate::QSlotObjectBase*, QObject*, void**, bool*) (qobjectdefs_impl.h:414)
==38850==    by 0x5B48327: QMetaObject::activate(QObject*, int, int, void**) (in /usr/lib/x86_64-linux-gnu/libQt5Core.so.5.12.8)
==38850==    by 0x4C5A128: QFileDialog::currentChanged(QString const&) (in /usr/lib/x86_64-linux-gnu/libQt5Widgets.so.5.12.8)
==38850==    by 0x4C6447E: QFileDialogPrivate::_q_currentChanged(QModelIndex const&) (in /usr/lib/x86_64-linux-gnu/libQt5Widgets.so.5.12.8)
==38850==    by 0x4C6B2F0: ??? (in /usr/lib/x86_64-linux-gnu/libQt5Widgets.so.5.12.8)
==38850==    by 0x5B481CF: QMetaObject::activate(QObject*, int, int, void**) (in /usr/lib/x86_64-linux-gnu/libQt5Core.so.5.12.8)
==38850==    by 0x5AD7EE6: QItemSelectionModel::currentChanged(QModelIndex const&, QModelIndex const&) (in /usr/lib/x86_64-linux-gnu/libQt5Core.so.5.12.8)
==38850== 
==38850== LEAK SUMMARY:
==38850==    definitely lost: 21,296 bytes in 33 blocks
==38850==    indirectly lost: 293,053 bytes in 5,703 blocks
==38850==      possibly lost: 59,607 bytes in 101 blocks
==38850==    still reachable: 275,820,517 bytes in 2,866,742 blocks
==38850==                       of which reachable via heuristic:
==38850==                         length64           : 21,144 bytes in 288 blocks
==38850==                         newarray           : 6,728 bytes in 75 blocks
==38850==         suppressed: 0 bytes in 0 blocks
==38850== Reachable blocks (those to which a pointer was found) are not shown.
==38850== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==38850== 
==38850== For lists of detected and suppressed errors, rerun with: -s
==38850== ERROR SUMMARY: 78 errors from 78 contexts (suppressed: 2 from 2)

[bernds]

Doesn't really look like any big leaks.
When you said you closed the file, did it disappear from the "completed jobs" list in the batch analysis window? Only at that point is all the memory freed (well, it ought to be).

I fixed up the branch and it should compile now. I had one build directory configured for a different C++ standard and so it worked for me.

[Trevoke]

It did disappear from the completed jobs list in the batch analysis window, yes! That freed the .02 GB as I mentioned.

I will try to compile the branch now.

[Trevoke]

Alright, the "test-lower-memory" branch yielded the following result for batch analysis after one game:

• total additional RAM taken up by the analysis process: 0.2GB (7.70GB to 7.90GB)
• total RAM freed up when opening the completed game, saving it, and closing it: ~0.01GB

It's just one game, so it's not comprehensive analysis, but 0.2GB is way better than 1.3GB! 200MB is probably still much more than is strictly required to keep in memory for any one game

Second game, 0.25GB used, ~0.01GB freed.
So after two games, RAM went up from 7.70GB to 8.20GB approximately.

Hope this helps!

[Trevoke]

Okay, another thing to note for some additional information, I've now batch-analyzed 6 games and the RAM usage on my box peaked at 8.5GB and after saving/closing the sixth game, it's back down to 8.43GB.

It's not very scientific because I have done or two other small things on the computer, but this is sufficiently promising that I'm going to take another shot at leaving this running all night, and hopefully wake up to another dozen or so games processed and a computer not crashed :D

[Trevoke]

Woke up to an additional 37 games batch-analyzed, a non-crashed computer, and a RAM usage at 12.8GB.

So it seems like analyzing 43 games total may lead to ~5GB used in RAM.

Keeping this running (I have another 20 or so games in this batch), I'll let you know what shakes out, but this is obviously a monumental improvement!

[Trevoke]

As a note, saving/closing games now, after 46 games, yields no noticeable change in RAM availability.

And now a lot of measurements will go out the door because I'm going to start using this computer for other things too :D

[bernds]

So I take it the branch has fixed the issue for you? Have you been using it for other purposes as well and has it been stable for you?

I'm debating whether to use this solution for the main branch. The memory savings are nice, but I don't really like it from a robustness point of view.

[Trevoke]

The branch is stable for general SGF editing and batch analysis. I haven't tried using q5go as a client for online play so I cannot comment on that.

FWIW, having done all of the Dosaku games now, the next batch of games I'd like to work on is the Shusaku games, which is 470 games, so I'm happy about these memory changes, they at least will allow me to keep it going overnight.

I understand if they're not perfect and you don't want to merge them in, though I would request that you keep looking for a solution that satisfies you more :)