Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Log when NN eval takes longer than one second. #1768

Open
mooskagh opened this issue Jul 31, 2022 · 4 comments
Open

Log when NN eval takes longer than one second. #1768

mooskagh opened this issue Jul 31, 2022 · 4 comments
Labels
good first issue Good for newcomers

Comments

@mooskagh
Copy link
Member

During WCCC and testing, we saw cases where the search couldn't stop for up to 27 seconds.

We suspect that it's backend that takes so long to compute a batch.

It would be nice to output a message to a log file when that happens, when NN takes longer than a second or so.
@mooskagh mooskagh added the good first issue Good for newcomers label Jul 31, 2022
@mooskagh
Copy link
Member Author

Now I suspect that's actually the tablebase probing that takes so long. So it would make sense to add similar logging there too.

@KarlKfoury
Copy link
Contributor

KarlKfoury commented Aug 29, 2024 •
edited
Loading

Does NN evaluation happen here?

@mooskagh
Copy link
Member Author

It's called here:

lc0/src/mcts/search.cc

Line 2192 in 08b41c4

void SearchWorker::RunNNComputation() { computation_->ComputeBlocking(); }

There are many backends, all of them implement ComputeBlocking().

KarlKfoury added a commit to KarlKfoury/lc0 that referenced this issue Aug 30, 2024
@KarlKfoury
calculating time elapsed from start to end of nn eval + logging if ti…
4045a1c 
…me > 1s

fixing issue LeelaChessZero#1768
@KarlKfoury
Copy link
Contributor

what about tablebase probing? i assume it happens here ?

KarlKfoury added a commit to KarlKfoury/lc0 that referenced this issue Aug 30, 2024
@KarlKfoury
added log to calculate database probing time
c48f5a7 
fixing LeelaChessZero#1768
we brought one
KarlKfoury added a commit to KarlKfoury/lc0 that referenced this issue Aug 30, 2024
@KarlKfoury
logging if tablebase probing duration > 1s
943ab53 
adressing issue LeelaChessZero#1768
@KarlKfoury KarlKfoury mentioned this issue Aug 30, 2024
Closed
KarlKfoury added a commit to KarlKfoury/lc0 that referenced this issue Aug 31, 2024
@KarlKfoury
calculating database probing time and logging a message if time > 1s
e9b7745 
adressing issue LeelaChessZero#1768
KarlKfoury added a commit to KarlKfoury/lc0 that referenced this issue Aug 31, 2024
@KarlKfoury
calculating neural network evaluation time and logging a message if t…
3871405 
…ime > 1s

adressing issue LeelaChessZero#1768
This was referenced Aug 31, 2024
Closed
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
good first issue Good for newcomers
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@mooskagh @KarlKfoury