Use eval broker lock when reading/modifying delay heap

[preetapan]

Fixes #4607

runDelayedEvalsWatcher attempted to read and write from the delay heap it uses without synchronization. This caused a panic due to the race mentioned in #4607. This PR uses the existing eval broker mutex to synchronize access.

[tantra35]

Why you use Lock/Unlock in nextDelayedEval, but this method is read only

[preetapan]

@tantra35 because nextDelayedEval reads from the heap (via the channel trigger delayedEvalsUpdateCh) while another goroutine modifies it (processEnqueue)

EDIT - I think you mean why I didn't use RLock? That's a valid question. Locking for reads should be sufficient here.

[jippi]

another panic bites the dust 🎉

[tantra35]

@preetapan I mean why not used Rlock/RUnlock instead Lock/Unlock?

[github-actions]

I'm going to lock this pull request because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active contributions.
If you have found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.