Get rid of time.After in for loops

[Shaddoll]

What changed?
Replace time.After in for loops with NewTimer + Reset or Sleep

Why?
before go 1.23, time.After in for loops can use a lot of memory if the timeout is too long and don't expire.
after go 1.23, time.After does allocation a lot in for loops.
NewTimer + Reset gets rid of this problem.

How did you test it?
existing unit tests

Potential risks

Release notes

Documentation Changes

[codecov]

Codecov Report

Attention: Patch coverage is 47.82609% with 12 lines in your changes missing coverage. Please review.

Project coverage is 73.20%. Comparing base (11c3626) to head (a64f23c).
Report is 3 commits behind head on master.

Files with missing lines	Patch %	Lines
service/history/queue/timer_queue_processor.go	0.00%	7 Missing ⚠️
service/history/queue/transfer_queue_processor.go	28.57%	4 Missing and 1 partial ⚠️

Additional details and impacted files

Files with missing lines	Coverage Δ
common/cache/metricsScopeCache.go	89.47% <100.00%> (+0.58%)	⬆️
...ervice/history/engine/engineimpl/query_workflow.go	50.73% <100.00%> (ø)
service/worker/scanner/tasklist/scavenger.go	80.35% <100.00%> (+0.54%)	⬆️
service/history/queue/transfer_queue_processor.go	35.38% <28.57%> (-0.04%)	⬇️
service/history/queue/timer_queue_processor.go	24.08% <0.00%> (-0.50%)	⬇️

... and 7 files with indirect coverage changes

---

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 11c3626...a64f23c. Read the comment docs.

[Groxx]

I got curious and dug around in how timers are created and scheduled, to see what the initial-0 would do, because I hadn't seen that pattern before.

Looks like it puts it into the timer heap, sorts it, and it fires ~immediately, so it's safe both before and after 1.23, though slightly wasteful (but it's a one-time tiny cost so it seems fine to ignore it).

1.23 is substantially more complex, but seems to be essentially the same. And probably has some optimizations for immediate-firing, given how the blog post talks about it more-reliably following both branches with very small delays. So safe and no worse + probably better there too 👍

---

Since this very nearly gets rid of ALL of our time.After use, yea, I think the minor added complexity is fine. It's a better habit for us to be in, since we have a significant amount of fast + perpetual loops, and we can almost just ban After now to prevent a costly leak from sneaking back in.

Check the ticker instances, since they seem kinda straightforward to me? Otherwise LGTM and seems fine either way.