CurseForge install times out

[uhthomas]

Describe the problem

I'm trying to run an 'All The Mods 8' server, but it keeps timing out whilst downloading assets. Every time the container restarts, it downloads just a few more mods before eventually timing out. It looks like the overall timeout is 45s, which is pretty low imo. Is there anyway to increase this? I don't see anything in the documentation.

minecraft-cf-atm8-0 minecraft-server [mc-image-helper] 11:33:08.118 ERROR : 'install-curseforge' command failed. Version is 1.26.2
minecraft-cf-atm8-0 minecraft-server reactor.core.Exceptions$ReactiveException: reactor.netty.internal.shaded.reactor.pool.PoolAcquireTimeoutException: Pool#acquire(Duration) has been pending for more than the configured timeout of 45000ms
minecraft-cf-atm8-0 minecraft-server    at reactor.core.Exceptions.propagate(Exceptions.java:408)
minecraft-cf-atm8-0 minecraft-server    at reactor.core.publisher.BlockingSingleSubscriber.blockingGet(BlockingSingleSubscriber.java:97)
minecraft-cf-atm8-0 minecraft-server    at reactor.core.publisher.Mono.block(Mono.java:1710)
minecraft-cf-atm8-0 minecraft-server    at me.itzg.helpers.curseforge.CurseForgeInstaller.processModpackZip(CurseForgeInstaller.java:434)
minecraft-cf-atm8-0 minecraft-server    at me.itzg.helpers.curseforge.CurseForgeInstaller.downloadAndProcessModpackZip(CurseForgeInstaller.java:385)
minecraft-cf-atm8-0 minecraft-server    at me.itzg.helpers.curseforge.CurseForgeInstaller.processModPack(CurseForgeInstaller.java:275)
minecraft-cf-atm8-0 minecraft-server    at me.itzg.helpers.curseforge.CurseForgeInstaller.installByRetrievingModpackZip(CurseForgeInstaller.java:169)
minecraft-cf-atm8-0 minecraft-server    at me.itzg.helpers.curseforge.CurseForgeInstaller.install(CurseForgeInstaller.java:138)
minecraft-cf-atm8-0 minecraft-server    at me.itzg.helpers.curseforge.CurseForgeInstaller.install(CurseForgeInstaller.java:102)
minecraft-cf-atm8-0 minecraft-server    at me.itzg.helpers.curseforge.InstallCurseForgeCommand.call(InstallCurseForgeCommand.java:180)
minecraft-cf-atm8-0 minecraft-server    at me.itzg.helpers.curseforge.InstallCurseForgeCommand.call(InstallCurseForgeCommand.java:24)
minecraft-cf-atm8-0 minecraft-server    at picocli.CommandLine.executeUserObject(CommandLine.java:2041)
minecraft-cf-atm8-0 minecraft-server    at picocli.CommandLine.access$1500(CommandLine.java:148)
minecraft-cf-atm8-0 minecraft-server    at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2461)
minecraft-cf-atm8-0 minecraft-server    at picocli.CommandLine$RunLast.handle(CommandLine.java:2453)
minecraft-cf-atm8-0 minecraft-server    at picocli.CommandLine$RunLast.handle(CommandLine.java:2415)
minecraft-cf-atm8-0 minecraft-server    at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2273)
minecraft-cf-atm8-0 minecraft-server    at picocli.CommandLine$RunLast.execute(CommandLine.java:2417)
minecraft-cf-atm8-0 minecraft-server    at picocli.CommandLine.execute(CommandLine.java:2170)
minecraft-cf-atm8-0 minecraft-server    at me.itzg.helpers.McImageHelper.main(McImageHelper.java:132)
minecraft-cf-atm8-0 minecraft-server    Suppressed: java.lang.Exception: #block terminated with an error
minecraft-cf-atm8-0 minecraft-server            at reactor.core.publisher.BlockingSingleSubscriber.blockingGet(BlockingSingleSubscriber.java:99)
minecraft-cf-atm8-0 minecraft-server            ... 18 common frames omitted
minecraft-cf-atm8-0 minecraft-server Caused by: reactor.netty.internal.shaded.reactor.pool.PoolAcquireTimeoutException: Pool#acquire(Duration) has been pending for more than the configured timeout of 45000ms
minecraft-cf-atm8-0 minecraft-server    at reactor.netty.internal.shaded.reactor.pool.AbstractPool$Borrower.run(AbstractPool.java:413)
minecraft-cf-atm8-0 minecraft-server    at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:68)
minecraft-cf-atm8-0 minecraft-server    at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:28)
minecraft-cf-atm8-0 minecraft-server    at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
minecraft-cf-atm8-0 minecraft-server    at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
minecraft-cf-atm8-0 minecraft-server    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
minecraft-cf-atm8-0 minecraft-server    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
minecraft-cf-atm8-0 minecraft-server    at java.base/java.lang.Thread.run(Unknown Source)
minecraft-cf-atm8-0 minecraft-server [init] ERROR failed to auto-install CurseForge modpack

Container definition

https://github.com/uhthomas/automata/blob/294172e7fc94f980a90ed97072a681983e935f23/k8s/unwind/minecraft/cf_atm8/stateful_set_list.cue#L34-L68

Container logs

See above.

[itzg]

That 45 seconds is not an overall timeout...that means all connections are being used and nothing has freed up in 45 seconds. Your network is running incredibly slow given how small mod files are. Set DEBUG to "true" to inspect the progress of each file download. Of if you're still stuck, post those debug logs here.

[uhthomas]

To be fair, the network was running pretty slow. I did some tinkering and the network came back to a reasonable speed and succeeded as expected. I would suggest that slow networks shouldn't block the deployment of this software, though it doesn't seem many people have this problem.

[itzg]

I would suggest that slow networks shouldn't block the deployment of this software

I don't understand the suggestion. It needs to download the mods somehow, so...how would it magically proceed without the mods being downloaded and/or a sufficient network to download them? There's always the options to pre-download mods rather than using the AUTO_CURSEFORGE automation.

[uhthomas]

Well, mods were downloading, just slowly. Given enough time it would have worked. Though, I understand if it's not something the project is interested in supporting given most networks should be faster than 1mbps.

[itzg]

I'm still confused about what you're thinking can be solved within the scope of AUTO_CURSEFORGE mechanism that I didn't already mention. It seems like network performance is an important facet of a server in general, so there has to be some kind of baseline assumption about the network quality.

I'm open to ideas.

[uhthomas]

An internet connection and a local network connection will have very different qualities. In my specific scenario, my ISP is taking their leisurely time to actually install a fibre connection and as such our internet connection is currently a 5G hotspot. The connection is mostly okay, sometimes less so. The local network however is 20Gbe, where we play on the server.

Again, I'm not sure if there's really any action to be taken here. It would be nice if the network client in mc-image-helper were more tolerant of slow internet connections, but I imagine that this issue will be less relevant as general connectivity continues to improve globally.

My main confusion was that nothing had stalled. Maybe 50~ of the mods had downloaded, and more mods would continue to be downloaded every time the kubelet restarted the pod. So, it seemed unnecessary to stop this process that would have eventually succeeded, even if not quickly.

[itzg]

Sorry, I later realized that playing on LAN would be a case where internet network performance cannot be assumed.

I'll do some research about gracefully blocking on slow connections with the networking library I'm using there in the helper.

[itzg]

Implementation note

pendingAcquireTimeout from connection pool section seems to have a possible value of -1 which perhaps means infinite blocking. Need to expose that as an option in SharedFetchArgs