[Bug] Webhooks opening infinite amount of files

[mphbig]

Request Type

Bug

Work Environment

Question	Answer
OS version (server)	Debian 9
OS version (client)	Windows 10
TheHive version / git hash	4.0.3-1
Package Type	DEB
Browser type & version	Chromium and Firefox latest

Problem Description

When sending a webhook message to a webhook listener (Nodered, Shuffle, ...) TheHive opens "files" (what linux calls files but its marked as "pipe" using lsof -u thehive).
These files are opened but never closed, and by the end of the day we reach the hard limit of 65536 files as our instance has a lot of traffic.

We reproduced this issue in clean virtual machines, with different distributions (Debian 9, Debian 10 and Oracle Linux 8) and different JRE versions (8 LTS, 11 LTS, 15) from Adopt OpenJDK. We got the same behaviour.
Each distribution was up-to-date as of 2021-01-10.

Steps to Reproduce

1. Install TheHive 4
2. Setup webhook
3. Do any action that will send a webhook message
4. Watch the opened file count increase as TheHive sends webhook message with the command lsof -u thehive | wc -l (assuming TheHive runs as user thehive)

Complementary information

This is the exception that appears in the logs when reaching the opened file hard limit.

2021-01-07 10:50:21,306 [ERROR] from akka.io.TcpListener in application-akka.actor.default-dispatcher-5342 [|] Accept error: could not accept new connection
java.io.IOException: Trop de fichiers ouverts
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:421)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:249)
        at akka.io.TcpListener.acceptAllPending(TcpListener.scala:128)
        at akka.io.TcpListener$$anonfun$bound$1.applyOrElse(TcpListener.scala:96)
        at akka.actor.Actor.aroundReceive(Actor.scala:537)
        at akka.actor.Actor.aroundReceive$(Actor.scala:535)
        at akka.io.TcpListener.aroundReceive(TcpListener.scala:36)
        at akka.actor.ActorCell.receiveMessage(ActorCell.scala:577)
        at akka.actor.ActorCell.invoke(ActorCell.scala:547)
        at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:270)
        at akka.dispatch.Mailbox.run(Mailbox.scala:231)
        at akka.dispatch.Mailbox.exec(Mailbox.scala:243)
        at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
        at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
        at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
        at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175)

You may find the logs with level debug when sending a webhook here : https://gist.github.com/mphbig/bac95e62e77e61e1149f005f74654ed1

[jmbodelon]

Same here in 4.0.4!

Any partial solution? Or just stop the webhooks?

[mphbig]

Solution implemented on our instance is a restart of TheHive twice a day.
Clearly not ideal but only solution I could come up with.

Another solution would be to rise the limit of opened files per user, but 65536 seems already pretty high imho.

[murphyt-mosaic451]

I also have a similar situation, rise of the files open until the server just crashes.
I tried changing the timeouts to be under 5 Mintues.. and files open are still rising.

[nadouani]

Hello guys, we will investigate this for 4.1