Previewing alerts fails with "too many substreams open" due to case similarity process

[fl0wc0ntr0l]

Request Type

Bug

Work Environment

Question	Answer
OS version (server)	Ubuntu 16.04 LTS
OS version (client)	Win10
TheHive version / git hash	2.12.1
Package Type	Docker
Browser type & version	Chrome v60

Problem Description

Attempting to preview certain alerts gives an error:

AlertEventCtrl: Cannot open substream for key 'Some([redacted])': too many substreams open

Steps to Reproduce

Not sure. This only seems to happen sporadically. Some alerts seem to cause this error, some don't. At a guess, alerts that exceed some unwritten threshold of similar cases seem to cause this error.

Possible Solutions

The problem appears to originate when finding similar cases to a particular alert. This seems to open too many substreams (to what, I have no idea) and cause previewing the alert to fail. See logs below.

Complementary information

Log info of a failure:

[info] o.e.ErrorHandler - GET /api/alert/[redacted]?similarity=1 returned 500
java.lang.IllegalStateException: Cannot open substream for key 'Some([redacted])': too many substreams open
        at akka.stream.impl.fusing.GroupBy$$anon$1.onPush(StreamOfStreams.scala:294)
        at akka.stream.impl.fusing.GraphInterpreter.processPush(GraphInterpreter.scala:747)
        at akka.stream.impl.fusing.GraphInterpreter.processEvent(GraphInterpreter.scala:710)
        at akka.stream.impl.fusing.GraphInterpreter.execute(GraphInterpreter.scala:616)
        at akka.stream.impl.fusing.GraphInterpreterShell.runBatch(ActorGraphInterpreter.scala:471)
        at akka.stream.impl.fusing.GraphInterpreterShell.receive(ActorGraphInterpreter.scala:423)
        at akka.stream.impl.fusing.ActorGraphInterpreter.akka$stream$impl$fusing$ActorGraphInterpreter$$processEvent(ActorGraphInterpreter.scala:603)
        at akka.stream.impl.fusing.ActorGraphInterpreter.akka$stream$impl$fusing$ActorGraphInterpreter$$shortCircuitBatch(ActorGraphInterpreter.scala:594)
        at akka.stream.impl.fusing.ActorGraphInterpreter$$anonfun$receive$1.applyOrElse(ActorGraphInterpreter.scala:619)
        at akka.actor.Actor$class.aroundReceive(Actor.scala:497)

[To-om]

Hi @Amateur-Professional,
This occurs when you have more than 100 cases similar with the current alert. Can you confirm you are in this situation ?

[fl0wc0ntr0l]

I would imagine so, or I wouldn't be getting this error ;)

How exactly would I check that anyway? The preview fails, so I can't see it from there, and the API call fails due to the nature of the error anyway. It doesn't appear that there would be any way I could check it.

[fl0wc0ntr0l]

After clearing out some old cases we no longer get the error, but that is a crappy "fix" for some undocumented, unconfigurable upper limit to related cases. Is there anything in the works to potentially adjust this? Especially since I am sure it will happen again.

[fl0wc0ntr0l]

Any updates for this? We keep seeing this issue, and with no other way to preview alerts, we have to either cut our past cases, or mark alerts as read without reading them. Neither is really acceptable for a case management system.

[fl0wc0ntr0l]

@To-om with our current case load we cannot keep cases around for longer than a week. We lose a ridiculous amount of data by pruning the cases to no longer show the error. We need some sort of solution, or at least some confirmation that somebody is working on it.

[derDuffy]

I still have the problem. I am running 3.0.8 and I have the limit set to maxSimilarCases to 30.
Anything I am missing here ?

[raomin]

Hi there, I have set maxSimilarCases to 1000 and I still get the same issue. Basically, regardless of what I set the maxSimilarCases to, I still get the error when trying to open an alert with more than 100 similar cases....
I'm using 3.0.9

[derDuffy]

Hello again,

I am still having the problems with 3.10.0.
I set maxSimilarCases to 30.

Any idea what can be done, I am currently filtering out observables that cause the issue when creating alerts but that's not really the intention :)

[saadkadhi]

@derDuffy @raomin we will look at it ASAP. Thank you for your patience.

[To-om]

@derDuffy @raomin Can you provide the related logs ?

[lsoumille]

Thanks for your answers. I'm working with @raomin on this. The maxSimilarCases was taken into account after killing all JVMs on the server and then restarting TheHive and Elasticsearch

[derDuffy]

I can provide logs, let me try the kill-all-then-restart method from above first :)

[derDuffy]

I did kill all JVMs and rebooted the systems, no effect. I sill have the error message. Here are the logs

#####################

2018-06-08 10:42:32,226 [INFO] from org.elastic4play.ErrorHandler in application-akka.actor.default-dispatcher-1218 - GET /api/alert/8123a1f876f2e618f7deb33a28fb7826?similarity=1 returned 500
java.lang.IllegalStateException: Cannot open substream for key 'Some(AWH7IhFYTT7DxHBKCCLT)': too many substreams open
at akka.stream.impl.fusing.GroupBy$$anon$1.onPush(StreamOfStreams.scala:298)
at akka.stream.impl.fusing.GraphInterpreter.processPush(GraphInterpreter.scala:499)
at akka.stream.impl.fusing.GraphInterpreter.processEvent(GraphInterpreter.scala:462)
at akka.stream.impl.fusing.GraphInterpreter.execute(GraphInterpreter.scala:368)
at akka.stream.impl.fusing.GraphInterpreterShell.runBatch(ActorGraphInterpreter.scala:571)
at akka.stream.impl.fusing.GraphInterpreterShell$AsyncInput.execute(ActorGraphInterpreter.scala:457)
at akka.stream.impl.fusing.GraphInterpreterShell.processEvent(ActorGraphInterpreter.scala:546)
at akka.stream.impl.fusing.ActorGraphInterpreter.akka$stream$impl$fusing$ActorGraphInterpreter$$processEvent(ActorGraphInterpreter.scala:728)
at akka.stream.impl.fusing.ActorGraphInterpreter.akka$stream$impl$fusing$ActorGraphInterpreter$$shortCircuitBatch(ActorGraphInterpreter.scala:718)
at akka.stream.impl.fusing.ActorGraphInterpreter$$anonfun$receive$1.applyOrElse(ActorGraphInterpreter.scala:744)
at akka.actor.Actor.aroundReceive(Actor.scala:517)
at akka.actor.Actor.aroundReceive$(Actor.scala:515)
at akka.stream.impl.fusing.ActorGraphInterpreter.aroundReceive(ActorGraphInterpreter.scala:653)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:527)
at akka.actor.ActorCell.invoke(ActorCell.scala:496)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)
at akka.dispatch.Mailbox.run(Mailbox.scala:224)
at akka.dispatch.Mailbox.exec(Mailbox.scala:234)
at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)