Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug] Duplicated entities after "db.janusgraph.forceDropAndRebuildIndex: true" with Elasticsearch index #2333

Closed
jpferrero opened this issue Feb 3, 2022 · 1 comment
Closed

[Bug] Duplicated entities after "db.janusgraph.forceDropAndRebuildIndex: true" with Elasticsearch index #2333

jpferrero opened this issue Feb 3, 2022 · 1 comment
Labels
bug TheHive4 TheHive4 related issues
Milestone
4.1.18

Comments

@jpferrero
Copy link

Request Type

Bug

Work Environment

Question Answer
OS version (server) CentOS
OS version (client) 7
Virtualized Env. True
Dedicated RAM 16 GB
vCPU 8
TheHive version / git hash 4.17
Package Type RPM
Database Cassandra
Index type Elasticsearch 7.16
Attachments storage Local

If I set the parameter 'db.janusgraph.forceDropAndRebuildIndex: true' when using Elasticsearch as index backend, after a while lots of entities are marked as duplicated and data from observables are lost. This is the log of the reindex

2022-02-03 21:10:36,751 [INFO] from org.thp.scalligraph.models.Database in application-akka.actor.default-dispatcher-13 [|mgmt-296117d3] Reindex job 178b1e59 is running
2022-02-03 21:10:37,182 [INFO] from org.janusgraph.graphdb.database.management.ManagementSystem in Thread-61 [|] Index update job successful for [global5]
2022-02-03 21:10:37,752 [INFO] from org.thp.scalligraph.models.Database in application-akka.actor.default-dispatcher-13 [|mgmt-296117d3] Reindex job 178b1e59 is finished
2022-02-03 21:10:37,891 [INFO] from akka.cluster.singleton.ClusterSingletonManager in application-akka.actor.default-dispatcher-13 [|] Singleton manager starting singleton actor [akka://application/user/integrityCheckSingletonManager/singleton]
2022-02-03 21:10:37,891 [INFO] from akka.cluster.singleton.ClusterSingletonManager in application-akka.actor.default-dispatcher-4 [|] ClusterSingletonManager state change [Start -> Oldest]
2022-02-03 21:10:37,930 [INFO] from akka.cluster.singleton.ClusterSingletonManager in application-akka.actor.default-dispatcher-18 [|] Singleton manager starting singleton actor [akka://application/system/singletonManagerCaseNumberLeader/CaseNumberLeader]
2022-02-03 21:10:37,930 [INFO] from akka.cluster.singleton.ClusterSingletonManager in application-akka.actor.default-dispatcher-18 [|] ClusterSingletonManager state change [Start -> Oldest]
2022-02-03 21:10:38,580 [INFO] from akka.cluster.singleton.ClusterSingletonManager in application-akka.actor.default-dispatcher-4 [|] Singleton manager starting singleton actor [akka://application/user/flowSingletonManager/singleton]
2022-02-03 21:10:38,580 [INFO] from akka.cluster.singleton.ClusterSingletonManager in application-akka.actor.default-dispatcher-21 [|] ClusterSingletonManager state change [Start -> Oldest]
2022-02-03 21:10:38,619 [INFO] from play.api.Play in main [|] Application started (Prod) (no global state)
2022-02-03 21:10:38,942 [INFO] from play.core.server.AkkaHttpServer in main [|] Listening for HTTP on /0:0:0:0:0:0:0:0:8000
2022-02-03 21:10:38,943 [INFO] from akka.cluster.singleton.ClusterSingletonProxy in application-akka.actor.default-dispatcher-21 [|] Singleton identified at [akka://application/system/singletonManagerCaseNumberLeader/CaseNumberLeader]
2022-02-03 21:10:39,593 [INFO] from akka.cluster.singleton.ClusterSingletonProxy in application-akka.actor.default-dispatcher-21 [|] Singleton identified at [akka://application/user/flowSingletonManager/singleton]
2022-02-03 21:10:51,487 [INFO] from akka.cluster.singleton.ClusterSingletonProxy in application-akka.actor.default-dispatcher-4 [|] Singleton identified at [akka://application/user/integrityCheckSingletonManager/singleton]
2022-02-03 21:10:51,487 [INFO] from akka.cluster.singleton.ClusterSingletonProxy in application-akka.actor.default-dispatcher-4 [|] Singleton identified at [akka://application/user/integrityCheckSingletonManager/singleton]
2022-02-03 21:10:51,487 [INFO] from akka.cluster.singleton.ClusterSingletonProxy in application-akka.actor.default-dispatcher-4 [|] Singleton identified at [akka://application/user/integrityCheckSingletonManager/singleton]
2022-02-03 21:10:51,487 [INFO] from akka.cluster.singleton.ClusterSingletonProxy in application-akka.actor.default-dispatcher-4 [|] Singleton identified at [akka://application/user/integrityCheckSingletonManager/singleton]
2022-02-03 21:10:51,488 [INFO] from akka.cluster.singleton.ClusterSingletonProxy in application-akka.actor.default-dispatcher-4 [|] Singleton identified at [akka://application/user/integrityCheckSingletonManager/singleton]
2022-02-03 21:10:51,489 [INFO] from akka.cluster.singleton.ClusterSingletonProxy in application-akka.actor.default-dispatcher-4 [|] Singleton identified at [akka://application/user/integrityCheckSingletonManager/singleton]
2022-02-03 21:10:51,489 [INFO] from akka.cluster.singleton.ClusterSingletonProxy in application-akka.actor.default-dispatcher-4 [|] Singleton identified at [akka://application/user/integrityCheckSingletonManager/singleton]
2022-02-03 21:10:51,489 [INFO] from akka.cluster.singleton.ClusterSingletonProxy in application-akka.actor.default-dispatcher-4 [|] Singleton identified at [akka://application/user/integrityCheckSingletonManager/singleton]
2022-02-03 21:10:51,489 [INFO] from akka.cluster.singleton.ClusterSingletonProxy in application-akka.actor.default-dispatcher-4 [|] Singleton identified at [akka://application/user/integrityCheckSingletonManager/singleton]
2022-02-03 21:10:51,490 [INFO] from akka.cluster.singleton.ClusterSingletonProxy in application-akka.actor.default-dispatcher-4 [|] Singleton identified at [akka://application/user/integrityCheckSingletonManager/singleton]
2022-02-03 21:10:51,490 [INFO] from akka.cluster.singleton.ClusterSingletonProxy in application-akka.actor.default-dispatcher-4 [|] Singleton identified at [akka://application/user/integrityCheckSingletonManager/singleton]
2022-02-03 21:10:51,490 [INFO] from akka.cluster.singleton.ClusterSingletonProxy in application-akka.actor.default-dispatcher-4 [|] Singleton identified at [akka://application/user/integrityCheckSingletonManager/singleton]
2022-02-03 21:10:51,490 [INFO] from akka.cluster.singleton.ClusterSingletonProxy in application-akka.actor.default-dispatcher-4 [|] Singleton identified at [akka://application/user/integrityCheckSingletonManager/singleton]
2022-02-03 21:10:51,636 [INFO] from org.thp.thehive.services.ObservableTypeIntegrityCheckOps in pool-12-thread-1 [|774df931] Found duplicate entities:

  • ObservableType(domain,false)
  • ObservableType(domain,false)
    2022-02-03 21:11:32,575 [INFO] from org.thp.scalligraph.AccessLogFilter in application-akka.actor.default-dispatcher-13 [00000001|] 127.0.0.1 GET / took 14ms and returned 308 0 bytes
    2022-02-03 21:11:32,648 [INFO] from org.thp.scalligraph.AccessLogFilter in application-akka.actor.default-dispatcher-4 [00000002|] 127.0.0.1 GET /index.html took 38ms and returned 200 1191 bytes
    (.....)
    2022-02-03 21:25:26,943 [INFO] from org.thp.thehive.services.DataIntegrityCheckOps in pool-9-thread-1 [|0956f1f9] Found duplicate entities:
  • Data(xxxxxxx,None)
  • Data(xxxxxxx,None)
    2022-02-03 21:25:28,008 [INFO] from org.thp.thehive.services.DataIntegrityCheckOps in pool-9-thread-1 [|3eefbb8a] Found duplicate entities:
  • Data(xxxxxx,None)
  • Data(xxxxxx,None)
    2022-02-03 21:25:28,505 [INFO] from org.thp.thehive.services.DataIntegrityCheckOps in pool-9-thread-1 [|04b74a3b] Found duplicate entities:

Steps to Reproduce

  1. Configure Elasticsearch index
  2. Force reindex with parameter 'db.janusgraph.forceDropAndRebuildIndex: true'
  3. The observables that are considered duplicated disappear from the alert/case:
    Screen Shot 2022-02-03 at 22 35 57
@jpferrero jpferrero added bug TheHive4 TheHive4 related issues labels Feb 3, 2022
@To-om To-om added this to the 4.1.18 milestone Feb 4, 2022
@To-om
Copy link
Contributor

To-om commented Feb 7, 2022

fixed by #2334

@To-om To-om closed this as completed Feb 7, 2022
@packetvitality packetvitality mentioned this issue Feb 7, 2022
Closed
@chberti chberti mentioned this issue Mar 17, 2022
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug TheHive4 TheHive4 related issues
Projects
None yet
Milestone
4.1.18
Development

No branches or pull requests
2 participants
@jpferrero @To-om