Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Can't start after upgrade thehive4 (4.1.16-1) over (4.0.0-1) [Bug] #2308

Closed
huesitosloco opened this issue Dec 30, 2021 · 5 comments
Closed

Can't start after upgrade thehive4 (4.1.16-1) over (4.0.0-1) [Bug] #2308

huesitosloco opened this issue Dec 30, 2021 · 5 comments
Labels
bug TheHive4 TheHive4 related issues
Milestone
4.1.17

Comments

@huesitosloco
Copy link

Request Type

Can't start The Hive after an upgrade [ (4.1.16-1) over (4.0.0-1)],

Work Environment

Question Answer
OS version (server) Ubuntu 18.04.4 LTS
Virtualized Env. True
Dedicated RAM 8 GB
vCPU 4
TheHive version / git hash (4.1.16-1) upgrading from (4.0.0-1)
Package Type DEB
Database Cassandra
Index type Lucene
Attachments storage Local

Problem Description

on the logs, I noticed a warning regarding a GLOBAL_OFFLINE variable that can not be overriden by the information found in /etc/thehive/application.conf

Full message:
2021-12-30 23:15:07,136 [WARN] from org.janusgraph.diskstorage.configuration.builder.ReadConfigurationBuilder in application-akka.actor.default-dispatcher-11 [|] Local setting index.search.index-name=scalligraph (Type: GLOBAL_OFFLINE) is overridden by globally managed value (janusgraph). Use the ManagementSystem interface instead of the local configuration to control this setting.

After this message, it continues to load, but encounters another problem:

2021-12-30 23:15:11,849 [TRACE] from org.thp.scalligraph.models.Database in application-akka.actor.default-dispatcher-11 [|mgmt-33c851d7] mgmt.makePropertyKey(colour).dataType(String.class).cardinality(SINGLE).make()
2021-12-30 23:15:11,857 [ERROR] from org.thp.scalligraph.models.Database in application-akka.actor.default-dispatcher-11 [|mgmt-33c851d7] Unable to add property colour
org.thp.scalligraph.InternalError: Property colour exists with incompatible type: SINGLE:class java.lang.String Vs SINGLE:class java.lang.Integer
	at org.thp.scalligraph.janus.JanusDatabase.addProperty(JanusDatabase.scala:409)
	at org.thp.scalligraph.janus.JanusDatabase.$anonfun$addProperties$4(JanusDatabase.scala:367)
	at org.thp.scalligraph.package$RichSeq.$anonfun$toTry$3(package.scala:17)
	at scala.collection.TraversableOnce$folder$1.apply(TraversableOnce.scala:196)
	at scala.collection.TraversableOnce$folder$1.apply(TraversableOnce.scala:194)
	at scala.collection.immutable.HashMap$HashMap1.foreach(HashMap.scala:399)
	at scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:725)
	at scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:725)
	at scala.collection.TraversableOnce.foldLeft(TraversableOnce.scala:199)
	at scala.collection.TraversableOnce.foldLeft$(TraversableOnce.scala:192)
	at scala.collection.AbstractTraversable.foldLeft(Traversable.scala:108)
	at org.thp.scalligraph.package$RichSeq.toTry(package.scala:16)
	at org.thp.scalligraph.janus.JanusDatabase.addProperties(JanusDatabase.scala:358)
	at org.thp.scalligraph.janus.JanusDatabase.$anonfun$createSchema$4(JanusDatabase.scala:310)
	at scala.util.Success.flatMap(Try.scala:251)
	at org.thp.scalligraph.janus.JanusDatabase.$anonfun$createSchema$3(JanusDatabase.scala:309)
	at scala.util.Success.flatMap(Try.scala:251)
	at org.thp.scalligraph.janus.JanusDatabase.$anonfun$createSchema$1(JanusDatabase.scala:308)
	at org.thp.scalligraph.janus.JanusDatabase.$anonfun$managementTransaction$6(JanusDatabase.scala:292)
	at scala.util.Try$.apply(Try.scala:213)
	at org.thp.scalligraph.janus.JanusDatabase.$anonfun$managementTransaction$4(JanusDatabase.scala:292)
	at scala.util.Try$.apply(Try.scala:213)
	at org.thp.scalligraph.utils.Retry.org$thp$scalligraph$utils$Retry$$runTry(Retry.scala:61)
	at org.thp.scalligraph.utils.Retry.withTry(Retry.scala:26)
	at org.thp.scalligraph.janus.JanusDatabase.managementTransaction(JanusDatabase.scala:288)
	at org.thp.scalligraph.janus.JanusDatabase.createSchema(JanusDatabase.scala:305)
	at org.thp.scalligraph.janus.JanusDatabaseProvider.$anonfun$get$3(JanusDatabaseProvider.scala:138)
	at scala.util.Success.$anonfun$map$1(Try.scala:255)
	at scala.util.Success.map(Try.scala:213)
	at scala.concurrent.Future.$anonfun$map$1(Future.scala:292)
	at scala.concurrent.impl.Promise.liftedTree1$1(Promise.scala:33)
	at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:33)
	at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:64)
	at org.thp.scalligraph.ContextPropagatingDispatcher$$anon$1.$anonfun$execute$2(ContextPropagatingDisptacher.scala:57)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
	at org.thp.scalligraph.DiagnosticContext$$anon$2.withContext(ContextPropagatingDisptacher.scala:77)
	at org.thp.scalligraph.ContextPropagatingDispatcher$$anon$1.$anonfun$execute$1(ContextPropagatingDisptacher.scala:57)
	at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:48)
	at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:48)
	at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
	at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
	at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
	at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175)
2021-12-30 23:15:11,861 [ERROR] from org.thp.scalligraph.utils.Retry in application-akka.actor.default-dispatcher-11 [|] uncaught error, not retrying
org.thp.scalligraph.InternalError: Property colour exists with incompatible type: SINGLE:class java.lang.String Vs SINGLE:class java.lang.Integer
	at org.thp.scalligraph.janus.JanusDatabase.addProperty(JanusDatabase.scala:409)
	at org.thp.scalligraph.janus.JanusDatabase.$anonfun$addProperties$4(JanusDatabase.scala:367)
	at org.thp.scalligraph.package$RichSeq.$anonfun$toTry$3(package.scala:17)
	at scala.collection.TraversableOnce$folder$1.apply(TraversableOnce.scala:196)
	at scala.collection.TraversableOnce$folder$1.apply(TraversableOnce.scala:194)
	at scala.collection.immutable.HashMap$HashMap1.foreach(HashMap.scala:399)
	at scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:725)
	at scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:725)
	at scala.collection.TraversableOnce.foldLeft(TraversableOnce.scala:199)
	at scala.collection.TraversableOnce.foldLeft$(TraversableOnce.scala:192)
	at scala.collection.AbstractTraversable.foldLeft(Traversable.scala:108)
	at org.thp.scalligraph.package$RichSeq.toTry(package.scala:16)
	at org.thp.scalligraph.janus.JanusDatabase.addProperties(JanusDatabase.scala:358)
	at org.thp.scalligraph.janus.JanusDatabase.$anonfun$createSchema$4(JanusDatabase.scala:310)
	at scala.util.Success.flatMap(Try.scala:251)
	at org.thp.scalligraph.janus.JanusDatabase.$anonfun$createSchema$3(JanusDatabase.scala:309)
	at scala.util.Success.flatMap(Try.scala:251)
	at org.thp.scalligraph.janus.JanusDatabase.$anonfun$createSchema$1(JanusDatabase.scala:308)
	at org.thp.scalligraph.janus.JanusDatabase.$anonfun$managementTransaction$6(JanusDatabase.scala:292)
	at scala.util.Try$.apply(Try.scala:213)
	at org.thp.scalligraph.janus.JanusDatabase.$anonfun$managementTransaction$4(JanusDatabase.scala:292)
	at scala.util.Try$.apply(Try.scala:213)
	at org.thp.scalligraph.utils.Retry.org$thp$scalligraph$utils$Retry$$runTry(Retry.scala:61)
	at org.thp.scalligraph.utils.Retry.withTry(Retry.scala:26)
	at org.thp.scalligraph.janus.JanusDatabase.managementTransaction(JanusDatabase.scala:288)
	at org.thp.scalligraph.janus.JanusDatabase.createSchema(JanusDatabase.scala:305)
	at org.thp.scalligraph.janus.JanusDatabaseProvider.$anonfun$get$3(JanusDatabaseProvider.scala:138)
	at scala.util.Success.$anonfun$map$1(Try.scala:255)
	at scala.util.Success.map(Try.scala:213)
	at scala.concurrent.Future.$anonfun$map$1(Future.scala:292)
	at scala.concurrent.impl.Promise.liftedTree1$1(Promise.scala:33)
	at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:33)
	at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:64)
	at org.thp.scalligraph.ContextPropagatingDispatcher$$anon$1.$anonfun$execute$2(ContextPropagatingDisptacher.scala:57)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
	at org.thp.scalligraph.DiagnosticContext$$anon$2.withContext(ContextPropagatingDisptacher.scala:77)
	at org.thp.scalligraph.ContextPropagatingDispatcher$$anon$1.$anonfun$execute$1(ContextPropagatingDisptacher.scala:57)
	at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:48)
	at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:48)
	at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
	at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
	at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
	at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175)

Steps to Reproduce

  1. Applied the changes to /etc/thehive/application.conf as writen in https://docs.thehive-project.org/thehive/operations/update/

This is an extract of my config file:

db.janusgraph {
  storage {
    backend: cql
    hostname: ["127.0.0.1"]
    cql {
      cluster-name: thp
      keyspace: thehive
      read-consistency-level: ONE
      write-consistency-level: ONE
    }
  }


 index {
      search {
        backend: lucene
        directory: /opt/thp/thehive/index
      }
    }
  // storage.backend: berkeleyje
  // storage.directory: /path/to/berkeleydb
  // berkeleyje.freeDisk: 200 
}

storage {

  provider: localfs
  localfs.directory: /opt/thp_data/files/thehive
  // provider: hdfs
  // hdfs {
  //   root: "hdfs://localhost:10000" # namenode server hostname
  //   location: "/thehive"           # location inside HDFS
  //   username: thehive              # file owner
  // }
}
  1. Restarted The Hive, everything works properly
  2. Installed the stable verion following this guide https://docs.thehive-project.org/thehive/installation-and-configuration/installation/step-by-step-guide/ , I executed only the steps on the "The Hive" section, since everything else seems to be already in place:
echo 'deb https://deb.thehive-project.org release main' | sudo tee -a /etc/apt/sources.list.d/thehive-project.list
sudo apt-get update
sudo apt-get install thehive4

4.-The hive does not come up again, everytime I restart the service (which appear as "running" as per "service thehive status") I get the same error stated above.

Possible Solutions

I did some reading, and it appears to be related to a variable that is set on the database, and can't be overwriten by the configuration file, since the warning stated the "harcoded" value of scalligraph can't be changed, and the error does refer scalligraph usage. I found a reference to change the GLOBAL_OFFLINE values, but I Tought it was better to ask.
https://docs.janusgraph.org/configs/#global-configuration

Complementary information

application.log.txt
install_application.log.txt

I added the logs from a service restart at application.log.txt, and full logs from the install at install_application.og.txt
@huesitosloco huesitosloco added bug TheHive4 TheHive4 related issues labels Dec 30, 2021
@marpoe
Copy link

marpoe commented Jan 18, 2022

Unfortunately I faced exactely the same issue. Coming from "4.0.2-1".

@To-om To-om added this to the 4.1.17 milestone Jan 18, 2022
@nadouani
Copy link
Contributor

@huesitosloco can you please add this property to your application.conf file

db.janusgraph.forceDropAndRebuildIndex: true

Restart TheHive, it will rebuild the index and index the data. Once done, you can remove the db.janusgraph.forceDropAndRebuildIndex config or set it to false

@mihai1978
Copy link

I have the same issue and the workarround with db.janusgraph.forceDropAndRebuildIndex is not working for me.

@nadouani
Copy link
Contributor

Hello @huesitosloco @mihai1978 we've investigated the issue with @marpoe and there is in fact a bug that needs to be fixed. Adding this issue to 4.1.17

@mihai1978
Copy link

I see, thank you for your answer.

To-om added a commit that referenced this issue Jan 22, 2022
@To-om
#2308 Fix schema creation when property type has changed
bc7364d
@To-om To-om closed this as completed Jan 22, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug TheHive4 TheHive4 related issues
Projects
None yet
Milestone
4.1.17
Development

No branches or pull requests
5 participants
@nadouani @To-om @marpoe @huesitosloco @mihai1978