[Bug] Unable to migrate to TH 4.0.5

[mamoedo]

Request Type

Bug

Work Environment

Question	Answer
OS version (server)	Debian
TheHive version / git hash	4.0.5
Package Type	Binary

Problem Description

Cannot migrate from TH3 to TH4.0.5. New exceptions are shown.

Steps to Reproduce

cd /opt/thehive/bin && cp /opt/thehive/conf/logback-migration.xml /etc/thehive/ && ./migrate --output /etc/thehive/application.conf --main-organisation Main-Org --es-index the_hive --es-uri http://192.167.0.20:19200 &> /tmp/migration.txt &

Possible Solutions

I don't know the possible solutions. It may be related to this issues opened in previous versions #1682

Complementary information

Some cases are migrated in between this 2 following logs:

[error] uncaught error, not retrying
java.lang.NullPointerException: null
[error] Exception raised, rollback (null)
[error] Action creation failure: java.lang.NullPointerException
[error] uncaught error, not retrying
java.lang.NullPointerException: null
[error] Exception raised, rollback (null)
[error] Action creation failure: java.lang.NullPointerException

[error] Exception raised, rollback (null)
[error] Job creation failure: java.lang.NullPointerException
[error] uncaught error, not retrying
java.lang.NullPointerException: null
        at org.janusgraph.graphdb.types.VertexLabelVertex.isPartitioned(VertexLabelVertex.java:41)
        at org.janusgraph.graphdb.database.idassigner.VertexIDAssigner.getVertexIDType(VertexIDAssigner.java:363)
        at org.janusgraph.graphdb.database.idassigner.VertexIDAssigner.assignID(VertexIDAssigner.java:153)
        at org.janusgraph.graphdb.database.StandardJanusGraph.assignID(StandardJanusGraph.java:474)
        at org.janusgraph.graphdb.transaction.StandardJanusGraphTx.addVertex(StandardJanusGraphTx.java:514)
        at org.janusgraph.graphdb.transaction.StandardJanusGraphTx.addVertex(StandardJanusGraphTx.java:532)
        at org.janusgraph.graphdb.transaction.StandardJanusGraphTx.addVertex(StandardJanusGraphTx.java:528)
        at org.janusgraph.graphdb.transaction.StandardJanusGraphTx.addVertex(StandardJanusGraphTx.java:98)
        at org.thp.thehive.connector.cortex.models.Job$$anon$7.create(Job.scala:22)
        at org.thp.thehive.connector.cortex.models.Job$$anon$7.create(Job.scala:22)

After this log, migration is stopped:

[error] uncaught error, not retrying
java.lang.IllegalStateException: Cannot access element because its enclosing transaction is closed and unbound
        at org.janusgraph.graphdb.transaction.StandardJanusGraphTx.getNextTx(StandardJanusGraphTx.java:305)
        at org.janusgraph.graphdb.vertices.AbstractVertex.it(AbstractVertex.java:53)
        at org.janusgraph.graphdb.vertices.AbstractVertex.it(AbstractVertex.java:37)
        at org.janusgraph.graphdb.internal.AbstractElement.isRemoved(AbstractElement.java:141)
        at java.util.ArrayList.removeIf(ArrayList.java:1413)
        at org.janusgraph.graphdb.transaction.StandardJanusGraphTx.getVertices(StandardJanusGraphTx.java:426)
        at org.janusgraph.graphdb.tinkerpop.JanusGraphBlueprintsTransaction.vertices(JanusGraphBlueprintsTransaction.java:138)
        at org.apache.tinkerpop.gremlin.process.traversal.step.map.GraphStep.lambda$new$0(GraphStep.java:69)
        at org.apache.tinkerpop.gremlin.process.traversal.step.map.GraphStep.processNextStart(GraphStep.java:157)
        at org.apache.tinkerpop.gremlin.process.traversal.step.util.AbstractStep.hasNext(AbstractStep.java:143)
[error] Exception raised, rollback (Cannot access element because its enclosing transaction is closed and unbound)
[error] Alert/Observable creation failure: java.lang.IllegalStateException: Cannot access element because its enclosing transaction is closed and unbound
[error] uncaught error, not retrying
org.janusgraph.core.JanusGraphException: ID block allocation on partition(13)-namespace(0) failed with an exception in 358.0 ms
        at org.janusgraph.graphdb.database.idassigner.StandardIDPool.waitForIDBlockGetter(StandardIDPool.java:143)
        at org.janusgraph.graphdb.database.idassigner.StandardIDPool.nextBlock(StandardIDPool.java:182)
        at org.janusgraph.graphdb.database.idassigner.StandardIDPool.nextID(StandardIDPool.java:208)
        at org.janusgraph.graphdb.database.idassigner.VertexIDAssigner.assignID(VertexIDAssigner.java:333)
        at org.janusgraph.graphdb.database.idassigner.VertexIDAssigner.assignID(VertexIDAssigner.java:182)
        at org.janusgraph.graphdb.database.idassigner.VertexIDAssigner.assignID(VertexIDAssigner.java:153)
        at org.janusgraph.graphdb.database.StandardJanusGraph.assignID(StandardJanusGraph.java:474)
        at org.janusgraph.graphdb.transaction.StandardJanusGraphTx.addVertex(StandardJanusGraphTx.java:514)
        at org.janusgraph.graphdb.transaction.StandardJanusGraphTx.addVertex(StandardJanusGraphTx.java:532)
        at org.janusgraph.graphdb.transaction.StandardJanusGraphTx.addVertex(StandardJanusGraphTx.java:528)
Caused by: java.util.concurrent.ExecutionException: java.lang.IllegalStateException: Store cannot be found: janusgraph_ids
        at java.util.concurrent.FutureTask.report(FutureTask.java:122)
        at java.util.concurrent.FutureTask.get(FutureTask.java:206)
        at org.janusgraph.graphdb.database.idassigner.StandardIDPool.waitForIDBlockGetter(StandardIDPool.java:139)
        at org.janusgraph.graphdb.database.idassigner.StandardIDPool.nextBlock(StandardIDPool.java:182)
        at org.janusgraph.graphdb.database.idassigner.StandardIDPool.nextID(StandardIDPool.java:208)
        at org.janusgraph.graphdb.database.idassigner.VertexIDAssigner.assignID(VertexIDAssigner.java:333)
        at org.janusgraph.graphdb.database.idassigner.VertexIDAssigner.assignID(VertexIDAssigner.java:182)
        at org.janusgraph.graphdb.database.idassigner.VertexIDAssigner.assignID(VertexIDAssigner.java:153)
        at org.janusgraph.graphdb.database.StandardJanusGraph.assignID(StandardJanusGraph.java:474)
        at org.janusgraph.graphdb.transaction.StandardJanusGraphTx.addVertex(StandardJanusGraphTx.java:514)
Caused by: java.lang.IllegalStateException: Store cannot be found: janusgraph_ids
        at org.janusgraph.diskstorage.cql.CQLStoreManager.lambda$null$23(CQLStoreManager.java:504)
        at io.vavr.control.Option.getOrElseThrow(Option.java:319)
        at org.janusgraph.diskstorage.cql.CQLStoreManager.lambda$mutateManyUnlogged$30(CQLStoreManager.java:504)
        at io.vavr.collection.Iterator$34.hasNext(Iterator.java:1503)
        at io.vavr.collection.Traversable.foldLeft(Traversable.java:471)
        at io.vavr.concurrent.Future.sequence(Future.java:549)
        at org.janusgraph.diskstorage.cql.CQLStoreManager.mutateManyUnlogged(CQLStoreManager.java:499)
        at org.janusgraph.diskstorage.cql.CQLStoreManager.mutateMany(CQLStoreManager.java:457)
        at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.mutate(CQLKeyColumnValueStore.java:383)
        at org.janusgraph.diskstorage.idmanagement.ConsistentKeyIDAuthority.lambda$getIDBlock$3(ConsistentKeyIDAuthority.java:317)
[error] Exception raised, rollback (ID block allocation on partition(13)-namespace(0) failed with an exception in 358.0 ms)
[error] Alert/Observable creation failure: org.janusgraph.core.JanusGraphException: ID block allocation on partition(13)-namespace(0) failed with an exception in 358.0 ms

[mamoedo]

@nadouani @To-om will this issue be checked out before TH 4.1.0 release?

[nadouani]

Hello @nadouani we will take a look

[To-om]

Hi,
The schema of your database hasn't been correctly set up. Did you start migration with an existing TH4 DB ?

[mamoedo]

@To-om I always start the migration without a database and I let the migration tool create it. It usually works if only 1 index is migrated, but it fails if more than 1 index are migrated (I think #1682 issue is key to this problem)