[Bug] Migration of usernames from ES to Cassandra

[crackytsi]

Bug

Work Environment

Question	Answer
OS version (server)	Debian
OS version (client)	10
TheHive version / git hash	4 RC3
Package Type	DEB

Problem Description

I have users in the format:
firstname.lastname@example.com
During import with migration tool they get renamed in TheHive4 to:
firstname.lastname.example.com@test.local

Additional this later leads to issues in migration:

[error] o.t.s.u.Retry - uncaught error, not retrying
org.thp.scalligraph.NotFoundError: User firstname.lastname@example.com not found
        at org.thp.scalligraph.services.VertexSrv.$anonfun$getOrFail$1(VertexSrv.scala:33)
        at scala.Option.fold(Option.scala:251)
        at org.thp.scalligraph.services.VertexSrv.getOrFail(VertexSrv.scala:33)
        at org.thp.thehive.migration.th4.Output.$anonfun$createCaseTask$4(Output.scala:337)
        at scala.Option.map(Option.scala:230)
        at org.thp.thehive.migration.th4.Output.$anonfun$createCaseTask$2(Output.scala:337)
        at org.thp.thehive.migration.th4.Output.$anonfun$authTransaction$1(Output.scala:161)
        at org.thp.scalligraph.janus.JanusDatabase.$anonfun$tryTransaction$10(JanusDatabase.scala:160)
        at scala.util.Try$.apply(Try.scala:213)
        at org.thp.scalligraph.janus.JanusDatabase.$anonfun$tryTransaction$8(JanusDatabase.scala:160)
[error] o.t.s.m.Database - Exception raised, rollback (User firstname.lastname@example.com not found)
[error] o.t.t.m.Migrate - Case/Task creation failure: org.thp.scalligraph.NotFoundError: User firstname.lastname@example.com not found
[error] o.t.t.m.Migrate - Case/Task/Log creation failure: org.thp.scalligraph.NotFoundError: Id AXILd5FkfZoD64Z1Yspu not found

Is there an option to preserve there login and do not append the default domain?

Possible Solutions

If username contains one "@" preserve the username.

[crackytsi]

I forgot to mention that "-" also gets translated to a "."

[crackytsi]

Strange is also that a rollback is performed, so task and observable is not created, but case stays.
Message is: User firstna.lastname@example.com not found, use system user, but the user is used (for the case, rest is not there).

[To-om]

I've fixed user translation. I hope I cover all cases.
I don't understand your last message.

[crackytsi]

@To-om
It is possible and valid that an email contains a "-" character (minus), this was currently also converted to a dot (.). Do you cover this case?

When I perform the migration and it fails because of missing user, it says: "Exception raised, rollback". So i though that tasks that were created are deleted again. But the case that was created stays in that as it is. That means after the migration, there are a lot of cases, that have no observables and no tasks, but the case was migrated from ES.

[To-om]

@crackytsi The dashes are correctly handled, I'll add some tests.
I've also add normation on users in case, task and caseTemplate. You should not see "user XX not found, use system user" anymore. Nevertheless, this is not an error but a warning. Task creation doesn't fail for that reason. Have you the complete error message ?

[crackytsi]

@To-om : I compiled based on your latest dev-commit TH4, but user migration still does not work as expected:

[^[[37minfo^[[0m] o.t.t.c.c.m.SchemaUpdater - Cortex database schema is outdated (0). Upgrading to version 1 ...
[^[[37minfo^[[0m] o.r.Reflections - Reflections took 65 ms to scan 1 urls, producing 40 keys and 214 values
[^[[37minfo^[[0m] o.t.t.c.c.m.CortexSchema - Loading model ActionContext
[^[[37minfo^[[0m] o.t.t.c.c.m.CortexSchema - Loading model AnalyzerTemplate
[^[[37minfo^[[0m] o.t.t.c.c.m.CortexSchema - Loading model Job
[^[[37minfo^[[0m] o.t.t.c.c.m.CortexSchema - Loading model Action
[^[[37minfo^[[0m] o.t.t.c.c.m.CortexSchema - Loading model ReportObservable
[^[[37minfo^[[0m] o.t.t.c.c.m.CortexSchema - Loading model ObservableJob
[^[[37minfo^[[0m] o.t.s.m.Database - Creating database schema
[^[[37minfo^[[0m] o.t.s.m.Database - Model Binary already exists, ignore it
[^[[37minfo^[[0m] o.t.s.m.Database - Model NextChunk already exists, ignore it
[^[[37minfo^[[0m] o.t.s.m.Database - Property endDate SINGLE:class java.util.Date already exists, ignore it
[^[[37minfo^[[0m] o.t.s.m.Database - Property data SINGLE:class java.lang.String already exists, ignore it
[^[[37minfo^[[0m] o.t.s.m.Database - Property attachmentId SINGLE:class java.lang.String already exists, ignore it
[^[[37minfo^[[0m] o.t.s.m.Database - Property folder SINGLE:class java.lang.String already exists, ignore it
[^[[37minfo^[[0m] o.t.s.m.Database - Property status SINGLE:class java.lang.String already exists, ignore it
[^[[37minfo^[[0m] o.t.s.m.Database - Property content SINGLE:class java.lang.String already exists, ignore it
[^[[37minfo^[[0m] o.t.s.m.Database - Property startDate SINGLE:class java.util.Date already exists, ignore it
[^[[33mwarn^[[0m] o.t.t.m.t.Output - User system not found, use system user
[^[[37minfo^[[0m] o.t.t.m.t.Output - Create organisation myOrg5
[^[[37minfo^[[0m] o.t.t.m.t.Output - Create user firstname.lastnameuser1.test.company.org
[^[[33mwarn^[[0m] o.t.t.m.t.Output - User firstname.lastnameuser1@test-company.org not found, use system user
[^[[37minfo^[[0m] o.t.t.m.t.Output - Create user firstname2.lastnameuser2.test.company.org
[^[[33mwarn^[[0m] o.t.t.m.t.Output - User firstname.lastnameuser1@test-company.org not found, use system user
[^[[37minfo^[[0m] o.t.t.m.t.Output - Create user firstname3.lastnameuser3.test.company.org
[^[[33mwarn^[[0m] o.t.t.m.t.Output - User firstname.lastnameuser1@test-company.org not found, use system user
[^[[37minfo^[[0m] o.t.t.m.t.Output - Create user firstname4.lastnameuser4.test.company.org
[^[[33mwarn^[[0m] o.t.t.m.t.Output - User firstname.lastnameuser1@test-company.org not found, use system user
[^[[37minfo^[[0m] o.t.t.m.t.Output - Create user firstname6.lastnameuser6.test.company.org
[^[[31merror^[[0m] o.t.t.m.t.Input - org.thp.thehive.migration.dto.InputUser read failure:List(
 - : JsonValidationError(List('roles' is undefined on object: {"name":"error2","updatedAt":1518167098512,"status":"Locked","preferences":"{}","createdAt":1497380352646,"createdBy":"firstname.lastnameuser1@test-company.org","updatedBy":"firstname.lastnameuser1@test-company.org","password":"㴛✓눾䂫䇈接е酳㋧鋔,245efe6875641fb6c99ed808d52291190708638b0abea1db175f9ae5c71bbe3a","_type":"user","_routing":"login5","_parent":null,"_id":"login5","_version":1}),WrappedArray()))

[crackytsi]

I used the new version on a complete new keyspace of Cassandra.
I also get a lot of errors like this, not sure if this is linked to the user issue or a different migration issue.

[error] o.t.t.m.Migrate - InputLog creation failure
org.thp.scalligraph.NotFoundError: Id AXHgLR9pfZoD64Z1YrYA not found
        at org.thp.thehive.migration.MigrationOps$IdMappingOps.$anonfun$fromInput$2(MigrationOps.scala:22)
        at scala.Option.fold(Option.scala:251)
        at org.thp.thehive.migration.MigrationOps$IdMappingOps.fromInput(MigrationOps.scala:22)
        at org.thp.thehive.migration.MigrationOps.$anonfun$migrateWithParent$1(MigrationOps.scala:44)
        at akka.stream.impl.fusing.Map$$anon$1.onPush(Ops.scala:54)
        at akka.stream.impl.fusing.GraphInterpreter.processPush(GraphInterpreter.scala:541)
        at akka.stream.impl.fusing.GraphInterpreter.execute(GraphInterpreter.scala:423)
        at akka.stream.impl.fusing.GraphInterpreterShell.runBatch(ActorGraphInterpreter.scala:624)
        at akka.stream.impl.fusing.GraphInterpreterShell$AsyncInput.execute(ActorGraphInterpreter.scala:501)
        at akka.stream.impl.fusing.GraphInterpreterShell.processEvent(ActorGraphInterpreter.scala:599)

[To-om]

Importing user without role doesn't fail any more.
The error "Id not found" appear when a element (most probably an audit) refers a missing data (because it has been deleted).