Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug]- Migration from Hive 3.4.4 to Hive 4.1.17 not working #2331

Closed
viverma5 opened this issue Feb 2, 2022 · 4 comments
Closed

[Bug]- Migration from Hive 3.4.4 to Hive 4.1.17 not working #2331

viverma5 opened this issue Feb 2, 2022 · 4 comments
Labels
bug TheHive4 TheHive4 related issues
Milestone
4.1.18

Comments

@viverma5
Copy link

viverma5 commented Feb 2, 2022 •
edited
Loading

Request Type

Bug

Work Environment

Question Answer
OS version (server) RedHat
OS version (client) 8.3.
Virtualized Env. True
Dedicated RAM 16 GB
vCPU 8
TheHive version / git hash 4.1.17
Package Type RPM,
Database Cassandra
Index type Lucene
Attachments storage Local
Browser type & version If applicable

Problem Description

Describe the problem/bug as clearly as possible.

Steps to Reproduce

  1. step 1
    Install The Hive 4-4.1.17 version on RHEL machine. Configure database as cassandra, index as lucene and file system is local.
    Start The Hive service.All goes well
  2. step 2
    Now, launch migration to target server with ES 6.8.21 and Hive version 3.4.4. Migration starts but it detects nothing to change and finishes with no change.

`[root@azucl20007 thehive]# /opt/thehive/bin/migrate --main-organisation StSOC --es-uri http://10.28.100.37:9200 --es-index the_hive --exclude-alert-types misp -o /etc/thehive/application.conf
[info] 1,666,167/1,769,472KiB(95%) GC:8 (cpu:5% 169ms)
[initialisation]
[info] Found ElasticSearch 6.8.21
[info] Found Index the_hive_15
[info] Found index with multiple types
[info] Initialising cluster
[info] Member is Up: akka://TheHiveMigration@127.0.0.1:35141
[info] 1,430,538/1,769,472KiB(81%) GC:9 (cpu:3% 373ms)
[initialisation]
[info] Full-text index is available (lucene:/opt/thp/thehive/index) single node
[info] Creating database schema
[info] The field data is indexed
[info] 1,458,874/1,769,472KiB(83%) GC:1 (cpu:0% 22ms)
[Finalisation] Organisation:1/1(240,525µs)
[info] Running check on Tag ...
[info] Check on Tag: no change needed
[info] Running check on Log ...
[info] Check on Log: no change needed
[info] Running check on Alert ...
[info] Check on Alert: no change needed
[info] Running check on Organisation ...
[info] Found duplicate entities:

  • Organisation(admin,organisation for administration)
  • Organisation(admin,organisation for administration)
    [info] Check on Organisation: duplicate:1
    [info] Running check on Data ...
    [info] Check on Data: no change needed
    [info] Running check on CaseTemplate ...
    [info] Check on CaseTemplate: no change needed
    [info] Running check on Profile ...
    [info] Found duplicate entities:
  • Profile(org-admin,Set(manageShare, manageAnalyse, manageTask, manageCaseTemplate, manageCase, manageUser, manageProcedure, managePage, manageObservable, manageTag, manageConfig, manageAlert, accessTheHiveFS, manageAction))
  • Profile(org-admin,Set(manageShare, manageAnalyse, manageTask, manageCaseTemplate, manageCase, manageUser, manageProcedure, managePage, manageObservable, manageTag, manageConfig, manageAlert, accessTheHiveFS, manageAction))
    [info] Found duplicate entities:
  • Profile(read-only,Set())
  • Profile(read-only,Set())
    [info] Found duplicate entities:
  • Profile(admin,Set(manageAnalyzerTemplate, manageUser, manageOrganisation, manageCustomField, managePlatform, manageConfig, manageTaxonomy, managePattern, manageObservableTemplate, manageProfile))
  • Profile(admin,Set(manageAnalyzerTemplate, manageUser, manageOrganisation, manageCustomField, managePlatform, manageConfig, manageTaxonomy, managePattern, manageObservableTemplate, manageProfile))
    [info] Found duplicate entities:
  • Profile(analyst,Set(manageShare, manageAnalyse, manageTask, manageCase, manageProcedure, managePage, manageObservable, manageAlert, accessTheHiveFS, manageAction))
  • Profile(analyst,Set(manageShare, manageAnalyse, manageTask, manageCase, manageProcedure, managePage, manageObservable, manageAlert, accessTheHiveFS, manageAction))
    [info] Check on Profile: duplicate:4
    [info] Running check on Case ...
    [info] Check on Case: no change needed
    [info] Running check on ImpactStatus ...
    [info] Found duplicate entities:
  • ImpactStatus(NoImpact)
  • ImpactStatus(NoImpact)
    [info] Found duplicate entities:
  • ImpactStatus(NotApplicable)
  • ImpactStatus(NotApplicable)
    [info] Found duplicate entities:
  • ImpactStatus(WithImpact)
  • ImpactStatus(WithImpact)
    [info] Check on ImpactStatus: duplicate:3
    [info] Running check on ResolutionStatus ...
    [info] Found duplicate entities:
  • ResolutionStatus(TruePositive)
  • ResolutionStatus(TruePositive)
    [info] Found duplicate entities:
  • ResolutionStatus(Indeterminate)
  • ResolutionStatus(Indeterminate)
    [info] Found duplicate entities:
  • ResolutionStatus(Other)
  • ResolutionStatus(Other)
    [info] Found duplicate entities:
  • ResolutionStatus(Duplicated)
  • ResolutionStatus(Duplicated)
    [info] Found duplicate entities:
  • ResolutionStatus(FalsePositive)
  • ResolutionStatus(FalsePositive)
    [info] Check on ResolutionStatus: duplicate:5
    [info] Running check on User ...
    [info] Found duplicate entities:
  • User(system@thehive.local,TheHive system user,false)
  • User(system@thehive.local,TheHive system user,false)
    [info] Found duplicate entities:
  • User(admin@thehive.local,Default admin user,false)
  • User(admin@thehive.local,Default admin user,false)
    [info] Check on User: no change needed
    [info] Running check on Task ...
    [info] Check on Task: no change needed
    [info] Running check on CustomField ...
    [info] Check on CustomField: no change needed
    [info] Running check on ObservableType ...
    [info] Found duplicate entities:
  • ObservableType(mail,false)
  • ObservableType(mail,false)
    [info] Found duplicate entities:
  • ObservableType(autonomous-system,false)
  • ObservableType(autonomous-system,false)
    [info] Found duplicate entities:
  • ObservableType(mail-subject,false)
  • ObservableType(mail-subject,false)
    [info] Found duplicate entities:
  • ObservableType(user-agent,false)
  • ObservableType(user-agent,false)
    [info] Found duplicate entities:
  • ObservableType(hash,false)
  • ObservableType(hash,false)
    [info] Found duplicate entities:
  • ObservableType(url,false)
  • ObservableType(url,false)
    [info] Found duplicate entities:
  • ObservableType(registry,false)
  • ObservableType(registry,false)
    [info] Found duplicate entities:
  • ObservableType(uri_path,false)
  • ObservableType(uri_path,false)
    [info] Found duplicate entities:
  • ObservableType(ip,false)
  • ObservableType(ip,false)
    [info] Found duplicate entities:
  • ObservableType(hostname,false)
  • ObservableType(hostname,false)
    [info] Found duplicate entities:
  • ObservableType(file,true)
  • ObservableType(file,true)
    [info] Found duplicate entities:
  • ObservableType(domain,false)
  • ObservableType(domain,false)
    [info] Found duplicate entities:
  • ObservableType(regexp,false)
  • ObservableType(regexp,false)
    [info] Found duplicate entities:
  • ObservableType(other,false)
  • ObservableType(other,false)
    [info] Found duplicate entities:
  • ObservableType(fqdn,false)
  • ObservableType(fqdn,false)
    [info] Found duplicate entities:
  • ObservableType(filename,false)
  • ObservableType(filename,false)
    [info] Check on ObservableType: duplicate:16
    [info] Running check on Observable ...
    [info] Check on Observable: no change needed
    [info] Migration finished
    [info] Member is Removed: akka://TheHiveMigration@127.0.0.1:35141 after Exiting
    `
  1. step 3...
    Migration ends with no change.
  2. Step4
    Install The Hive 4-4.0.4 version and attempt migration. This time it works perfectly.
@viverma5 viverma5 added bug TheHive4 TheHive4 related issues labels Feb 2, 2022
@To-om
Copy link
Contributor

To-om commented Feb 4, 2022

The migration tool found an index with multiple types whereas the index uses a single type. The detection doesn't work correctly (it need a fix). You can force by adding the parameter --es-single-type true.

@To-om To-om added this to the 4.1.18 milestone Feb 4, 2022
@viverma5
Copy link
Author

viverma5 commented Feb 4, 2022 •
edited
Loading

Hello @To-om

I have tested it by adding --es-single-type true in the migration script.
This time migration started but I see warning, where one of the customField named "userGroup" in Hive3, is not able to be created in Hive 4 and we get repeated warnings like below-

[warn] Unable to set custom field userGroup="APAC_User" to case #4379: org.thp.scalligraph.NotFoundError: Custom field userGroup not found

This issue is only for this specific customField userGroup. Not sure, what could be the problem.

Besides, once migration was over, I saw that we got 5 ORGANIZATIONS created [2 for admin and 3 for the one which we used as MAIN organization] instead of 2 . Even after re-indexing status is same.

@viverma5
Copy link
Author

viverma5 commented Feb 4, 2022 •
edited
Loading

Hello @To-om ,

After looking into the logs, I realised error initially while creating customFields in the migration script.

``[error] CustomField creation failure: com.fasterxml.jackson.core.io.JsonEOFException: Unexpected end-of-input: expected close marker for Array

I see that CustField is truncated at 7691 characters. I tried to change the data of the CF in Hive 3 to see if it makes any difference, but everytime it fails at same character position 7691
Do we have any kind of restriction?
This issue does not happen if I upgrade from 3.4.4 - 4.0.4-X.X.X

@To-om To-om mentioned this issue Feb 6, 2022
Closed
6 tasks
To-om added a commit that referenced this issue Feb 6, 2022
@To-om
#2331 Use index version to determine if it is single type
f9fab93
@To-om
Copy link
Contributor

To-om commented Feb 6, 2022

Single type detection and custom field truncation problems are fixed in 4.1.18

@To-om To-om closed this as completed Feb 6, 2022
@packetvitality packetvitality mentioned this issue Feb 7, 2022
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug TheHive4 TheHive4 related issues
Projects
None yet
Milestone
4.1.18
Development

No branches or pull requests
2 participants
@To-om @viverma5