[Enhancement] When observable data is too big, use hash

[To-om]

Request Type

Enhancement

Feature Description

Fields which contains more that 32k of data cannot be indexed, and breaks index engine. The aim of this issue is to store huge observable data in a dedicated unindexed field (named fullData) and store the hash in indexed field (instead of the real value). This change must be implemented in the observable creation and in the observable search (in properties).

Existing data must also be processed but the schema evolution cannot be used because the index may be broken. The processing can use the immense term processing of Scalligraph TheHive-Project/ScalliGraph#17

In order to fix existing data, the following configuration must be set:

db.janusgraph {
  immenseTermProcessing: {
    data: observableHashToIndex
  }
}

This make the next startup slower because the whole database must be crawled.
IMPORTANT This configuration should be present only for one startup to fix the data. It should be removed as soon as the process if finished.