[Remove discover] Implement embeddable dashboard on Threat hunting module

[asteriscos]

Description

We have to implement the embeddable dashboard on Threat hunting -> dashboard tab and deprecate any use of kibana-integrations components.

Warning

The embeddable panel id must be unique including general and agents visualizations. Otherwise, the visualizations will not refresh when we pin an agent, because they are cached by id

Current Threat Hunting screens

Tasks

• Implement the embeddable dashboard on Threat hunting -> dashboard tab
  • Migrate visualizations to embeddable panels
  • Add new searchbar
  • Migrate bottom table to new discover
  • Add SampleDataMessage
  • Add conditionality to show the dashboard if there are results.
  • Add loadings
  • Add message that there are no results
  • Change index pattern selection mechanism (Dependence: [Remove discover] Change index pattern selector mechanism #6434)
  • Change pinned agent recognition mechanism (Dependence: Standardize filter management mechanisms #6499)
  • Once the two previous points have been applied, corroborate filter behavior with respect to hide alerts, allow agents and filter order.
• Check the following:
  • Each visualization, if applicable, must have interaction so that it adds the corresponding filter(s) upon clicking.
  • The visualizations have to be updated according to the filters applied in the searchbar.
  • The visualizations have to be updated when a search is performed in the searchbar.
  • If there are no results, the corresponding message must appear that there are no results and the visualizations should not be rendered.
  • If there is SampleData, the corresponding SampleData message must appear.
  • If an agent is pinned, the views must be updated and changed to the agent view, if applicable.
  • Check filter behavior with respect to hide alerts, allow agents and filter order.

Source task

• [Remove discover] Refactor modules dashboards with new embeddable component #6477

[jbiset]

Update 06/03/2024

Progress was made by migrating the Threat Hunting overview definitions. The visualizations are already made with embeddables. Likewise, the visualizations are integrated with the searchbar bidirectionally.
To continue:

• It remains to adjust the aesthetics of the KPIs
• Make color corrections.
• Add bottom table
• Handle changing visualization panels when an agent is pinned.

Evidence

[jbiset]

Update 07/03/2024

Added the withPinnedAgent HOC to add the pinnedAgent prop that allows DashboardThreatHunting to know which panels to render based on whether or not an agent is pinned.
This pinnedAgent prop is then used as a parameter of the getDashboardPanels method which will return the corresponding panels depending on whether or not there is a pinned agent.
It is important to note that changing the structure of normal panels to the panels of a pinned agent should not share the ids of their panels. Below is an example of each one:

Note

The examples below only have the definitions changed from the first two visualizations. So does the screenshot shown below.

Structure of normal panels

  const panels = {
    '1': {
      gridData: {
        w: 28,
        h: 13,
        x: 0,
        y: 0,
        i: '1',
      },
      type: 'visualization',
      explicitInput: {
        id: '1',
        savedVis: getVisStateTop10AlertLevelEvolution(indexPatternId),
      },
    },
    '2': {
      gridData: {
        w: 20,
        h: 13,
        x: 28,
        y: 0,
        i: '2',
      },
      type: 'visualization',
      explicitInput: {
        id: '2',
        savedVis: getVisStateTop10MITREATTACKS(indexPatternId),
      },
    },
    '3': {
      gridData: {
        w: 15,
        h: 12,
        x: 0,
        y: 13,
        i: '3',
      },
      type: 'visualization',
      explicitInput: {
        id: '3',
        savedVis: getVisStateTop5Agents(indexPatternId),
      },
    },
    '4': {
      gridData: {
        w: 33,
        h: 12,
        x: 15,
        y: 13,
        i: '4',
      },
      type: 'visualization',
      explicitInput: {
        id: '4',
        savedVis: getVisStateAlertEvolutionTop5Agents(indexPatternId),
      },
    },
  };

Structure of panels with pinned agent

  const pinnedAgentPanels = {
    '5': {
      gridData: {
        w: 24,
        h: 13,
        x: 0,
        y: 0,
        i: '5',
      },
      type: 'visualization',
      explicitInput: {
        id: '5',
        savedVis:
          getVisStatePinnedAgentTop10AlertGroupsEvolution(indexPatternId),
      },
    },
    '6': {
      gridData: {
        w: 24,
        h: 13,
        x: 24,
        y: 0,
        i: '6',
      },
      type: 'visualization',
      explicitInput: {
        id: '6',
        savedVis: getVisStateTop5Alerts(indexPatternId),
      },
    },
    '3': {
      gridData: {
        w: 15,
        h: 12,
        x: 0,
        y: 13,
        i: '3',
      },
      type: 'visualization',
      explicitInput: {
        id: '3',
        savedVis: getVisStateTop5Agents(indexPatternId),
      },
    },
    '4': {
      gridData: {
        w: 33,
        h: 12,
        x: 15,
        y: 13,
        i: '4',
      },
      type: 'visualization',
      explicitInput: {
        id: '4',
        savedVis: getVisStateAlertEvolutionTop5Agents(indexPatternId),
      },
    },
  };

Capture showing the change of the dashboard depending on whether or not there is an agent pinned

Evidence_Change_Pinned_Agent.webm

[jbiset]

Update 08/03/2024

The aesthetics of the KPIs were adjusted using embeddable visualizations.
It was decided to use embeddable visualizations for the KPIs to unlink them from the previous version and maintain a standard version to display the KPIs.
Finished migrating visualization definitions when an agent is pinned
Visualizations are changed from pie to donut
Interaction with KPIs is analyzed so that it is filtered

It is analyzed that the queries for the new KPIs are correct according to the previous queries.

Total metric

Level 12 or Above metric

Authentication failure metric

Authentication success metric

To continue:

• Add the bottom table
• Add interaction with KPIs
• Add SampleDataMessage
• Add Loadings
• Add messages when there are no results

Current screen without agent pinned

Current screen with pinned agent

[jbiset]

Update 13/03/2024

• The interaction was added to the KPIs so that they add the corresponding filter when clicking on them.
• Links were added to the lower table on the columns agent.id, rule.mitre.id and rule.id.
• Added updating of visible columns when an agent is pinned.

Evidence

Evidence_Changes_2024-03-13.webm

[jbiset]

Update 18/04/2024

• Latest changes from the 4.9.0 branch are merge into the Issue branch and conflicts are resolved
• Changed index pattern selection mechanism with new data source
• Changed pinned agent recognition mechanism with the getPinnedAgentFilter method of PatternDataSourceFilterManager
• Fixed DataGrid params with new data source indexPattern
• Rendering conditions changed depending on data source load
• Checked filter behavior with respect to hide alerts and allow agents.

Hide alerts and allow agents evidence

Current screen

Pinned agent behavior

Evidence_Pinned_Agent_2024-04-18.webm

[jbiset]

Update 19/04/2024

• DiscoverNoResults and LoadingSpinner components are replaced with common components
• The rendering problem is analyzed with respect to results obtained from the data source and the results obtained from embedded visualizations.

[jbiset]

Update 22/04/2024

• Clean code
• Dashboard conditions and definitions were fixed.
• The interaction of the dashboards is tested

Current behavior

Evidence_1.webm

[jbiset]

Update 24/04/2024

• Removed unnecessary general/threat hunting in tabFilters in common data to remove duplicate filters
• Removed unused files
• Added dateRange param to fetchData in dashboard useEffect
• Added hide-filter-control class to hide the button that allows you to affect all the filters in the search bar
• Use searchbarProps deconstruction in dashboard
• Changed Threat Hunting columns file name