Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Server-side create/update ingest pipelines #62744

Conversation

alisonelizabeth
Copy link
Contributor

@alisonelizabeth alisonelizabeth commented Apr 7, 2020

This PR adds the create and update endpoints to the ingest pipelines app.

PUT /api/ingest_pipelines

Create a new pipeline

Example body

{
    "name": "my_pipeline",
    "description": "pipeline description",
    "version": 1,
    "processors" : [
      {
        "set" : {
          "field": "foo",
          "value": "bar"
        }
      }
    ],
    onFailure: [
      {
         set: {
           field: 'error.message',
           value: '{{ failure_message }}'
       	 }
      }
    ]
}

PUT /api/ingest_pipelines/<existing_pipeline>

Update an existing pipeline

Example body

{
    "description": "updated pipeline description",
    "version": 2,
    "processors" : [
      {
        "set" : {
          "field": "foo",
          "value": "bar"
        }
      }
    ]
}

@alisonelizabeth alisonelizabeth added Team:Kibana Management Dev Tools, Index Management, Upgrade Assistant, ILM, Ingest Node Pipelines, and more release_note:skip Skip the PR/issue when compiling release notes labels Apr 7, 2020
@elasticmachine
Copy link
Contributor

Pinging @elastic/es-ui (Team:Elasticsearch UI)

@alisonelizabeth
Copy link
Contributor Author

@elasticmachine merge upstream

@alisonelizabeth
Copy link
Contributor Author

@elasticmachine merge upstream

@alisonelizabeth alisonelizabeth marked this pull request as ready for review April 7, 2020 21:12
@alisonelizabeth alisonelizabeth requested a review from a team as a code owner April 7, 2020 21:12
Copy link
Contributor

@jloleysens jloleysens left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left a few comments that you can choose to address, otherwise I am happy for this to be merged once CI passes!

I just checked that Kibana starts locally. #test-confidence

};

// Remove any undefined values
return Object.entries(pipeline).reduce((pipelineDefinition: any, [key, value]) => {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just curious, what issues was having e.g., { version: undefined ... } causing?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I noticed we had followed this pattern elsewhere, but you bring up a good point. It's not causing any issues atm, so I removed this logic for now.

});
}
} catch (e) {
// Silently swallow error
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we check for 404 here as a success indicator and return internal error otherwise?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't know if we would want to block on this. WDYT?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm going to leave this as is for now, so I can merge this PR and start working on the UI. I'm open to revisiting it in a follow-up PR though.

},
license.guardApiRoute(async (ctx, req, res) => {
const { callAsCurrentUser } = ctx.core.elasticsearch.dataClient;
const { name } = req.params as typeof paramsSchema.type;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should consider modifying license service gaurdApiRoute to preserve type information:

  guardApiRoute<P, Q, B>(handler: RequestHandler<P, Q, B>) {
    const license = this;

    return function licenseCheck(
      ctx: RequestHandlerContext,
      request: KibanaRequest<P, Q, B>,

Then we can remove as declarations (although I can see we may want to keep as Pipeline on the body).

@kibanamachine
Copy link
Contributor

💚 Build Succeeded

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

@alisonelizabeth alisonelizabeth merged commit 7de73c8 into elastic:feature/ingest-node-pipelines Apr 8, 2020
@alisonelizabeth alisonelizabeth deleted the ingest-node-pipelines/create-edit-routes branch April 8, 2020 19:25
license,
lib: { isEsError },
}: RouteDependencies): void => {
router.put(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To follow REST principles it would be better to use POST to create a resource. Tiny change but it goes a long way 😄

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point. I can change this in a follow-up PR. I think I may have used put here originally to align with the underlying ES API.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Change made via 6bba3da

jloleysens added a commit that referenced this pull request Jun 16, 2020
* initial plugin setup

* add smoke test

* fix license check

* refactor plugin setup

* Server-side create/update ingest pipelines (#62744)

* List pipelines (#62785)

* First iteration of ingest table

* Add action placeholders

* Refactor list and address PR feedback

Refactored the list into smaller pieces and assemble in main.tsx

Also addressed feedback on copy, removed unused notifications dep

* WiP on flyout

Showing name in title

* Add reload button

* Finish first version of flyout

* Slight update to copy

* `delete` -> `edit`

* Address PR feedback

Copy and a11y updates

* Add on failure JSON to flyout if it is available

* Add details json block file and remove ununsed import

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

* [Ingest pipelines] Create pipeline UI (#63017)

* First vertical slice of pipeline editor component

* Made a space for common parameters

* [Ingest pipelines] Edit pipeline page (#63522)

* First iteration of CRUD functionality working

* WiP on moving the pipeline editor to pipeline processor editor

* Finish refactor to work with passing state out

* Refactor and fix tests

* [Ingest pipelines] Polish details panel and empty list (#63926)

* Address some early feedback and use FormDataProvider

FormDataProvider gives a more declarative way to listen to form
state updates.

Also refactored the state reader mechanism.

* [Ingest pipelines] Delete pipeline (#63635)

* [Ingest Node Pipelines] Clone Pipeline (#64049)

* First iteration of clone functionality

Wired up for both the list table and the details flyout in the
list section.

* satisfy eslint

* Turn on sorting for the list table

* Clean up const declarations

* Address PR feedback

Sentence-casify and update some other copy.

* Mark edit and delete as primary actions in list table

* Handle URI encoded chars in pipeline name when cloning

* Update to using the more flexible controlled component pattern

To make this component and mappings editor more consistent, both
will expose a more traditional controlled component interface to
consumers.

In the current implementation this requires that consumers use
`useCallback` for `onUpdate` handlers to avoid an infinite
rendering cycle and to avoid staleness bugs in their `onUpdate`
handlers they need to think about what might make the handler
stale.

This approach comes with the benefits and flexibility of
controlled components at the cost of slightly more complex
consumption of the components.

In future, we can explore adding the uncontrolled component
interface too so that consumers have the option to more simply
render and pull data out only when needed

* Handle sub-form validity

The pipelines processor editor not emits overall validity to
consumers

* Fix Jest test

* Refactor some names

prepareDataOut -> serialize
prepareDataIn -> deserialize
EditorProcessor -> ProcessorInternal

* Mark as private

* Major WiP

Started working on the drag-and-drop-tree and updated some
typings stuff

* [Ingest node pipelines] Privileges (#63850)

* Create privileges check for ingest pipelines app

Also moved the public side logic for checking and rendering
privilege related messages to es_ui_shared/public following the
new __packages_do_not_import__ convention.

* Add ,

* Fix import paths

* Address PR feedback

Fix i18n strings (remove reference to snapshot and restore) and
fix copy referencing snapshot and restore - all copy-pasta errors.

Also remove unused field from missing privileges object.

* Fix issue from resolving merge conflicts

* Add missing app privilege

* Use non-deprecated privilige name

* First iteration of drag and drop tree on feature parity

* First steps toward add on failure handler

Updated reducer logic to create the next copy value using immer.

* First iteration of nested tree with add on failure working

* Refactor and some UI layout updates

- Remove the "id" field on processors for now
- Implement the nested remove and update functionality again

* Remove immer (not call stack safe)

Refac to remove immer and reimplemented the immutable set and
get functions.

* [Ingest Node Pipelines] More lenient treatment of on-failure value (#64411)

* Move file to components folder

* [Ingest pipelines] Simulate pipeline (#64223)

* Updated tree rendering

- turn off dropzones for children
- fixed up some padding and margins
- fixed integration with pipeline_form.tsx

The current implementation still has a lot of jank stemming from
the UX with DnD. Unfortunately the nesting has opened a number
of issues here.

* [Ingest Node Pipelines] Show flyout after editing or creating a pipeline (#64409)

* Show flyout after editing or creating a pipeline

* JSX comment

* Show not found flyout

Copied from CCR

* update not found flyout and fix behavior when viewing details from table

* Reset pipeline name in URI when closing flyout

* Remove encodeURI

Already using encodingURIComponent for unsafe string.

Co-authored-by: Alison Goryachev <alisonmllr20@gmail.com>

* Clarification of terms

- addProcessor -> addTopLevelProcessor (same in editor modes)
- Expanded comment on ProcessorSelector

* Implement move between lists functionality

Added tests to the reducer for moving in and out of a recusrsive
structure.

* fix TS

* Prevent nesting a parent inside of its own child

* Add comment

* [Ingest pipelines] Cleanup (#64794)

* address review feedback

* remove unused import

* Big refactor to tree rendering structure

DnD tree now converts the nested tree to a flat one and only
consists of _1_ droppable area with a flat array of draggables
that can be combined.

Using the existing logic in the reducer combined with translating
the flat structure changes to a format the nested reducer can
understand looks like a really promising avenue.

There still seems to be a bug with a longer list where items do
not interact properly.

* Remove unused component

* A number of NB changes

- Fixed a subtle serialisation bug with on_failure in options
- Fixed a performance bug by using useRef in pipeline form
- Memoized the drag and drop tree
- Removed use of "isValid" from form lib... I think this should
be removed entirely.

* fix bad conflict resolution

* Implemented a slightly better destination resolution algo.

Also added tests.

* Fix subtle staleness bug, whitelist keys for setValue

* NB styling fix!!

Due to a parent's setting of overflow: hidden the drag and drop
tree had a dead zone that would equal the page overflow limit,
unsetting on the component itself (overflow: auto) relaxes this
limitation.

* Fix stale delete bug too

* Update naming of editor modes and update comments

* Use field types combo box

* Add delete confirmation modal

* Refactor delete modal component file name

* Better visual integration with existing form

* Update layout and styling of form

- added some padding around the new processors editor field
- Updated use of flex box in the pipeline form fields

* Move pipeline processor copy into pipeline processor component

The test button is also now inside of the pipeline processor
editor. Eventually all of this functionality should be moved
into the pipeline processor editor.

* First step of refactor to moving between trees

* First iteration of x-tree drag and drop

* Remove unused import

* Fix jest test types

* Fix up minor i18n issues and fix up layout of on failure

* Remove unnecessary prop

* Update spacing above add processor button to make it more center

* Fix destination resolution algo

* Update dragging resolver unit tests and add a lot more comments

* Use one sorting algo (removed use of euiReorder for now)

* Add placeholder tests and update comments

* Quite a big refactor

- Remove DraggableLocation entirely, only use ProcessorSelector
this simplifies mapping back to reducer instructions quite a lot
- Add tests for special case dragging behaviour

* Fix off by one bug in tests and implementation 🤦🏼‍♂

- Also move processor reducer to it's own folder

* Update behaviour for dragging up across trees and add tests

* Fix combine instruction

* Fix test and i18n issues

* Remove background color

* Fix selector after selector refactor

* A major performance

- Do not re-render the entire tree when editing settings

* Fix component smoke test

* Fix reading value from processor state using processor selector

* [Ingest pipelines] Custom processor form (#66022)

* Re add background color and refactor name of processor item

* Fix file naming and refactor 🚜 dnd tree rendering

- deserialze and serialize were backwards, fixed
- the dnd tree was rendering a flat tree which needed to be
mapped back to the nested tree for updates. This is still the
case but we do use recursive rendering. This enables tree nodes
to better hold local state like whether they are collapsed or not.

* Fix getting of initial on failure processors value

* Update padding styles for containers

* A lot of styling updates to get closer to look of mockup

* A WiP version of th click-tree an alternative to dnd

As a response to really long pipelines we may pivot away from
dnd.

* Remove dnd tree

* clean up reamining dnd tree references

* Clean up and refactor of tree component

To simplify the logic of the tree component processor id's
were added. Also had to update the jest spec to support this

* Added the ability to duplicate a processor

* Fix types in test

* Added duplicate functionality to ui

* Memoize tree components

* address es lint issues

* remove unused import

* Fix editing of custom json

* Address form performance issues

* Add all known missing processors

* Add ability to cancel move

* Fix staleness in test and view request flyouts

* fix type issue

* Remove unused translations and skip funcitonal test for now

* add todo comment

* Fix type issues

* remove poc styles

* disable other move buttons if we have a selected processor

* Refactor drop zone pin to button and add some styling

* Refactor processor editor item

* Update styling and use icon for cancel move action too

* fix nasty integration bug

* some minor optimizations

* prevent parent from being placed in own on failure handler

* Re-add cancel button

* Re-introduce failure processors toggle

* Fix typo

* Add Handler types for processor editor item

* Fix staleness bug for type, refactor classname and fix duplicate
bug not immutably copying values

* Experimenting with padels (revert this to undo if no further
changes have been made)

* Add description and unique ids

* Share links via component-wide context rather than props

* Virtualized list and back to outline dropzones

* Refactor id getter to a service and make it an incrementing number

* Temporary fix for double rendering issue.

This means the pipeline processors component is not controllable
but fixes the double rendering even when processors have not
changed. This is a problem coming from the ingest pipelines
plugin form system outside of this component

* add todo comment

* remove euicode element

* properly handle duplicate flow

* attempt to fix i18n

* split private_tree into it's own component and add comments

* refactor 🚜. rename Tree to ProcessorsTree and move things around

* do not delete the top level arrays for processors and onfailures

* fix typescript error

* Move duplicate, addOnFailure and delete actions into ctx menu

* remove unused import

* add support for pressing esc key to cancel move

* Add outside click listener

* always prompt before deleting a processor

* refactor remove distinction between adding top level and on fail

* add processor button to tree

* Hide the add on failure context menu item for processors with
failure handlers

* Reinstated x-tree moving and highlight and disable for buttons
on move and on edit

* removing ids step 1: remove idGenerator

* remove ids step 2: added inline text input

Also refactored a lot of the tree actions away. Now using
context and the processors editor tree item component for
dispatching actions. The tree item has access to the dispatch
and to the selector which makes it a well positioned component.

Also saves on some props drilling.

* Slight improvement to styling of text input (border)

* Re-implement missing failure toggle test

* address type todo

* Address many type issues and fix yarn.lock

* re-enable create pipeline functional test

* prevent multiple flyouts from opening

* change flyout title when editing an on-failure processor

* absolutely position the failure handlers label

when we render a dropzone on the label, then we move the label
up without affecting overall component height

* fix description behaviour not removing tag if empty

* some minor clean up

* add onflyoutopen cb to tests

* refactor processors editor item to multiple files

also refactored i18n into it's own file. would be good to come
up with a pttaern for doing this more broadly.

* fix add on-failure handler in context menu after refactor

* tag -> new description field

Co-authored-by: Alison Goryachev <alisonmllr20@gmail.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Co-authored-by: Alison Goryachev <alison.goryachev@elastic.co>
jloleysens added a commit to jloleysens/kibana that referenced this pull request Jun 16, 2020
* initial plugin setup

* add smoke test

* fix license check

* refactor plugin setup

* Server-side create/update ingest pipelines (elastic#62744)

* List pipelines (elastic#62785)

* First iteration of ingest table

* Add action placeholders

* Refactor list and address PR feedback

Refactored the list into smaller pieces and assemble in main.tsx

Also addressed feedback on copy, removed unused notifications dep

* WiP on flyout

Showing name in title

* Add reload button

* Finish first version of flyout

* Slight update to copy

* `delete` -> `edit`

* Address PR feedback

Copy and a11y updates

* Add on failure JSON to flyout if it is available

* Add details json block file and remove ununsed import

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

* [Ingest pipelines] Create pipeline UI (elastic#63017)

* First vertical slice of pipeline editor component

* Made a space for common parameters

* [Ingest pipelines] Edit pipeline page (elastic#63522)

* First iteration of CRUD functionality working

* WiP on moving the pipeline editor to pipeline processor editor

* Finish refactor to work with passing state out

* Refactor and fix tests

* [Ingest pipelines] Polish details panel and empty list (elastic#63926)

* Address some early feedback and use FormDataProvider

FormDataProvider gives a more declarative way to listen to form
state updates.

Also refactored the state reader mechanism.

* [Ingest pipelines] Delete pipeline (elastic#63635)

* [Ingest Node Pipelines] Clone Pipeline (elastic#64049)

* First iteration of clone functionality

Wired up for both the list table and the details flyout in the
list section.

* satisfy eslint

* Turn on sorting for the list table

* Clean up const declarations

* Address PR feedback

Sentence-casify and update some other copy.

* Mark edit and delete as primary actions in list table

* Handle URI encoded chars in pipeline name when cloning

* Update to using the more flexible controlled component pattern

To make this component and mappings editor more consistent, both
will expose a more traditional controlled component interface to
consumers.

In the current implementation this requires that consumers use
`useCallback` for `onUpdate` handlers to avoid an infinite
rendering cycle and to avoid staleness bugs in their `onUpdate`
handlers they need to think about what might make the handler
stale.

This approach comes with the benefits and flexibility of
controlled components at the cost of slightly more complex
consumption of the components.

In future, we can explore adding the uncontrolled component
interface too so that consumers have the option to more simply
render and pull data out only when needed

* Handle sub-form validity

The pipelines processor editor not emits overall validity to
consumers

* Fix Jest test

* Refactor some names

prepareDataOut -> serialize
prepareDataIn -> deserialize
EditorProcessor -> ProcessorInternal

* Mark as private

* Major WiP

Started working on the drag-and-drop-tree and updated some
typings stuff

* [Ingest node pipelines] Privileges (elastic#63850)

* Create privileges check for ingest pipelines app

Also moved the public side logic for checking and rendering
privilege related messages to es_ui_shared/public following the
new __packages_do_not_import__ convention.

* Add ,

* Fix import paths

* Address PR feedback

Fix i18n strings (remove reference to snapshot and restore) and
fix copy referencing snapshot and restore - all copy-pasta errors.

Also remove unused field from missing privileges object.

* Fix issue from resolving merge conflicts

* Add missing app privilege

* Use non-deprecated privilige name

* First iteration of drag and drop tree on feature parity

* First steps toward add on failure handler

Updated reducer logic to create the next copy value using immer.

* First iteration of nested tree with add on failure working

* Refactor and some UI layout updates

- Remove the "id" field on processors for now
- Implement the nested remove and update functionality again

* Remove immer (not call stack safe)

Refac to remove immer and reimplemented the immutable set and
get functions.

* [Ingest Node Pipelines] More lenient treatment of on-failure value (elastic#64411)

* Move file to components folder

* [Ingest pipelines] Simulate pipeline (elastic#64223)

* Updated tree rendering

- turn off dropzones for children
- fixed up some padding and margins
- fixed integration with pipeline_form.tsx

The current implementation still has a lot of jank stemming from
the UX with DnD. Unfortunately the nesting has opened a number
of issues here.

* [Ingest Node Pipelines] Show flyout after editing or creating a pipeline (elastic#64409)

* Show flyout after editing or creating a pipeline

* JSX comment

* Show not found flyout

Copied from CCR

* update not found flyout and fix behavior when viewing details from table

* Reset pipeline name in URI when closing flyout

* Remove encodeURI

Already using encodingURIComponent for unsafe string.

Co-authored-by: Alison Goryachev <alisonmllr20@gmail.com>

* Clarification of terms

- addProcessor -> addTopLevelProcessor (same in editor modes)
- Expanded comment on ProcessorSelector

* Implement move between lists functionality

Added tests to the reducer for moving in and out of a recusrsive
structure.

* fix TS

* Prevent nesting a parent inside of its own child

* Add comment

* [Ingest pipelines] Cleanup (elastic#64794)

* address review feedback

* remove unused import

* Big refactor to tree rendering structure

DnD tree now converts the nested tree to a flat one and only
consists of _1_ droppable area with a flat array of draggables
that can be combined.

Using the existing logic in the reducer combined with translating
the flat structure changes to a format the nested reducer can
understand looks like a really promising avenue.

There still seems to be a bug with a longer list where items do
not interact properly.

* Remove unused component

* A number of NB changes

- Fixed a subtle serialisation bug with on_failure in options
- Fixed a performance bug by using useRef in pipeline form
- Memoized the drag and drop tree
- Removed use of "isValid" from form lib... I think this should
be removed entirely.

* fix bad conflict resolution

* Implemented a slightly better destination resolution algo.

Also added tests.

* Fix subtle staleness bug, whitelist keys for setValue

* NB styling fix!!

Due to a parent's setting of overflow: hidden the drag and drop
tree had a dead zone that would equal the page overflow limit,
unsetting on the component itself (overflow: auto) relaxes this
limitation.

* Fix stale delete bug too

* Update naming of editor modes and update comments

* Use field types combo box

* Add delete confirmation modal

* Refactor delete modal component file name

* Better visual integration with existing form

* Update layout and styling of form

- added some padding around the new processors editor field
- Updated use of flex box in the pipeline form fields

* Move pipeline processor copy into pipeline processor component

The test button is also now inside of the pipeline processor
editor. Eventually all of this functionality should be moved
into the pipeline processor editor.

* First step of refactor to moving between trees

* First iteration of x-tree drag and drop

* Remove unused import

* Fix jest test types

* Fix up minor i18n issues and fix up layout of on failure

* Remove unnecessary prop

* Update spacing above add processor button to make it more center

* Fix destination resolution algo

* Update dragging resolver unit tests and add a lot more comments

* Use one sorting algo (removed use of euiReorder for now)

* Add placeholder tests and update comments

* Quite a big refactor

- Remove DraggableLocation entirely, only use ProcessorSelector
this simplifies mapping back to reducer instructions quite a lot
- Add tests for special case dragging behaviour

* Fix off by one bug in tests and implementation 🤦🏼‍♂

- Also move processor reducer to it's own folder

* Update behaviour for dragging up across trees and add tests

* Fix combine instruction

* Fix test and i18n issues

* Remove background color

* Fix selector after selector refactor

* A major performance

- Do not re-render the entire tree when editing settings

* Fix component smoke test

* Fix reading value from processor state using processor selector

* [Ingest pipelines] Custom processor form (elastic#66022)

* Re add background color and refactor name of processor item

* Fix file naming and refactor 🚜 dnd tree rendering

- deserialze and serialize were backwards, fixed
- the dnd tree was rendering a flat tree which needed to be
mapped back to the nested tree for updates. This is still the
case but we do use recursive rendering. This enables tree nodes
to better hold local state like whether they are collapsed or not.

* Fix getting of initial on failure processors value

* Update padding styles for containers

* A lot of styling updates to get closer to look of mockup

* A WiP version of th click-tree an alternative to dnd

As a response to really long pipelines we may pivot away from
dnd.

* Remove dnd tree

* clean up reamining dnd tree references

* Clean up and refactor of tree component

To simplify the logic of the tree component processor id's
were added. Also had to update the jest spec to support this

* Added the ability to duplicate a processor

* Fix types in test

* Added duplicate functionality to ui

* Memoize tree components

* address es lint issues

* remove unused import

* Fix editing of custom json

* Address form performance issues

* Add all known missing processors

* Add ability to cancel move

* Fix staleness in test and view request flyouts

* fix type issue

* Remove unused translations and skip funcitonal test for now

* add todo comment

* Fix type issues

* remove poc styles

* disable other move buttons if we have a selected processor

* Refactor drop zone pin to button and add some styling

* Refactor processor editor item

* Update styling and use icon for cancel move action too

* fix nasty integration bug

* some minor optimizations

* prevent parent from being placed in own on failure handler

* Re-add cancel button

* Re-introduce failure processors toggle

* Fix typo

* Add Handler types for processor editor item

* Fix staleness bug for type, refactor classname and fix duplicate
bug not immutably copying values

* Experimenting with padels (revert this to undo if no further
changes have been made)

* Add description and unique ids

* Share links via component-wide context rather than props

* Virtualized list and back to outline dropzones

* Refactor id getter to a service and make it an incrementing number

* Temporary fix for double rendering issue.

This means the pipeline processors component is not controllable
but fixes the double rendering even when processors have not
changed. This is a problem coming from the ingest pipelines
plugin form system outside of this component

* add todo comment

* remove euicode element

* properly handle duplicate flow

* attempt to fix i18n

* split private_tree into it's own component and add comments

* refactor 🚜. rename Tree to ProcessorsTree and move things around

* do not delete the top level arrays for processors and onfailures

* fix typescript error

* Move duplicate, addOnFailure and delete actions into ctx menu

* remove unused import

* add support for pressing esc key to cancel move

* Add outside click listener

* always prompt before deleting a processor

* refactor remove distinction between adding top level and on fail

* add processor button to tree

* Hide the add on failure context menu item for processors with
failure handlers

* Reinstated x-tree moving and highlight and disable for buttons
on move and on edit

* removing ids step 1: remove idGenerator

* remove ids step 2: added inline text input

Also refactored a lot of the tree actions away. Now using
context and the processors editor tree item component for
dispatching actions. The tree item has access to the dispatch
and to the selector which makes it a well positioned component.

Also saves on some props drilling.

* Slight improvement to styling of text input (border)

* Re-implement missing failure toggle test

* address type todo

* Address many type issues and fix yarn.lock

* re-enable create pipeline functional test

* prevent multiple flyouts from opening

* change flyout title when editing an on-failure processor

* absolutely position the failure handlers label

when we render a dropzone on the label, then we move the label
up without affecting overall component height

* fix description behaviour not removing tag if empty

* some minor clean up

* add onflyoutopen cb to tests

* refactor processors editor item to multiple files

also refactored i18n into it's own file. would be good to come
up with a pttaern for doing this more broadly.

* fix add on-failure handler in context menu after refactor

* tag -> new description field

Co-authored-by: Alison Goryachev <alisonmllr20@gmail.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Co-authored-by: Alison Goryachev <alison.goryachev@elastic.co>
jloleysens added a commit that referenced this pull request Jun 16, 2020
* initial plugin setup

* add smoke test

* fix license check

* refactor plugin setup

* Server-side create/update ingest pipelines (#62744)

* List pipelines (#62785)

* First iteration of ingest table

* Add action placeholders

* Refactor list and address PR feedback

Refactored the list into smaller pieces and assemble in main.tsx

Also addressed feedback on copy, removed unused notifications dep

* WiP on flyout

Showing name in title

* Add reload button

* Finish first version of flyout

* Slight update to copy

* `delete` -> `edit`

* Address PR feedback

Copy and a11y updates

* Add on failure JSON to flyout if it is available

* Add details json block file and remove ununsed import

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

* [Ingest pipelines] Create pipeline UI (#63017)

* First vertical slice of pipeline editor component

* Made a space for common parameters

* [Ingest pipelines] Edit pipeline page (#63522)

* First iteration of CRUD functionality working

* WiP on moving the pipeline editor to pipeline processor editor

* Finish refactor to work with passing state out

* Refactor and fix tests

* [Ingest pipelines] Polish details panel and empty list (#63926)

* Address some early feedback and use FormDataProvider

FormDataProvider gives a more declarative way to listen to form
state updates.

Also refactored the state reader mechanism.

* [Ingest pipelines] Delete pipeline (#63635)

* [Ingest Node Pipelines] Clone Pipeline (#64049)

* First iteration of clone functionality

Wired up for both the list table and the details flyout in the
list section.

* satisfy eslint

* Turn on sorting for the list table

* Clean up const declarations

* Address PR feedback

Sentence-casify and update some other copy.

* Mark edit and delete as primary actions in list table

* Handle URI encoded chars in pipeline name when cloning

* Update to using the more flexible controlled component pattern

To make this component and mappings editor more consistent, both
will expose a more traditional controlled component interface to
consumers.

In the current implementation this requires that consumers use
`useCallback` for `onUpdate` handlers to avoid an infinite
rendering cycle and to avoid staleness bugs in their `onUpdate`
handlers they need to think about what might make the handler
stale.

This approach comes with the benefits and flexibility of
controlled components at the cost of slightly more complex
consumption of the components.

In future, we can explore adding the uncontrolled component
interface too so that consumers have the option to more simply
render and pull data out only when needed

* Handle sub-form validity

The pipelines processor editor not emits overall validity to
consumers

* Fix Jest test

* Refactor some names

prepareDataOut -> serialize
prepareDataIn -> deserialize
EditorProcessor -> ProcessorInternal

* Mark as private

* Major WiP

Started working on the drag-and-drop-tree and updated some
typings stuff

* [Ingest node pipelines] Privileges (#63850)

* Create privileges check for ingest pipelines app

Also moved the public side logic for checking and rendering
privilege related messages to es_ui_shared/public following the
new __packages_do_not_import__ convention.

* Add ,

* Fix import paths

* Address PR feedback

Fix i18n strings (remove reference to snapshot and restore) and
fix copy referencing snapshot and restore - all copy-pasta errors.

Also remove unused field from missing privileges object.

* Fix issue from resolving merge conflicts

* Add missing app privilege

* Use non-deprecated privilige name

* First iteration of drag and drop tree on feature parity

* First steps toward add on failure handler

Updated reducer logic to create the next copy value using immer.

* First iteration of nested tree with add on failure working

* Refactor and some UI layout updates

- Remove the "id" field on processors for now
- Implement the nested remove and update functionality again

* Remove immer (not call stack safe)

Refac to remove immer and reimplemented the immutable set and
get functions.

* [Ingest Node Pipelines] More lenient treatment of on-failure value (#64411)

* Move file to components folder

* [Ingest pipelines] Simulate pipeline (#64223)

* Updated tree rendering

- turn off dropzones for children
- fixed up some padding and margins
- fixed integration with pipeline_form.tsx

The current implementation still has a lot of jank stemming from
the UX with DnD. Unfortunately the nesting has opened a number
of issues here.

* [Ingest Node Pipelines] Show flyout after editing or creating a pipeline (#64409)

* Show flyout after editing or creating a pipeline

* JSX comment

* Show not found flyout

Copied from CCR

* update not found flyout and fix behavior when viewing details from table

* Reset pipeline name in URI when closing flyout

* Remove encodeURI

Already using encodingURIComponent for unsafe string.

Co-authored-by: Alison Goryachev <alisonmllr20@gmail.com>

* Clarification of terms

- addProcessor -> addTopLevelProcessor (same in editor modes)
- Expanded comment on ProcessorSelector

* Implement move between lists functionality

Added tests to the reducer for moving in and out of a recusrsive
structure.

* fix TS

* Prevent nesting a parent inside of its own child

* Add comment

* [Ingest pipelines] Cleanup (#64794)

* address review feedback

* remove unused import

* Big refactor to tree rendering structure

DnD tree now converts the nested tree to a flat one and only
consists of _1_ droppable area with a flat array of draggables
that can be combined.

Using the existing logic in the reducer combined with translating
the flat structure changes to a format the nested reducer can
understand looks like a really promising avenue.

There still seems to be a bug with a longer list where items do
not interact properly.

* Remove unused component

* A number of NB changes

- Fixed a subtle serialisation bug with on_failure in options
- Fixed a performance bug by using useRef in pipeline form
- Memoized the drag and drop tree
- Removed use of "isValid" from form lib... I think this should
be removed entirely.

* fix bad conflict resolution

* Implemented a slightly better destination resolution algo.

Also added tests.

* Fix subtle staleness bug, whitelist keys for setValue

* NB styling fix!!

Due to a parent's setting of overflow: hidden the drag and drop
tree had a dead zone that would equal the page overflow limit,
unsetting on the component itself (overflow: auto) relaxes this
limitation.

* Fix stale delete bug too

* Update naming of editor modes and update comments

* Use field types combo box

* Add delete confirmation modal

* Refactor delete modal component file name

* Better visual integration with existing form

* Update layout and styling of form

- added some padding around the new processors editor field
- Updated use of flex box in the pipeline form fields

* Move pipeline processor copy into pipeline processor component

The test button is also now inside of the pipeline processor
editor. Eventually all of this functionality should be moved
into the pipeline processor editor.

* First step of refactor to moving between trees

* First iteration of x-tree drag and drop

* Remove unused import

* Fix jest test types

* Fix up minor i18n issues and fix up layout of on failure

* Remove unnecessary prop

* Update spacing above add processor button to make it more center

* Fix destination resolution algo

* Update dragging resolver unit tests and add a lot more comments

* Use one sorting algo (removed use of euiReorder for now)

* Add placeholder tests and update comments

* Quite a big refactor

- Remove DraggableLocation entirely, only use ProcessorSelector
this simplifies mapping back to reducer instructions quite a lot
- Add tests for special case dragging behaviour

* Fix off by one bug in tests and implementation 🤦🏼‍♂

- Also move processor reducer to it's own folder

* Update behaviour for dragging up across trees and add tests

* Fix combine instruction

* Fix test and i18n issues

* Remove background color

* Fix selector after selector refactor

* A major performance

- Do not re-render the entire tree when editing settings

* Fix component smoke test

* Fix reading value from processor state using processor selector

* [Ingest pipelines] Custom processor form (#66022)

* Re add background color and refactor name of processor item

* Fix file naming and refactor 🚜 dnd tree rendering

- deserialze and serialize were backwards, fixed
- the dnd tree was rendering a flat tree which needed to be
mapped back to the nested tree for updates. This is still the
case but we do use recursive rendering. This enables tree nodes
to better hold local state like whether they are collapsed or not.

* Fix getting of initial on failure processors value

* Update padding styles for containers

* A lot of styling updates to get closer to look of mockup

* A WiP version of th click-tree an alternative to dnd

As a response to really long pipelines we may pivot away from
dnd.

* Remove dnd tree

* clean up reamining dnd tree references

* Clean up and refactor of tree component

To simplify the logic of the tree component processor id's
were added. Also had to update the jest spec to support this

* Added the ability to duplicate a processor

* Fix types in test

* Added duplicate functionality to ui

* Memoize tree components

* address es lint issues

* remove unused import

* Fix editing of custom json

* Address form performance issues

* Add all known missing processors

* Add ability to cancel move

* Fix staleness in test and view request flyouts

* fix type issue

* Remove unused translations and skip funcitonal test for now

* add todo comment

* Fix type issues

* remove poc styles

* disable other move buttons if we have a selected processor

* Refactor drop zone pin to button and add some styling

* Refactor processor editor item

* Update styling and use icon for cancel move action too

* fix nasty integration bug

* some minor optimizations

* prevent parent from being placed in own on failure handler

* Re-add cancel button

* Re-introduce failure processors toggle

* Fix typo

* Add Handler types for processor editor item

* Fix staleness bug for type, refactor classname and fix duplicate
bug not immutably copying values

* Experimenting with padels (revert this to undo if no further
changes have been made)

* Add description and unique ids

* Share links via component-wide context rather than props

* Virtualized list and back to outline dropzones

* Refactor id getter to a service and make it an incrementing number

* Temporary fix for double rendering issue.

This means the pipeline processors component is not controllable
but fixes the double rendering even when processors have not
changed. This is a problem coming from the ingest pipelines
plugin form system outside of this component

* add todo comment

* remove euicode element

* properly handle duplicate flow

* attempt to fix i18n

* split private_tree into it's own component and add comments

* refactor 🚜. rename Tree to ProcessorsTree and move things around

* do not delete the top level arrays for processors and onfailures

* fix typescript error

* Move duplicate, addOnFailure and delete actions into ctx menu

* remove unused import

* add support for pressing esc key to cancel move

* Add outside click listener

* always prompt before deleting a processor

* refactor remove distinction between adding top level and on fail

* add processor button to tree

* Hide the add on failure context menu item for processors with
failure handlers

* Reinstated x-tree moving and highlight and disable for buttons
on move and on edit

* removing ids step 1: remove idGenerator

* remove ids step 2: added inline text input

Also refactored a lot of the tree actions away. Now using
context and the processors editor tree item component for
dispatching actions. The tree item has access to the dispatch
and to the selector which makes it a well positioned component.

Also saves on some props drilling.

* Slight improvement to styling of text input (border)

* Re-implement missing failure toggle test

* address type todo

* Address many type issues and fix yarn.lock

* re-enable create pipeline functional test

* prevent multiple flyouts from opening

* change flyout title when editing an on-failure processor

* absolutely position the failure handlers label

when we render a dropzone on the label, then we move the label
up without affecting overall component height

* fix description behaviour not removing tag if empty

* some minor clean up

* add onflyoutopen cb to tests

* refactor processors editor item to multiple files

also refactored i18n into it's own file. would be good to come
up with a pttaern for doing this more broadly.

* fix add on-failure handler in context menu after refactor

* tag -> new description field

Co-authored-by: Alison Goryachev <alisonmllr20@gmail.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Co-authored-by: Alison Goryachev <alison.goryachev@elastic.co>

Co-authored-by: Alison Goryachev <alisonmllr20@gmail.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Co-authored-by: Alison Goryachev <alison.goryachev@elastic.co>
mistic added a commit to mistic/kibana that referenced this pull request Jun 24, 2020
chore(NA): merge changes on renovate config file

chore(NA): move everything under alias lodash4

chore(NA): missing lodash to lodash4 alias

chore(NA): migrate packages to lodash4

chore(NA): update renovate cfg file

chore(NA): skip failing test

chore(NA): first remove of alias lodash4

chore(NA): remove lodash4 alias

chore(NA): remove skip from test

[ES UI] Error handling (#68809)

[Console] Scrub the lead and trailing brackets from ipv6 host names (#68991)

* scrub the lead and trailing brackets from ipv6 host names

* Update comment

* refactor: scrub -> sanitize

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

fix spec color highlighting not working on vega vis (#68995)

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

[SECURITY] rename server plugin to siem (#69044)

* rename server plugin to siem to avoid privilege issue

* review from alerting

* missing change with rename

* fix tests

* missing api integration test

* fix api integration spaces

Type safe browser.executeAsync (#69018)

* make browserAsync type safe

* adopt tests

* prefer unknown over any

* simplify signature

fixes 'Configures a new connector' flaky test (#69011)

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

Embed dashboard by value example & some embeddable clean up (#67783)

Added example for using dashboard container by value
1.1 Refactored embeddable explorer e2e test to use new example, removed not needed kbn_tp_embeddable_explorer plugin.
For embeddable explorer examples went away from using getFactoryById() to improve type checks
There is new component a replacement for EmbeddableFactoryRenderer with slightly more flexible api: EmbeddableRenderer.
3.1 We can improve it going forward to support more use case

[Endpoint] Adding alerts route (#68183)

* Adding alerts route

* Adding related alerts generator changes, tests, and script updates

* Fixing missed parameter

* Aligning the AlertEvent and ResolverEvent definition

* Fixing type errors

* Fixing import error

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

[Monitoring] Ensure we use existing Elasticsearch config (#68389)

* Ensure we use existing Elasticsearch config

* Use separate type for this to ensure custom properties work

* PR suggestions

* PR feedback

* PR feedback

* Fix type issues

* PR feedback

[APM] Changes to duration formatting (#69039)

* [APM] Changes to duration formatting

* Change the threshold for showing microseconds to 1 millisecond instead of 10. This means you now get "900 µs/1.3 ms/20.0 ms" instead of "900 µs/1300 µs/20 ms."
* Change milliseconds formatted with `asDuration` to be `asDecimal` instead of `asInteger`. That means you get "0.0 ms/2.5 ms/3.0 ms" instead of "0 ms/2 ms/3 ms."
* Tables were all using their own module-scoped functions that called `asDuration` to format things as decimal milliseconds. Extract this to an `asMillisecondDuration` function exported from the duration helpers and use it in all the tables.
* Change `getResponseTimeseries` in chart selectors to use `asDuration` to make all chart timeseries units consistent.
* Don't export `convertTo` from the duration helpers as it's now not used anywhere. Always use a more specific exported function for more consistency.
* Change ">=" to "≥" in the ML flyout text.

* Update e2e snapshot

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

[SECURITY SOLUTION] Task/endpoint details to fleet (#68710)

[SECURITYSOLUTION][INGEST] Fixes endpoint link, adds link from endpoint details to fleet

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

Set wrap to the errorLink so it doesn't go outside of box (#67797)

* set wrap to the errorLink so it doesnt go outside of box

* updating

* set wrap to the errorLink so it doesnt go outside of box

* updating

* Show full message on hover

Co-authored-by: Nathan L Smith <smith@nlsmith.com>

[Metrics UI] Add preview feature for metric threshold alerts (#67684)

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

[skip-ci] Rename @elastic/pulse to @elastic/kibana-telemetry (#69181)

try to fix integration test by making template more specific (#69138)

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

[Alerting] Update Index Threshold to use columns in EuiExpression (#69051)

Add enroll agent action to config action menu (#68840)

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

Don't use filter to get default config & output (#69088)

* Don't use filter to get default config & output

the KQL parsing takes up a lot of CPU time. Avoid it with a search string

Also add a `refresh: false` as suggested by @bkobel

* Does restoring await fix CI?

* Remove 'refresh: false' from agent enroll

Does this this fix the failing test? It does keep the PR more focused

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

[SIEM][Detections] In progress alert state (#68569)

[Endpoint] Remove dependency on ingest for the index patterns (#69058)

* Remove dependency on ingest for the index patterns

* Fixing the tests

* Fixing test

* Use variable instead of class

Updates the Release Notes content in CONTRIBUTING (#69032)

* Updates the Release Notes content in CONTRIBUTING

* Release Notes guidelines

* Fixes bulleted list indentation

Fix plugin lifecycle log to only include server plugins (#68686)

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

[Ingest Manager] Add ability to copy an agent config (#68737)

* Add copy agent config endpoint, schema types, and request hook

* Add copy config to context menu and modal

* Change data sources to be copied via bulk instead of singles in parallel

* Add copy endpoint to openapi spec file

* Add api integration tests

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

[ML] Allow editing of model_snapshot_retention_days (#69003)

* [ML] Allow editing of model_snapshot_retention_days

* removing commented code

* flooring number input

* adding daily_model_snapshot_retention_after_days

* updating default values

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

Remove obsolete custom types from super date picker handlers. (#69038)

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

Using correct index patterns (#69208)

[Endpoint] [ES Archiver] Allowing create option to be passed through the cli for es archiver (#69191)

* Allowing create option to be passed through the cli

* Using kebab casing

[Security] siem to securitySolution SO migration (#68776)

chore(NA): include hidden files when creating package with plugin helpers (#68247)

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

[Reporting/Server] register plugin routes synchronously (#68976)

* register routes synchronously

* back out some refactoring

* comment fix

* fix tests

* register route handler context provider

* Add function level comments in core methods

* fix tests

* revert editor help

* route context is the ReportingStart contract

* Fix reporting job route tests

* Fix generation tests

Co-authored-by: Joel Griffith <joel.griffith@elastic.co>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

[release-notes] add script to generate release notes from PRs (#68816)

Co-authored-by: spalger <spalger@users.noreply.github.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

Sort filter fields by items that start with typed characters. (#68585)

[kbn/optimizer] share all plugin bundles (#68986)

Co-authored-by: spalger <spalger@users.noreply.github.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

[CI] GitHub check details link to test reports and errorlevel (#69157)

[Alerting] Adds API for executing actions immediately (#68897)

This PR adds an API to the AlertsClient which allows users to execute actions immediately, rather than via TaskManager.

This also moves the TaskManager based execution to a new api on ActionsClient along side the immediate execution.

stabilize flaky embeddables/adding_children take 2 (#68873)

* wait for loading indicator to be deleted

* improve with retry

* Update test/examples/embeddables/adding_children.ts

Co-authored-by: Spencer <email@spalger.com>

Co-authored-by: Spencer <email@spalger.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

Update vega version (#68639)

* update vega version

* a11y skip test

* a11y skip test attempt 2

* adding back all a11y tests

* fix failed karma test

* remove extra  0BSD license

* coalesce yarn.lock versions a little

* update kbn/pm dist

* fix CI

* fix Vega View for ML

Co-authored-by: Michail Yasonik <michail.yasonik@elastic.co>
Co-authored-by: spalger <spalger@users.noreply.github.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

[APM] Fix confusing request/minute viz (#69143)

[Ingest pipelines] Fix create pipeline button copy (#69118)

[ILM] Update doc link (#68923)

[ML] Model snapshot management (#68182)

* [ML] Model snapshot management

* more updates

* adding calendar range

* updating layout

* multiple calendars

* moving calendar creator

* fixing chart issues

* fixing chart issues

* improving calendar rendering

* adding capabilities checks

* code clean up

* fixing end time argument type

* fix translations

* code clean up

* comments based on review

* changes based on review

* fixing include

* adding useMemo to theme function

[Maps] Security layer wizards (#68290)

* [Maps] security layer wizard

* index pattern select

* use IndexPattern

* destination layer

* source layer

* cache security index patterns

* line layer

* use top hits

* tslint

* unit test

* fix loading speed and i18n

* handle case where ui setting siem:defaultIndex does not exist

* tslint

* review feedback

* extract lazy load bundle from map embeddables

* expose createSecurityLayers via plugin start

* tslint

* export MapsPluginStart

* fix path move

* review feedback

* fix import

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

[Ingest Node Pipelines] Pipeline Processors Editor (#66021)

* initial plugin setup

* add smoke test

* fix license check

* refactor plugin setup

* Server-side create/update ingest pipelines (#62744)

* List pipelines (#62785)

* First iteration of ingest table

* Add action placeholders

* Refactor list and address PR feedback

Refactored the list into smaller pieces and assemble in main.tsx

Also addressed feedback on copy, removed unused notifications dep

* WiP on flyout

Showing name in title

* Add reload button

* Finish first version of flyout

* Slight update to copy

* `delete` -> `edit`

* Address PR feedback

Copy and a11y updates

* Add on failure JSON to flyout if it is available

* Add details json block file and remove ununsed import

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

* [Ingest pipelines] Create pipeline UI (#63017)

* First vertical slice of pipeline editor component

* Made a space for common parameters

* [Ingest pipelines] Edit pipeline page (#63522)

* First iteration of CRUD functionality working

* WiP on moving the pipeline editor to pipeline processor editor

* Finish refactor to work with passing state out

* Refactor and fix tests

* [Ingest pipelines] Polish details panel and empty list (#63926)

* Address some early feedback and use FormDataProvider

FormDataProvider gives a more declarative way to listen to form
state updates.

Also refactored the state reader mechanism.

* [Ingest pipelines] Delete pipeline (#63635)

* [Ingest Node Pipelines] Clone Pipeline (#64049)

* First iteration of clone functionality

Wired up for both the list table and the details flyout in the
list section.

* satisfy eslint

* Turn on sorting for the list table

* Clean up const declarations

* Address PR feedback

Sentence-casify and update some other copy.

* Mark edit and delete as primary actions in list table

* Handle URI encoded chars in pipeline name when cloning

* Update to using the more flexible controlled component pattern

To make this component and mappings editor more consistent, both
will expose a more traditional controlled component interface to
consumers.

In the current implementation this requires that consumers use
`useCallback` for `onUpdate` handlers to avoid an infinite
rendering cycle and to avoid staleness bugs in their `onUpdate`
handlers they need to think about what might make the handler
stale.

This approach comes with the benefits and flexibility of
controlled components at the cost of slightly more complex
consumption of the components.

In future, we can explore adding the uncontrolled component
interface too so that consumers have the option to more simply
render and pull data out only when needed

* Handle sub-form validity

The pipelines processor editor not emits overall validity to
consumers

* Fix Jest test

* Refactor some names

prepareDataOut -> serialize
prepareDataIn -> deserialize
EditorProcessor -> ProcessorInternal

* Mark as private

* Major WiP

Started working on the drag-and-drop-tree and updated some
typings stuff

* [Ingest node pipelines] Privileges (#63850)

* Create privileges check for ingest pipelines app

Also moved the public side logic for checking and rendering
privilege related messages to es_ui_shared/public following the
new __packages_do_not_import__ convention.

* Add ,

* Fix import paths

* Address PR feedback

Fix i18n strings (remove reference to snapshot and restore) and
fix copy referencing snapshot and restore - all copy-pasta errors.

Also remove unused field from missing privileges object.

* Fix issue from resolving merge conflicts

* Add missing app privilege

* Use non-deprecated privilige name

* First iteration of drag and drop tree on feature parity

* First steps toward add on failure handler

Updated reducer logic to create the next copy value using immer.

* First iteration of nested tree with add on failure working

* Refactor and some UI layout updates

- Remove the "id" field on processors for now
- Implement the nested remove and update functionality again

* Remove immer (not call stack safe)

Refac to remove immer and reimplemented the immutable set and
get functions.

* [Ingest Node Pipelines] More lenient treatment of on-failure value (#64411)

* Move file to components folder

* [Ingest pipelines] Simulate pipeline (#64223)

* Updated tree rendering

- turn off dropzones for children
- fixed up some padding and margins
- fixed integration with pipeline_form.tsx

The current implementation still has a lot of jank stemming from
the UX with DnD. Unfortunately the nesting has opened a number
of issues here.

* [Ingest Node Pipelines] Show flyout after editing or creating a pipeline (#64409)

* Show flyout after editing or creating a pipeline

* JSX comment

* Show not found flyout

Copied from CCR

* update not found flyout and fix behavior when viewing details from table

* Reset pipeline name in URI when closing flyout

* Remove encodeURI

Already using encodingURIComponent for unsafe string.

Co-authored-by: Alison Goryachev <alisonmllr20@gmail.com>

* Clarification of terms

- addProcessor -> addTopLevelProcessor (same in editor modes)
- Expanded comment on ProcessorSelector

* Implement move between lists functionality

Added tests to the reducer for moving in and out of a recusrsive
structure.

* fix TS

* Prevent nesting a parent inside of its own child

* Add comment

* [Ingest pipelines] Cleanup (#64794)

* address review feedback

* remove unused import

* Big refactor to tree rendering structure

DnD tree now converts the nested tree to a flat one and only
consists of _1_ droppable area with a flat array of draggables
that can be combined.

Using the existing logic in the reducer combined with translating
the flat structure changes to a format the nested reducer can
understand looks like a really promising avenue.

There still seems to be a bug with a longer list where items do
not interact properly.

* Remove unused component

* A number of NB changes

- Fixed a subtle serialisation bug with on_failure in options
- Fixed a performance bug by using useRef in pipeline form
- Memoized the drag and drop tree
- Removed use of "isValid" from form lib... I think this should
be removed entirely.

* fix bad conflict resolution

* Implemented a slightly better destination resolution algo.

Also added tests.

* Fix subtle staleness bug, whitelist keys for setValue

* NB styling fix!!

Due to a parent's setting of overflow: hidden the drag and drop
tree had a dead zone that would equal the page overflow limit,
unsetting on the component itself (overflow: auto) relaxes this
limitation.

* Fix stale delete bug too

* Update naming of editor modes and update comments

* Use field types combo box

* Add delete confirmation modal

* Refactor delete modal component file name

* Better visual integration with existing form

* Update layout and styling of form

- added some padding around the new processors editor field
- Updated use of flex box in the pipeline form fields

* Move pipeline processor copy into pipeline processor component

The test button is also now inside of the pipeline processor
editor. Eventually all of this functionality should be moved
into the pipeline processor editor.

* First step of refactor to moving between trees

* First iteration of x-tree drag and drop

* Remove unused import

* Fix jest test types

* Fix up minor i18n issues and fix up layout of on failure

* Remove unnecessary prop

* Update spacing above add processor button to make it more center

* Fix destination resolution algo

* Update dragging resolver unit tests and add a lot more comments

* Use one sorting algo (removed use of euiReorder for now)

* Add placeholder tests and update comments

* Quite a big refactor

- Remove DraggableLocation entirely, only use ProcessorSelector
this simplifies mapping back to reducer instructions quite a lot
- Add tests for special case dragging behaviour

* Fix off by one bug in tests and implementation 🤦🏼‍♂

- Also move processor reducer to it's own folder

* Update behaviour for dragging up across trees and add tests

* Fix combine instruction

* Fix test and i18n issues

* Remove background color

* Fix selector after selector refactor

* A major performance

- Do not re-render the entire tree when editing settings

* Fix component smoke test

* Fix reading value from processor state using processor selector

* [Ingest pipelines] Custom processor form (#66022)

* Re add background color and refactor name of processor item

* Fix file naming and refactor 🚜 dnd tree rendering

- deserialze and serialize were backwards, fixed
- the dnd tree was rendering a flat tree which needed to be
mapped back to the nested tree for updates. This is still the
case but we do use recursive rendering. This enables tree nodes
to better hold local state like whether they are collapsed or not.

* Fix getting of initial on failure processors value

* Update padding styles for containers

* A lot of styling updates to get closer to look of mockup

* A WiP version of th click-tree an alternative to dnd

As a response to really long pipelines we may pivot away from
dnd.

* Remove dnd tree

* clean up reamining dnd tree references

* Clean up and refactor of tree component

To simplify the logic of the tree component processor id's
were added. Also had to update the jest spec to support this

* Added the ability to duplicate a processor

* Fix types in test

* Added duplicate functionality to ui

* Memoize tree components

* address es lint issues

* remove unused import

* Fix editing of custom json

* Address form performance issues

* Add all known missing processors

* Add ability to cancel move

* Fix staleness in test and view request flyouts

* fix type issue

* Remove unused translations and skip funcitonal test for now

* add todo comment

* Fix type issues

* remove poc styles

* disable other move buttons if we have a selected processor

* Refactor drop zone pin to button and add some styling

* Refactor processor editor item

* Update styling and use icon for cancel move action too

* fix nasty integration bug

* some minor optimizations

* prevent parent from being placed in own on failure handler

* Re-add cancel button

* Re-introduce failure processors toggle

* Fix typo

* Add Handler types for processor editor item

* Fix staleness bug for type, refactor classname and fix duplicate
bug not immutably copying values

* Experimenting with padels (revert this to undo if no further
changes have been made)

* Add description and unique ids

* Share links via component-wide context rather than props

* Virtualized list and back to outline dropzones

* Refactor id getter to a service and make it an incrementing number

* Temporary fix for double rendering issue.

This means the pipeline processors component is not controllable
but fixes the double rendering even when processors have not
changed. This is a problem coming from the ingest pipelines
plugin form system outside of this component

* add todo comment

* remove euicode element

* properly handle duplicate flow

* attempt to fix i18n

* split private_tree into it's own component and add comments

* refactor 🚜. rename Tree to ProcessorsTree and move things around

* do not delete the top level arrays for processors and onfailures

* fix typescript error

* Move duplicate, addOnFailure and delete actions into ctx menu

* remove unused import

* add support for pressing esc key to cancel move

* Add outside click listener

* always prompt before deleting a processor

* refactor remove distinction between adding top level and on fail

* add processor button to tree

* Hide the add on failure context menu item for processors with
failure handlers

* Reinstated x-tree moving and highlight and disable for buttons
on move and on edit

* removing ids step 1: remove idGenerator

* remove ids step 2: added inline text input

Also refactored a lot of the tree actions away. Now using
context and the processors editor tree item component for
dispatching actions. The tree item has access to the dispatch
and to the selector which makes it a well positioned component.

Also saves on some props drilling.

* Slight improvement to styling of text input (border)

* Re-implement missing failure toggle test

* address type todo

* Address many type issues and fix yarn.lock

* re-enable create pipeline functional test

* prevent multiple flyouts from opening

* change flyout title when editing an on-failure processor

* absolutely position the failure handlers label

when we render a dropzone on the label, then we move the label
up without affecting overall component height

* fix description behaviour not removing tag if empty

* some minor clean up

* add onflyoutopen cb to tests

* refactor processors editor item to multiple files

also refactored i18n into it's own file. would be good to come
up with a pttaern for doing this more broadly.

* fix add on-failure handler in context menu after refactor

* tag -> new description field

Co-authored-by: Alison Goryachev <alisonmllr20@gmail.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Co-authored-by: Alison Goryachev <alison.goryachev@elastic.co>

[ML] DF Analytics Results: do not hide query bar and table on query syntax error (#69196)

* do not hide query bar and table if query syntax error

* check for query error in analytics exploration

[Ingest Manager] Use search param vs KQL for events count (#69209)

* Replace filter with search in events count

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

[Ingest Manager] Use search param vs KQL to find default agent (#69211)

* Remove another instance of filter for is_default

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

[SIEM][Detection Engine] Fixes a stale view/query within the signals table when re-opening/closing signals

Fixes a one liner bug where we were not using waitFor and could have stale views of data from the timeline. This removes that by adding a `waitFor` when setting the signals to be closed, open, in-progress, etc...

This also adds a new `waitFor` for the end to end tests and fixes other tests to use that rather than doing a wait for 5 seconds. This should keep the end to end backend tests fast.

Before this you could sometimes try to re-open 3 signals like below and it would not change the signals to being open but rather re-query the stale view and show the same signals as being closed when they are not closed:
![crop_bug](https://user-images.githubusercontent.com/1151048/84713572-8b5e7f00-af28-11ea-80b6-2ad67f16b7df.gif)

Now with this PR, those signals will show as being re-opened and the closed signals will update correctly within the same UI on the same tab.

- [x] [Unit or functional tests](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility) were updated or added to match the most common scenarios

[Observability] Update landing page copy and content (#69247)

* removing try it section

* removing try it section

feat: instrument navigation changes using RUM agent (#67633)

* feat: instrument navigation changes using RUM agent

* chore: rebase and change application contract

* chore: fix type tests

* docs: update public.md doc

* chore: remove internal application export

[Uptime] Monitor availability reporting (#67790)

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

[DOCS] Adds classification to data frame analytics overview (#69068)

Bump react-redux (#69182)

[kbn/es] only make one attempt in tests to avoid timeout (#69197)

Co-authored-by: spalger <spalger@users.noreply.github.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

Resolve security cloud test failures (#68935)

Co-authored-by: Joe Portner <5295965+jportner@users.noreply.github.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

SIEM] Moves validation up to the common section

Moves validation up to the common section so it can be used by others in common for API boundary validation.

[QA] Code coverage: fix flaky tests (#69272)

* skip test

* [page_objects/common_page] update closing toast

use navigateToUrl instead of window location (#69167)

[kbn/optimizer] fix windows compatibility (#69304)

Co-authored-by: spalger <spalger@users.noreply.github.com>

[Search service] Refactor the way server-side search strategies are registered (#68452)

* [search] Refactor the way search strategies are registered/retrieved on the server

* Fix types and tests and update docs

* Fix failing test

* Fix build of example plugin

* Fix functional test

* Make server strategies sync

Co-authored-by: Liza K <liza.katz@elastic.co>

[ci/getCheckoutInfo] retry fetching upstream changes to calculate mergeBase (#69320)

Co-authored-by: spalger <spalger@users.noreply.github.com>

[QA] [Code Coverage] Doc update (#69204)

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

[CI] Fix packer cache git branch reference value (#69207)

[kbn/pm] only count cached project (#69113)

Co-authored-by: spalger <spalger@users.noreply.github.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

[Reporting] Prepare export type definitions for Task Manager (#65213)

[APM] Fix service maps not loading when there are no APM ML jobs (#69240)

* Closes #69238 by handling 404 thrown error from the query for APM ML jobs.

* Improved coded readability

* moved anomaly job fetch to new function getApmAnomalyDetectionJobs for
improved readability and only handle a 404 status code response else throw

[BundleRefPlugin] resolve imports to files too (#69241)

Co-authored-by: spalger <spalger@users.noreply.github.com>

[DOCS] Fixes license management links (#69347)

[Component template] Details flyout (#68732)

[SIEM] Adds example unit test to convert KQL using a nested query

Adds example unit test to convert KQL using a nested query

- [x] [Unit or functional tests](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility) were updated or added to match the most common scenarios

Explore underlying data (#68496)

* feat: 🎸 stub discover_enhanced plugin

* feat: 🎸 improve view in discover action

* feat: 🎸 add URL generator to "View in Discover" action

* feat: 🎸 implement navigation and getHref in view raw logs actio

* fix: 🐛 disable action in "edit" mode

* refactor: 💡 renamce context menu view in discover action

* feat: 🎸 rename action to "explore data"

* fix: 🐛 correctly generate action path

* feat: 🎸 add internationalization to "explore action"

* fix: 🐛 correctly parse generated Discover URL path

* test: 💍 setup basic functional tests

* refactor: 💡 modularize url generation logic

* feat: 🎸 export CommonlyUsed type

* test: 💍 add test subjects to panel custom time range modal

* test: 💍 add index patterna and time range functional tests

* refactor: 💡 rename action file

* refactor: 💡 use URL generator from Discover plugin's contract

* test: 💍 add "Explore raw data" action unit tests

* fix: 🐛 import share plugin to check if it is enabled

* Update x-pack/plugins/discover_enhanced/public/actions/view_in_discover/explore_data_context_menu_action.ts

Co-authored-by: Matthias Wilhelm <ankertal@gmail.com>

* chore: 🤖 add discover_enhanced to KibanaApp codeowners

* test: 💍 improve "Explore underlying data" functional tests

* test: 💍 improve <a> link assertion

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Co-authored-by: Matthias Wilhelm <ankertal@gmail.com>

[APM] Only add decimals for numbers below 10 (#69334)

Replaces the Custom Color Picker on TSVB with the EuiColorPicker (#68888)

* Replace the Custom Color Picker on TSVB with the EuiColorPicker

* Remove the custom picker sass

* Remove private modules of eui and the custom color swatches

* Clear the color

* changes in test implementation

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

[SIEM][Timeline] Persist timeline to localStorage (#67156)

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Co-authored-by: Xavier Mouligneau <189600+XavierM@users.noreply.github.com>

feat: 🎸 don't show drilldown action in "edit" mode (#69371)

[ML] Functional tests - add more recognize and setup module API tests (#69251)

This PR adds a couple recognize module and setup module API integration tests.

[ML] Functional tests - Reduce DFA job model memory (#69295)

With the improvements in the model memory estimation for DFA jobs, the required memory limit could be reduced, which allows execution in a low-memory environment (like the 1GB ML node in a cloud trial).

Update dependency @elastic/charts to v19.5.2 (#69126)

Update @elastic/charts to 19.5.2

Co-authored-by: nickofthyme <nick.ryan.partridge@gmail.com>

[SIEM][Timeline] Minor timeline improvement (#69386)

[DOCS] Updates titles in Maps docs (#68703)

* [DOCS] Updates titles in Maps docs

* [DOCS] Changes Elastic Maps to Maps to match the UI

* [DOCS] Fixes titles

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

[Ingest Manager][Endpoint] Add Endpoint Create Policy flow with Ingest (#68955)

* Ingest: add data-test-subj prop support to Ingest components
* Ingest: Add Context Provider to bridge History from Kibana to Hash router
* Ingest: Added support for route state in Create Datasource page
* Endpoint: Add Create button to Polices List header
* Endpoint: Added support for passing of state from endpoint to ingest on policy
* Endpoint: additional functional test cases

[APM] Replace ML index queries with searching via mlAnomalySearch API (#69099)

* Closes #69092 by replacing direct queries on ml indices with seaching
via the `mlAnomalySearch` client API + job_id filters. Also removes
`getMlIndex` since it is no longer relevant.

* Use the mlCapabilities API to ensure the required license is active for ml queries

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

Drilldown docs 2 (#69375)

* docs: ✏️ add drilldown user docs link to REAMDE

* feat: 🎸 wire drilldown docs link into <HelloBar> component

* feat: 🎸 set new page attributes on drilldown docs link

* feat: 🎸 add external prop to link to show icon

* fix: 🐛 remove rel= attribute

* fix: 🐛 add TypeScript type for drilldown docLink

[Endpoint] add policy data to Host list UI (#69202)

[Ingest Pipelines] Add doc links to processor types (#69279)

* added doc links to processor types

* Bring doc links in line with mappings editor

Also refactor the processors type map

* remove helpText prop from Custom field

* fix i18n

* rename doc button and refactor type map const name

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

Removing flaky axe rule (#69418)

[Ingest Manager] Replace `datasources` with `inputs` when generating agent config (#69226)

* Adjust agent config generation, change `datasources` to `inputs`

* Add dataset.type

* Remove dead code

* Revert "Add dataset.type"

This reverts commit fbcf50cbe2b6a93536386c82d754a6a7c7f2b8e9.

* Update endpoint policy test assertion

[ML] Add Anomaly Swimlane Embeddable to the dashboard from the Anomaly Explorer page (#68784)

* [ML] WIP attach swimlane embeddable to dashboard from the explorer page

* [ML] fix deps

* [ML] getDefaultPanelTitle

* [ML] fix TS issue

* [ML] DashboardService

* [ML] unit tests

* [ML] redirect to the dashboard

* [ML] swimlane_panel

* [ML] Anomaly Timeline panel

* [ML] swimlane container

* [ML] fix ts

* [ML] Add multiple swimlanes

* [ML] fix SwimlaneType usage

* [ML] disable edit button on update

* [ML] fix i18n translation key

* [ML] use ViewMode enum

* [ML] use navigateToUrl

* [ML] TODO for edit dashboard

* [ML] check kibana dashboard capabilities

* [ML] mlApiServicesProvider

* [ML] mlResultsServiceProvider

* [ML] fix alignment

* [ML] labels and tooltips

* [ML] fix ts issue for proxyHttpStart

* [ML] canEditDashboards check

* [ML] fix TS

* [ML] update add_to_dashboard_control.tsx

* [ML] add form label, disable control on empty swimlanes selection

* [ML] resolve PR review comments

* [ML] e2e test

* [ML] increase panel padding

* [ML] position in row

* [ML] update e2e

* [ML] add data-test-subj for search box

* [ML] PR remarks

[docker] add spaces settings (#69019)

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

[Maps] layer wizard select re-design (#69313)

* [Maps] layer wizard select re-design

* review feedback

* tslint

* add unit test

* use smaller gutters

* review feedback

skip flaky suite (#52854)

[APM] License feature tracking for service maps (#69455)

Use the license feature API to register the service maps feature and track its usage when the API endpoint is accessed.

Fixes #64850.

fixes screenshots upload (#69392)

Remove unused deps (#69243)

* remove unused deps

* move types in devDeps

[Observability] Creates "Add data" links in all Observability app headers (#69016)

* Add data options to all Observability app headers

* Updated failing snapshot

* [Uptime] Update snapshot

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

Beats Management: Pre-migration cleanup (#69155)

* adapt KibanaDatabaseAdapter to use core ES service

* remove unused `exposeStaticDir` method

* create empty KP plugin

* remove unused and broken tests

* delete unused wallaby config file

* delete unused utils

* delete unused d.ts and move formsy definition to new plugin

* fix findNonExistentItems

* remove dead code and useless exports from common package

* delete non migratable test suite + remove unused test adapter

* remove yet another unused adapter

* restore awaits in KibanaDatabaseAdapter

Add section about marble testing to `TESTING.md` (#68749)

* Add section about marble testing

* improve `callServerAPI` example

* review comments

* add comment on abort observable anti-pattern

savedObjects: add `score` to repository.find results (#68894)

* add `score` to repository.find results

* update generated doc

* fix FTR result set

* remove score from exports

* fix FTR for find API

* fix label

* fix tsdoc

Add functional test for Kibana embedded in iframe (#68544)

* convert kbn test config into TS

* add test  for Kibana embedded in iframe

* run embedded tests in functional suite

* ignore tls errors in functional tests by default

* switch test to https

* remove env vars mutation

* allow to pass ssl config to Kibana

* pass ssl config to axios

* adopt KbnClient interfaces

* adopt KibanaServer

* use KbnRequester in security service

* set sameSiteCookies:None in test

* acceptInsecureCerts in chrome

* remove leftovers

* fix type error

* remove unnecessary field

* address comments

* refactor plugin

* refactor test

* make acceptInsecureCerts configurable

* run firefox tests on ci

* up TS version

* fix firefox.sh script

* fix path

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

[APM] Add error rate chart to Errors overview and detail views (#67327)

* creating error rate chart

* adding error line chart

* creating error rate chart

* using date_histogram

* reapplying prettier style

* changing to theme color

* dont sync tooltips

* adding avg on error charts

* addressing pr comments

* adding possibility to disable legend toggle

* removing x-axis ticks from histogram

* return no percent when transaction count doesn return hits

* addressing PR comments

* addressing PR comments

* returning null when there is no transaction count

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

[APM] Change the unsaved changes text color (#69493)

Fixes dark mode issue where the text would render dark on a dark background.

[refactor] remove drilldown plugin, move components to uiActionsEnhanced (#69403)

* refactor: remove drilldown plugin by moving components to uiActionsEnhanced

* fix

* fix mistake in test

* fix i18n

HttpService: duplicate some APIs from `setup` to `start` and clean mocks. (#69021)

* add apis to http start contract and clean mocks

* add apis to http start contract and clean mocks

* add `getStartContract` method

* fix new calls

* remove isTlsEnabled

* deprecates HttpServiceSetup.auth

[SIEM] Fixes REST formatter bugs from io-ts migration

Fixes io-ts formatter bugs for REST and validation by:

* First trying to get the correct key from the io-ts context. If no keys are found, then it will fall back on trying to get the first name from the context.
* If the key is a value and an object then this will do a `JSON.stringify()` on the value object
* This fixes a few places where `formatError` was not being used within the code base resulting in `[object Object]` within the validations to show up.

- [x] [Unit or functional tests](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility) were updated or added to match the most common scenarios

[RUM Dashboard] Initial Version (#68778)

Co-authored-by: Casper Hübertz <casper@formgeist.com>

Fix home page loading if telemetry plugin disabled (#69394)

* Fix home page loading

* Fix jest test, update telemetry mocks

[Maps] Remove bounds param (#69396)

[IngestManager] fix endpoint setup in api integration tests (#69503)

[APM] Service map download in debug mode (#69350)

* [APM] Service map download in debug mode

Add a download button when debug mode is enabled that downloads JSON of
the map.

Add an upload button to the Storybook.

Embed documentation on input and output state (#69443)

* Embed documentation on input and output state

* json -> js

* Add section on how id is used by custom time range badge action to determine isCompatible

Revert "[RUM Dashboard] Initial Version (#68778)"

This reverts commit 72111702e95214594d5d56bc856e1d5c78e6d802.

[Ingest Manager] Fleet require encrypted saved object encryption key … (#69408)

[rfc][skip-ci][reporting] Rendering API RFC (#64372)

* WIP working on reporting rfc round 2

* First draft is complete

[Index template] Refactor index template wizard (#69037)

Co-authored-by: Alison Goryachev <alisonmllr20@gmail.com>

Amend export used for default (#69158)

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

[SIEM][Exceptions] - Updates exception structure and corresponding UI types (#69120)

This PR is meant to update the `ExceptionListItemSchema.entries` structure to align with the most recent conversations regarding the need for a more explicit depiction of `nested` fields. To summarize:

- Adds schema validation for requests and responses within `lists/public/exceptions/api.ts`. It was super helpful in catching existing bugs. Anyone that uses the api will run through this validation. If the client tries to send up a malformed request, the request will not be made and an error returned. If the request is successful, but somehow the response is malformed, an error is returned. There may be some UX things to figure out about how to best communicate these errors to the user, or if surfacing the raw error is fine.
- Updates `entries` structure in lists plugin api
- Updates hooks and tests within `lists/public` that make reference to new structure
- Updates and adds unit tests for updated schemas
- Removes unused temporary types in `security_solution/public/common/components/exceptions/` to now reference updated schema
- Updates UI tests
- Updates `lists/server/scripts`

[Ingest Pipelines] Add test coverage for ingest pipelines editor component (#69283)

* first iteration of CIT tests

* address pr feedback

- use dot notation where we can
- use string literals instead of + concatentation

[Ingest Pipelines] Encode URI component pipeline names (#69489)

* Properly encode URI component pipeline names

* safely URI decode to handle % case

[Console] Added license headers to worker files (#69387)

* added license headers to worker files

* add missing @notice

* update notice.txt

* fix notice

* remove start-end license block markers

* Added license pre-amble

don't include group fields with no child fields in index pattern (#69457)

Update endpoint event and alert types (#69292)

* start redoing types

* finish redoing types

* fix bad test

* rework tests

* fix more types

* fix test

* Fix endpoints test and render error

* add deletePolicyStream to alerts api tests

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Co-authored-by: Paul Tavares <paul.tavares@elastic.co>

[Security Solution] Add cypress tests for global search bar (#68535)

[RUM Dashboard] Initial version resubmit (#69531)

Co-authored-by: Casper Hübertz <casper@formgeist.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

[Ingest Manager] Use long polling for agent checkin (#68922)

Fixes home page welcome link (#69539)

[Maps] Migrate maps client router to react (#65079)

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

[CCR] Fix follower indices table not updating after pausing (#69228)

Document authentication settings. (#69284)

Co-authored-by: Kaarina Tungseth <kaarina.tungseth@elastic.co>
Co-authored-by: gchaps <33642766+gchaps@users.noreply.github.com>

[SIEM][Detection Engine] Fixes 7.8 and 7.9 upgrade issue within rules where you can get the error "params invalid: [lists]: definition for this key is missing"

* https://github.com/elastic/kibana/issues/69463
* See here for manual backport to 7.8: https://github.com/elastic/kibana/pull/69434

This fixes a bug where if you import rules and set your overwrite to `true` multiple times in a row within 7.7 you can end up with a lists array. When upgrading to 7.8, we change the name of `lists` to `exceptions_lists` and suddenly when you enable/disable a rule you can get the following error below:

![image](https://user-images.githubusercontent.com/1151048/84945824-fa60e280-b0a4-11ea-8e05-bffdec2e4765.png)

The fix is to allow the lists array still if it is present within saved objects to avoid seeing this error screen and being tolerant. We also fix the area of code that is causing the data bug so it cannot happen again with `exceptions_list` which is what the name of lists was renamed to causing this problem.

Note that this has unit tests and I also manually tested this by intentionally injecting a `lists` and `exceptions_lists` and using the UI to verify there wasn't another validation spot that needed to be relaxed to allow for the data.

- [x] [Unit or functional tests](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility) were updated or added to match the most common scenarios

[DOCS] Adds kibana-pull attribute for release docs (#69554)

skip tests using hostDetailsPolicyResponseActionBadge

Merge/restyle nodes table (#69098)

Adds panel views and drilldowns to Resolver

[ftr] add support for docker servers (#68173)

Co-authored-by: spalger <spalger@users.noreply.github.com>

Remove endpoint alert fields from signal mapping (#68934)

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

[DOCS] Add related link to the ingest management docs (#69467)

* [DOCS] Add related link to the ingest management docs

* Add link to ingest manager topic in Kibana

* Remove link to ingest manager topic in kibana

Resolving conflicts (#69597)

[ML] Data Grid Histograms (#68359)

Adds support for histogram charts to data grid columns.
- Adds a toggle button to the data grid's header to enabled/disable column charts.
- When enabled, the charts get rendered as part of the data grid header.
- Histogram charts will get rendered for fields based on date, number, string and boolean.

[APM] Add support for dark mode (#69362)

* Use theme colors for stacktrace

* [APM] Use theme for all components

* Switch HoC withTheme to useTheme and convert classes to hooks

* Change hardcoded white to euiColorEmptyShade for ServiceMap

* Snapshots and Legends fix

* Switch to context and add test helper

* Fix tests and update snaps

* New snaps + new anomaly detection

* Remove shallow from testHelpers

* Remove commented tests

* Fix prettier

* Pass correct theme to cytoscape

* Fix ServiceMap

* fixes some rendering issues in service maps

* removed the old anomaly detection logic from service map popover contents, since it's been moved to a new component.

* Fix eslint, tsc lint issues and unit tests

* Remove types for styled-components default theme

* Update x-pack/plugins/apm/public/components/shared/KueryBar/Typeahead/Suggestions.js

Co-authored-by: Casper Hübertz <casper@formgeist.com>

* fix OuterTheme

* Ise function declaration instead of expression

Co-authored-by: Balthazar Gronon <git@balthazar.dev>
Co-authored-by: Balthazar Gronon <bgronon@gmail.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Co-authored-by: Oliver Gupte <olivergupte@gmail.com>
Co-authored-by: Casper Hübertz <casper@formgeist.com>

[Endpoint] add policy empty state (#69449)

Fixing resolver alert generation (#69587)

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

[Security_Solution][Endpoint] Resolver leverage ancestry array for queries  (#69264)

* Adding alerts route

* Adding related alerts generator changes, tests, and script updates

* Fixing missed parameter

* Aligning the AlertEvent and ResolverEvent definition

* Fixing type errors

* Fixing import error

* Adding ancestry functionality in generator

* Creating some tests for ancestry field

* Making progress on the ancestry

* Fixing the ancestry verification

* Fixing existing tests

* Removing unused code and fixing test

* Adding more comments

* Fixing endgame queries

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

skip failing suite (#69595)

[Endpoint] Fix flaky endpoints list unit test (#69591)

* Fix flaky endpoints list unit test
* un-skip test

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

disable pageLoadMetrics job, it's gotten really flaky

[SECURITY] Introduce kibana nav (#68862)

* Change the bootstrap of the app

* rename SiemPageName to SecurityPageName

* modify alerts routes

* modify cases routes

* modify hosts routes

* modify network routes

* modify overview routes

* modify timelines routes

* wip change management route

* change route for common

* some fixing from the first commit

* modify route for management

* update url format hook to use history

* bug when you click on external alerts from host or network

* improvement from josh feedback

* redirect siem to security solution

* a little clean up

* Fix types

* fix breadcrumbs

* fix unit test

* Update index.tsx

* Fix cypress

* bug remove timeline when you are in case configure

* Fix functionel test for management

* Fix redirect siem + ml

* fixes some cypress tests

* adds 'URL compatibility' test

* bring ml back to alerts

* review I

* Fix memory leak in timelines page

* fix storage bug for timeline search bar

* fix endpoint merge + functional test

* avoid timeline flyout toggle

* Fix link to ml score

* Fix breadcrumb

* Fix management url

* fix unit test

* fixes typecheck issue

* fixes remaining url cypress tests

* fixes timeline scenario

* fix link to details rule from timeline

* review remove absolute path for consistency

* Fixing resolver alert generation (#69587)

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

* [Security_Solution][Endpoint] Resolver leverage ancestry array for queries  (#69264)

* Adding alerts route

* Adding related alerts generator changes, tests, and script updates

* Fixing missed parameter

* Aligning the AlertEvent and ResolverEvent definition

* Fixing type errors

* Fixing import error

* Adding ancestry functionality in generator

* Creating some tests for ancestry field

* Making progress on the ancestry

* Fixing the ancestry verification

* Fixing existing tests

* Removing unused code and fixing test

* Adding more comments

* Fixing endgame queries

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

* fix cypress test

* skip failing suite (#69595)

* [Endpoint] Fix flaky endpoints list unit test (#69591)

* Fix flaky endpoints list unit test
* un-skip test

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

* remove flaky test

Co-authored-by: patrykkopycinski <contact@patrykkopycinski.com>
Co-authored-by: Gloria Hornero <snootchie.boochies@gmail.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Co-authored-by: Jonathan Buttner <56361221+jonathan-buttner@users.noreply.github.com>
Co-authored-by: spalger <spalger@users.noreply.github.com>
Co-authored-by: Paul Tavares <56442535+paul-tavares@users.noreply.github.com>

Bump jest related packages (#58095)

[IM] Move template wizard steps to shared directory + update legacy details panel (#69559)

Migrate dashboard mode (#69305)

Closes: #67469

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

[Observability]Adding specific Return APIs for each plugin (#69257)

* Adding specific apis for each plugin

* adding metric hosts stat

* addressing PR comment

* addressing PR comments

* changing series to key/value

* exporting interfaces

* adding label to stat

* refactoring types

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

[ML] fix ml api services (#69681)

[Discover] Unskip context navigation functional test (#68771)

[APM] Use asPercent to format breakdown chart (#69384)

Co-authored-by: Søren Louv-Jansen <sorenlouv@gmail.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

Updates swim lane UI text (#69454)

Fix Advanced Settings Panel number editing in Graph (#69672)

Remove App communication from URL (#67064)

Removed all inter-app communication via url in favour of a new service in the embeddable start contract called the state transfer service.

[Security_Solution] Split up indices (#69589)

* Fixing resolver alert generation

* Splitting indices up

* Removing tests that could randomly fail because of the generation code

* Adding support for multiple indices

* Updating archives with the new index names

* Removing alerts data stream

* Switching to process instead of fake

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

completes top-level navigation tests (#69694)

[CI] Record Github commit statuses outside of PRs (#69432)

Document workaround for using HashRouter (#69140)

* add section on hashrouter

* review comments

Disabled multiple select for preconfigured connectors to avoid requesting bulk delete on them (#69459)

[chore] TS 3.9: convert ts-ignore to ts-expect-error (#69541)

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

[Monitoring] Love for APM (#69052)

* Fix broken colors for APM

* Use default derivative

* Fix UI issues with APM

* Add new charts

* Fix tests

* Use EUI color palette

* Remove old translations

* PR feedback

* Fix tests

* Fix up overview page

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

[ML] Transform: Table enhancements (#69307)

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

[ML] Transform: Enable force delete if one of the transforms failed (#69472)

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

fix link to analytics results from management (#69550)

adds kibana navigation tests (#69733)

Add plugin API for customizing the logging configuration (#68704)

[Maps] Remove extra layer of telemetry nesting under "attributes" (#66137)

* Return attributes when telemetry created instead of whole saved object. Update integration test

* Change 'maps-telemetry' to 'maps'

* No need to create a saved object anymore. This is leftover from task manager telemetry mgmt

* Add test confirming attrs undefined. Change tests to check for 'maps' iso 'maps-telemetry'

* Add two more tests confirming expected telemetry shape

* Review feedback. Use TELEMETRY_TYPE constant and set to APP_ID

[Index Management] Fix API Integration Test and use of `timestamp_field` (#69666)

* fix types and functional api integration test

* access timestamp field name in object

* temporarily skip the API integration test and fix ts issue

[SECURITY] Add endpoint alerts url (#69707)

* Add back endpoint alerts url

* hack to move on

* fix type

* fix test

remove scroll in drag & drop context (#69710)

[Metrics UI] Add inventory alert preview (#68909)

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

[Obs] Update Observability landing page text (#69727)

Support deep links inside of `RelayState` for SAML IdP initiated login. (#69401)

[ML] Fixes anomaly chart and validation for one week bucket span (#69671)

* [ML] Fixes anomaly chart and validation for one week bucket span

* [ML] Fix interval Jest tests

Migrate legacy import/export endpoints (#69474)

* migrate legacy export routes to `legacy_export` plugin

* adapt unit tests

* remove already dead (already moved) libs

Fixes #69344: Don't allow empty string for server.basePath config (#69377)

* Fixes #69344: Don't allow empty string for server.basePath config

* Remove unused basepath group

Redirect to Logged Out UI on SAML Logout Response. Prefer Login Selector UI to Logged Out UI whenever possible. (#69676)

Properly redirect legacy URLs (#68284)

expose DocLinks API from start only (#68745)

* exose DocLinks API from start only

* update docs

* fix type errors

* update docs

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

remove top-nav. it is not rendered in KP (#69488)

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

[APM] Storybook theme fixes (#69730)

* [APM] Storybook theme fixes

The changes adding theme support in #69362 broke some of the Storybook stories.

Add decorators to wrap some of the stories in the theme context.

This should be done in a global decorator, but our current storybook setup doesn't support this. It also would be nice to be able to switch between light/dark mode, but that's something we can add in the future.

* Remove unused import

* Adds missing decorator to cytoscape examples + adds a new real-world example

Co-authored-by: Oliver Gupte <olivergupte@gmail.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

adding Stats interface with type (#69784)

[data.search.aggs]: Add AggConfig.toSerializedFieldFormat (#69114)

[Ingest Manager] Do not await in start. Return a Promise (#69505)

  1. Do not `await` in the public `start` lifecycle method. Fixes https://github.com/elastic/kibana/issues/66125
PR based on https://github.com/elastic/kibana/issues/66125#issuecomment-640790837 & https://github.com/elastic/kibana/issues/66125#issuecomment-642218799
  2. Change `success` to be Promise

[pre-req] Convert Palettes and Components (#69065)

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

[Lens] Stabilize filter popover (#69519)

* stabilize filter popovwer

* remove text exclusion

Add lists plugin to optimized security_solution TS config (#69705)

As security_solution continues to integrate with lists, the absents of
these types will lead to lots of implicit anys and false positives.

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

[data.search.aggs] Move agg-specific field formats to search service (#69586)

[Security][Network] Exclude glob-only (*) Index Pattern from map layers (#69736)

* Exclude glob-only (*) index pattern from map layers

This pattern is a special case that our map should ignore, as including
it causes all indexes to be queried.

* Ignore CCS glob pattern in our embedded map

Users may have this pattern for cross-cluster search, and it should
similarly be excluded when matching Security indexes.

[IM] Move common step containers to shared (#69713)

[Maps] New mappings: maps-telemetry -> maps (#69816)

Don't set a min-length on encryption key for reportin (#69827)

[ML] DF Analytics Creation: add progress indicator (#69583)

* add progress indicator to creation wizard page

* only show progress bar if job is started immediately

* add title and switch to timeout

* fix progress check

* clean up interval on unmount

* fix types

* clear interval if stats undefined. show progress if job created

[Uptime] Fix charts dark theme (#69748)

[Ingest Manager] Support registration of server side callbacks for Create Datasource API (#69428)

* Ingest: Expose `registerExternalCallback()` method out of Ingest server `start` lifecycle
* Ingest: Add support for External Callbacks on REST `createDatasourceHandler()`
* Ingest: expose DatasourceServices to Plugin start interface
* Endpoint: Added Endpoint Ingest handler for Create Datasources
  - Also moved the temporary logic from the middleware
    to the handler (still temporary)

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

[IngestManager] Expose agent authentication using access key (#69650)

* [IngestManager] Expose agent authentication using access key

* Add unit tests to authenticateAgentWithAccessToken service

[APM] Pulls legacy ML code from service maps and integrations (#69779)

* Pulls out existing ML integration from the service maps

* - removes ML job creation flyout in integrations menu on the service details UI
- removes ML searches and transforms in the transaction charts API
- removes unused shared functions and types related to the legacy ML integration

* removes unused translations for APM anomaly detection

* Adds tags to TODOs for easy searching later

Convert Positionable, RenderToDom and RenderWithFn to functional/hooks/no recompose. (#68202)

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
release_note:skip Skip the PR/issue when compiling release notes Team:Kibana Management Dev Tools, Index Management, Upgrade Assistant, ILM, Ingest Node Pipelines, and more
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants