[Fleet] Add custom integrations API #112481

thomasneirynck · 2021-09-16T20:12:25Z

Part of #93084

Summary

internal changes:

adds new plugin custom_integrations. This plugin can be used by others to register integrations that need to show up on the fleet-integration UX app
register tutorials inside home plugin
typing changes. Some types are moving into custom_integrations. Others became more generic to accomodate non-epr integration-types.

user-facing changes:

adds new categories and cards to the fleet-integration app. Since it's only registering tutorials, this is just the EMS-card for now. The APM card tutorial remains hidden. APM will need similar replacement behavior, just like Beats

Checklist

Delete any items that are not applicable to this PR.

Unit or functional tests were updated or added to match the most common scenarios

For maintainers

This was checked for breaking API changes and was labeled appropriately

…tom_categories

kibanamachine · 2021-09-21T22:16:43Z

⏳ Build in-progress, with failures

continuous-integration/kibana-ci/pull-request
Commit: fc0ed69
Storybooks not built yet
This comment will update when the build is complete

Failed CI Steps

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

…tom_categories

joshdover

Did a first pass look at the basic structure and I generally think the approach makes sense. This will definitely be cleaner once we can move the merging logic to the server-side and provide a single API endpoint that returns the merged and normalized data.

src/plugins/custom_integrations/server/custom_integration_registry.ts

x-pack/plugins/fleet/public/applications/integrations/sections/epm/screens/home/index.tsx

src/plugins/custom_integrations/common/index.ts

src/plugins/custom_integrations/kibana.json

src/plugins/custom_integrations/server/types.ts

src/plugins/home/server/services/tutorials/tutorials_registry.ts

src/plugins/home/server/services/tutorials/lib/tutorial_schema.ts

src/plugins/custom_integrations/public/index.ts

elasticmachine · 2021-09-22T15:23:33Z

Pinging @elastic/fleet (Team:Fleet)

kpollich

Took a look here and went through @joshdover's previous comments. I don't have anything much to add of concern other than what he already mentioned and what was resolved there.

I think the typing is a bit of a lingering concern, but I'm comfortable with what we have set up for IntegrationCard objects, etc here in this PR.

I pulled this PR down locally and verified the functionality of the EMS tutorial integration card. Things look good!

Thanks for all your work on this! 🚀

...gins/fleet/public/applications/integrations/sections/epm/components/package_card.stories.tsx

…tom_categories

TinaHeiligers · 2021-09-23T20:44:16Z

src/plugins/custom_integrations/README.md

@@ -0,0 +1,9 @@
+# customIntegrations


The docs will need fleshing out and conversion to .mdx if we want to include them in the new dev docs

thx. my preference is to hold off on dev-docs in this PR. These will evolve fairly quickly. Would prefer to introduce only in separate PR when contract stabilizes.

pgayvallet

Some NITs and comments, overall LGTM

src/plugins/home/kibana.json

src/plugins/home/server/plugin.ts

src/plugins/home/server/services/tutorials/tutorials_registry.ts

TinaHeiligers

The approach looks ok to me although the new custom_integrations plugin is sorely missing unit tests.

I know it seems like a harmless enough plugin but we need to make sure it gives new users a great first impression!

IMO, we should also add support for localization as soon as possible. 7.16 will be around for a while.

src/plugins/custom_integrations/common/index.ts

src/plugins/home/server/plugin.ts

TinaHeiligers

LGTM

…tom_categories

thomasneirynck · 2021-09-27T13:43:06Z

@TinaHeiligers thx for the review! I added unit test coverage and a new api integration test for the new plugin.

kibanamachine · 2021-09-27T15:47:50Z

💚 Build Succeeded

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
`customIntegrations`	-	6	+6
`fleet`	569	574	+5
total			+11

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
`customIntegrations`	-	66	+66
`fleet`	1085	1095	+10
total			+76

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`fleet`	595.4KB	596.7KB	+1.3KB

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
`customIntegrations`	-	2.4KB	+2.4KB
`fleet`	130.7KB	132.4KB	+1.7KB
total			+4.1KB

Unknown metric groups

API count

id	before	after	diff
`customIntegrations`	-	66	+66
`fleet`	1186	1196	+10
total			+76

History

💚 Build #155584 succeeded 6d63a02
💔 Build #155519 failed d73f38d
💚 Build #155454 succeeded e96bc0b
💚 Build #155112 succeeded 77a571f
💔 Build #155091 failed c05ff99

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

kibanamachine · 2021-09-27T15:57:59Z

💔 Backport failed

Status	Branch	Result
❌	7.x	Commit could not be cherrypicked due to conflicts

To backport manually run:
node scripts/backport --pr 112481

Add a new plugin `custom_integrations`. This plugin allows for the registration of data-integrations tutorials. The Fleet-integrations app will display these alongside the existing Elastic Agent integrations. # Conflicts: # packages/kbn-optimizer/limits.yml

dikshachauhan-qasource · 2021-09-30T09:09:19Z

Hi @thomasneirynck

As the merges are now available for above PR, is it possible that we can have mocks for user-facing changes.

Also, it will be great if you could provide us more info on how to validate above.

Currently, attempted to look out changes on latest 8.0 snapshot build available on cloud-qa platform, however couldn't find as described in ticket.

Further, we observed an inactive link for custom inputs under integration tab. So we have reported issue [8.0]: Custom inputs link under integration tab is inactive..

cc @kpollich @EricDavisX

Thanks
QAS

EricDavisX · 2021-09-30T13:34:56Z

@dikshachauhan-qasource since it is closed, I would also follow up with an email, to the author after a day if there is no response. it would be good to confirm if we have mocks, but we may not. :) thanks for testing and poking it.

thomasneirynck · 2021-09-30T13:57:03Z

@dikshachauhan-qasource these are primarly internal changes. The only user-facing change is the additiong of the EMS-tutorial as a new card.

So from the pointers at the screenshot here, #112481 (comment), that's not related to this change. Doing a custom-search, should behave exactly as on 7.15. If there are bugs there (e.g. inactive link). Can you log as a new issue? thx!

dikshachauhan-qasource · 2021-10-01T10:14:44Z

Hi @thomasneirynck

Thanks for the feedback.

We have validated above related issue on shipped 7.15 build and found it working fine.

Further, as per ticket, Please assign all the related tickets that will be coming under this feature change, so that we can record them and validate at our end.

Thanks
QAS

thomasneirynck added 3 commits September 16, 2021 16:10

initial boilerplate

b87be67

setup dependency in fleet

5ca5307

add custom cards to UX (boilerplate)

c6175ad

thomasneirynck mentioned this pull request Sep 20, 2021

[Fleet] Unified Integrations view #93084

Closed

63 tasks

thomasneirynck added 6 commits September 21, 2021 09:49

remove cruft

46a67e5

isolate epr from cards

3844e8a

make cards generic

a40cee6

remove epr-typing from cards

950a9a2

remove epr tpying from card-lists

4bc93e0

more typing

5c4258e

thomasneirynck requested review from joshdover and kpollich September 21, 2021 19:46

thomasneirynck added chore release_note:skip Skip the PR/issue when compiling release notes Team:Fleet Team label for Observability Data Collection Fleet team v7.16.0 v8.0.0 labels Sep 21, 2021

Merge branch 'master' of github.com:elastic/kibana into fleet/add_cus…

4d0a763

…tom_categories

thomasneirynck changed the title ~~[Fleet] Add custom categories~~ [Fleet] Add custom integrations Sep 21, 2021

thomasneirynck added 2 commits September 21, 2021 17:34

more type fixes

df06051

Fix icon issue

fc0ed69

Merge branch 'master' of github.com:elastic/kibana into fleet/add_cus…

9f217dd

…tom_categories

joshdover reviewed Sep 22, 2021

View reviewed changes

thomasneirynck marked this pull request as ready for review September 22, 2021 15:23

thomasneirynck requested review from a team as code owners September 22, 2021 15:23

thomasneirynck mentioned this pull request Sep 23, 2021

Add Elasticsearch Clients as non-Elastic Agent integrations (MVP) #109281

Closed

kpollich approved these changes Sep 23, 2021

View reviewed changes

...gins/fleet/public/applications/integrations/sections/epm/components/package_card.stories.tsx Show resolved Hide resolved

thomasneirynck added 3 commits September 23, 2021 14:15

move badge-labels back inside card-component

e1e4c63

remove unused

cfc5c4d

Merge branch 'master' of github.com:elastic/kibana into fleet/add_cus…

e96bc0b

…tom_categories

TinaHeiligers reviewed Sep 23, 2021

View reviewed changes

pgayvallet approved these changes Sep 23, 2021

View reviewed changes

TinaHeiligers reviewed Sep 23, 2021

View reviewed changes

src/plugins/custom_integrations/common/index.ts Outdated Show resolved Hide resolved

src/plugins/home/server/plugin.ts Outdated Show resolved Hide resolved

src/plugins/home/server/plugin.ts Outdated Show resolved Hide resolved

thomasneirynck added 3 commits September 23, 2021 17:47

feedback

29e7028

remove cruft

e9e769e

add jest boilerplate

d73f38d

TinaHeiligers self-requested a review September 23, 2021 23:58

TinaHeiligers approved these changes Sep 23, 2021

View reviewed changes

thomasneirynck added 3 commits September 24, 2021 16:52

Fill in test coverage

7fddfb7

Merge branch 'master' of github.com:elastic/kibana into fleet/add_cus…

6d63a02

…tom_categories

Merge branch 'master' of github.com:elastic/kibana into fleet/add_cus…

0450645

…tom_categories

thomasneirynck merged commit be1ee57 into elastic:master Sep 27, 2021

thomasneirynck added the auto-backport Deprecated - use backport:version if exact versions are needed label Sep 27, 2021

thomasneirynck mentioned this pull request Sep 27, 2021

[7.x] [Fleet] Add custom integrations API (#112481) #113136

Merged

EricDavisX mentioned this pull request Sep 30, 2021

[8.0]: Custom inputs link under integration tab is inactive. #113508

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Fleet] Add custom integrations API #112481

[Fleet] Add custom integrations API #112481

thomasneirynck commented Sep 16, 2021 •

edited

Loading

kibanamachine commented Sep 21, 2021

joshdover left a comment

elasticmachine commented Sep 22, 2021

kpollich left a comment

TinaHeiligers Sep 23, 2021

thomasneirynck Sep 23, 2021

pgayvallet left a comment

TinaHeiligers left a comment

TinaHeiligers left a comment

thomasneirynck commented Sep 27, 2021 •

edited

Loading

kibanamachine commented Sep 27, 2021

API count

kibanamachine commented Sep 27, 2021

dikshachauhan-qasource commented Sep 30, 2021

EricDavisX commented Sep 30, 2021

thomasneirynck commented Sep 30, 2021

dikshachauhan-qasource commented Oct 1, 2021

[Fleet] Add custom integrations API #112481

[Fleet] Add custom integrations API #112481

Conversation

thomasneirynck commented Sep 16, 2021 • edited Loading

Summary

Checklist

For maintainers

kibanamachine commented Sep 21, 2021

⏳ Build in-progress, with failures

Failed CI Steps

History

joshdover left a comment

Choose a reason for hiding this comment

elasticmachine commented Sep 22, 2021

kpollich left a comment

Choose a reason for hiding this comment

TinaHeiligers Sep 23, 2021

Choose a reason for hiding this comment

thomasneirynck Sep 23, 2021

Choose a reason for hiding this comment

pgayvallet left a comment

Choose a reason for hiding this comment

TinaHeiligers left a comment

Choose a reason for hiding this comment

TinaHeiligers left a comment

Choose a reason for hiding this comment

thomasneirynck commented Sep 27, 2021 • edited Loading

kibanamachine commented Sep 27, 2021

💚 Build Succeeded

Metrics [docs]

Module Count

Public APIs missing comments

Async chunks

Page load bundle

API count

History

kibanamachine commented Sep 27, 2021

💔 Backport failed

dikshachauhan-qasource commented Sep 30, 2021

EricDavisX commented Sep 30, 2021

thomasneirynck commented Sep 30, 2021

dikshachauhan-qasource commented Oct 1, 2021

thomasneirynck commented Sep 16, 2021 •

edited

Loading

thomasneirynck commented Sep 27, 2021 •

edited

Loading