[Fleet] Agent logs UI #83356

jen-huang · 2020-11-12T22:50:40Z

Summary

Resolves #77189

Added new agent logs UI as Logs tab in agent details page that shows logs for that agent; the previous Activity log tab and related code was removed
The new logs UI uses Infra's <LogStream> component, controlled by a query bar, dropdown filters, and date picker
- By default when user enters this UI, they will see all logs from the past 1 day from elastic_agent dataset
- User can filter on any number of elastic_agent.* datasets to view logs from, the options are populated from existing logs data
- User can filter on any number of log levels, the options are populated from existing logs data
- The log entries shown does not update automatically with new data, the user will need to change their queries/filters/date range to update the entries, or click Apply in the Quick select interface of the date picker
- When user clicks View in Logs, they will be taken to Logs UI with the current state of their view, that is: current agent ID + selected datasets + selected log levels + date picker range

Development notes

Restructured plugin start dependencies to consolidate them, rename useCore to useStartServices and replace usage of useStartDeps (removed useSetupDeps that was never used)
Added additional flexbox styling to app layout so that page content takes up the full height between header and beta messaging, this was done so that the LogStream component can automatically expand to the available height
Moved some local APM typings to top-level x-pack/typings folder to resolve type check errors, the local APM module declarations are unreadable during x-pack/test type check due to some messy dependency trees 😭
- cytoscape_dagre.d.ts and react_vis.d.ts were moved
- x-pack/test/plugin_functional/plugins/resolver_test imports from -> x-pack/plugins/security_solution -> x-pack/plugins/fleet/public -> x-pack/plugins/infra/public (added in this PR) -> x-pack/plugins/infra/public
- Attempting to install @types/react-vis failed with Unsupported URL Type "link:": link:../elastic-safer-lodash-set and @types/cytoscape-dagre doesn't exist, so moving the declaration files seemed to be the cleanest solution

Follow up work

[Fleet] Update agent log query bar with new props #83735 Update agent log query bar with new props
[Fleet] Add URL state to agent logs UI #83736 Add URL state to agent logs UI

Screenshots

jen-huang · 2020-11-16T19:14:22Z

@elasticmachine merge upstream

ph · 2020-11-16T19:17:25Z

@jen-huang You moved it to review, but it still in draft?

jen-huang · 2020-11-16T19:21:07Z

@ph Yes, this PR is ready for review but it's in draft to avoid triggering codeowners for changes to /infra files that came from merging an outstanding LogStream bug fix (linked in description).

ph · 2020-11-16T19:30:14Z

@EricDavisX @kevinlog FYI this should include endpoint log too.

EricDavisX · 2020-11-16T23:03:11Z

@jen-huang is it too late to make any suggestions while you're in the code? It would be a slight step back for users who have very little Agent activity going on to have to select something to see the last activity (if it wasn't within the last 15 minutes) - could we show some special default 'hybrid' mode where we show up to 10 items or the last 15 minutes if there are more than 10? I know I may be asking you to change a UI component, lol, just citing a common use case that may have additional clicks in it now.

Also, would love to do an 'automation and feature coverage' review with you! We are finalizing a process guideline, I will send you the draft and we can chat it. Thank you.

jen-huang · 2020-11-16T23:17:36Z

@EricDavisX Unfortunately the <LogStream> components requires finite start and end timestamps to load log entries from. Note that as long as the agent is connected to Fleet, we'll always get FleetGateway entries periodically by default. The quick select option of the date picker also makes it very easy to select a larger range:

I think it's worth discussing what a good default date range is, though. That is easy to change, I think Logs UI actually uses last 1 day by default. WDYT? cc @hbharding

EricDavisX · 2020-11-17T13:18:23Z

I think 'last 1 Day' would suffice, thanks for the discourse

...plications/fleet/sections/agents/agent_details_page/components/agent_logs/filter_dataset.tsx

…eStartServices and replace usage of useStartDeps

elasticmachine · 2020-11-18T22:42:56Z

Pinging @elastic/ingest-management (Feature:Fleet)

smith

APM changes look good.

jasonrhodes · 2020-11-19T00:49:36Z

I'm not reviewing this but I just wanted to say that this looks amazing and it makes me very happy. :D

…tream

nchaulet · 2020-11-19T18:05:13Z

I am curious to know a little more on the refactoring of useCore to useStartService what are the reasons behind?

jen-huang · 2020-11-19T18:32:34Z

@nchaulet Good question, here are my reasons:

Previously, we had useCore, useStartDeps, useSetupDeps. These are locked down to Kibana core, and Fleet's start and setup dependency plugins. For this PR, I needed to introduce a storage service that the data plugin component assumes is part of the KibanaContextProvider. This storage service is not part of Kibana core, nor is it an external plugin dependency. Combining the three into a single useStartServices allows us to instantiate services like these into our app context.
I think it's not very important to differentiate between a service from core/start/setup from a dev view, we just want to get an outside service from "somewhere". Combining everything into useStartServices removes that overhead.
While I was looking at how to get storage into the app context, I saw that this pattern of combining core/start/all other services is being used by other apps now: security solution example, visualize example

x-pack/plugins/fleet/public/applications/fleet/constants/page_paths.ts

...public/applications/fleet/sections/agents/agent_details_page/components/agent_logs/index.tsx

nchaulet

A few comments but nothing really blocking, I tested locally working really well 🚀

…tream

kibanamachine · 2020-11-20T03:06:30Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: b049374

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
`fleet`	418	433	+15

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`fleet`	1.1MB	1.1MB	+23.0KB

Distributable file count

id	before	after	diff
`default`	42942	42943	+1

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
`fleet`	373.0KB	374.6KB	+1.6KB

History

💚 Build #89029 succeeded 1933a49
💔 Build #89011 failed d2ccc1a
💚 Build #88781 succeeded c3325e7
💔 Build #88115 failed 062083f2811b3496f9bb1825be57373e15502aed
💔 Build #87788 failed 6314c63c1f5c1d0d4d2c6293b0cc8c4de75f2437

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

* Initial agent log table * Add data and storage services to Kibana context, rename useCore to useStartServices and replace usage of useStartDeps * Initial attempt at adding query bar for log stream * Adjust app layout to allow page content to be full height * Dataset and log level filters, split into smaller files * Use data plugin's `QueryStringInput` component for query bar * Add open in Logs UI link * Clean up unused storage dependency * Remove agent activity log and related dead code * Clean up i18n * Clean up plugin deps, fix routing * Add back storage dependency that data plugin components need * Remove dependency on infra for logs UI link * Change default date range to last one day * Adjust panel padding * Add comment * Move module declarations to top-level x-pack/typings * Fix missed renaming * Remove unused path * Move building of log stream query into separate service with unit tests * Adjust test conditions

jen-huang self-assigned this Nov 12, 2020

jen-huang changed the title ~~[Fleet] Agent logs UI using~~ [Fleet] Agent logs UI Nov 12, 2020

jen-huang added Feature:Fleet Fleet team's agent central management project release_note:enhancement Team:Fleet Team label for Observability Data Collection Fleet team v7.11.0 v8.0.0 labels Nov 12, 2020

EricDavisX mentioned this pull request Nov 17, 2020

[Fleet] Testing over Agent collection of Agent & Endpoint log files elastic/beats#22549

Closed

16 tasks

hbharding mentioned this pull request Nov 17, 2020

[Fleet] Define UX design for information that is displayed when the user clicks on “Agent” to view its details #81872

Open

hbharding reviewed Nov 17, 2020

View reviewed changes

...plications/fleet/sections/agents/agent_details_page/components/agent_logs/filter_dataset.tsx Show resolved Hide resolved

jen-huang mentioned this pull request Nov 18, 2020

Add iconType and disableLanguageSwitcher options to QueryStringInput #83700

Merged

1 task

jen-huang added 12 commits November 18, 2020 11:49

Initial agent log table

6e15008

Add data and storage services to Kibana context, rename useCore to us…

5142fc2

…eStartServices and replace usage of useStartDeps

Initial attempt at adding query bar for log stream

be25ca0

Adjust app layout to allow page content to be full height

d3072e2

Dataset and log level filters, split into smaller files

8ddee9b

Use data plugin's QueryStringInput component for query bar

986728b

Add open in Logs UI link

d72ea46

Clean up unused storage dependency

c035af4

Remove agent activity log and related dead code

b8676e1

Clean up i18n

a65c3d9

Clean up plugin deps, fix routing

d477970

Add back storage dependency that data plugin components need

02e2b8a

jen-huang force-pushed the ingest/agent-logstream branch from 062083f to c3325e7 Compare November 18, 2020 22:26

This was referenced Nov 18, 2020

[Fleet] Update agent log query bar with new props #83735

Closed

[Fleet] Add URL state to agent logs UI #83736

Closed

jen-huang marked this pull request as ready for review November 18, 2020 22:42

jen-huang requested a review from a team as a code owner November 18, 2020 22:42

jen-huang requested a review from a team November 18, 2020 22:42

smith approved these changes Nov 18, 2020

View reviewed changes

jasonrhodes added the Epic:Shared Logs Component label Nov 19, 2020

Merge remote-tracking branch 'upstream/master' into ingest/agent-logs…

d2ccc1a

…tream

Fix missed renaming

1933a49

nchaulet reviewed Nov 19, 2020

View reviewed changes

x-pack/plugins/fleet/public/applications/fleet/constants/page_paths.ts Outdated Show resolved Hide resolved

nchaulet reviewed Nov 19, 2020

View reviewed changes

...public/applications/fleet/sections/agents/agent_details_page/components/agent_logs/index.tsx Outdated Show resolved Hide resolved

nchaulet approved these changes Nov 19, 2020

View reviewed changes

jen-huang added 4 commits November 19, 2020 16:49

Merge remote-tracking branch 'upstream/master' into ingest/agent-logs…

bf7699b

…tream

Remove unused path

07cdd9d

Move building of log stream query into separate service with unit tests

0889437

Adjust test conditions

b049374

nchaulet mentioned this pull request Nov 20, 2020

[Fleet] Allow to send SETTINGS action #83707

Merged

jen-huang merged commit a8913d3 into elastic:master Nov 20, 2020

jen-huang deleted the ingest/agent-logstream branch November 20, 2020 18:49

jen-huang mentioned this pull request Nov 20, 2020

[7.x] [Fleet] Agent logs UI (#83356) #83961

Merged

jen-huang mentioned this pull request Nov 23, 2020

[Fleet] Use new QueryStringInput props #84108

Merged

EricDavisX mentioned this pull request Nov 23, 2020

[Fleet] Testing over Ability to Set Agent Log Level in Kibana Fleet UI #84119

Closed

12 tasks

amolnater-qasource mentioned this pull request Jan 27, 2021

[Fleet] Show dates filter for "Future Timestamps" under agent Logs tab on navigating to Logs Stream changes to default value "Last 1 day". #89393

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Fleet] Agent logs UI #83356

[Fleet] Agent logs UI #83356

jen-huang commented Nov 12, 2020 •

edited

Loading

jen-huang commented Nov 16, 2020

ph commented Nov 16, 2020

jen-huang commented Nov 16, 2020 •

edited

Loading

ph commented Nov 16, 2020

EricDavisX commented Nov 16, 2020

jen-huang commented Nov 16, 2020

EricDavisX commented Nov 17, 2020

elasticmachine commented Nov 18, 2020

smith left a comment

jasonrhodes commented Nov 19, 2020

nchaulet commented Nov 19, 2020

jen-huang commented Nov 19, 2020 •

edited

Loading

nchaulet left a comment

kibanamachine commented Nov 20, 2020

[Fleet] Agent logs UI #83356

[Fleet] Agent logs UI #83356

Conversation

jen-huang commented Nov 12, 2020 • edited Loading

Summary

Development notes

Follow up work

Screenshots

jen-huang commented Nov 16, 2020

ph commented Nov 16, 2020

jen-huang commented Nov 16, 2020 • edited Loading

ph commented Nov 16, 2020

EricDavisX commented Nov 16, 2020

jen-huang commented Nov 16, 2020

EricDavisX commented Nov 17, 2020

elasticmachine commented Nov 18, 2020

smith left a comment

Choose a reason for hiding this comment

jasonrhodes commented Nov 19, 2020

nchaulet commented Nov 19, 2020

jen-huang commented Nov 19, 2020 • edited Loading

nchaulet left a comment

Choose a reason for hiding this comment

kibanamachine commented Nov 20, 2020

💚 Build Succeeded

Metrics [docs]

Module Count

Async chunks

Distributable file count

Page load bundle

History

jen-huang commented Nov 12, 2020 •

edited

Loading

jen-huang commented Nov 16, 2020 •

edited

Loading

jen-huang commented Nov 19, 2020 •

edited

Loading