Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ML] Explain Log Rate Spikes: Fix grouping edge cases. #140891

Merged
merged 18 commits into from
Sep 20, 2022

Conversation

walterra
Copy link
Contributor

@walterra walterra commented Sep 17, 2022

Summary

Part of #138117.
Follow up to #140814.
Fixes #140848.

  • Change point groups might miss individual change points that were not returned by the frequent_items agg as part of groups. This PR now adds each missing one as an individual additional group.
  • Only return groups if there's at least one group with more than one item, otherwise fall back to basic table with significant terms.
  • Changes the UI behaviour to show the regular table by default and the grouping switch set to off.
  • Adds p-value column to grouped table and defaults to sorting by that column similar to table with indidivual items.

Checklist

@walterra walterra added :ml release_note:skip Skip the PR/issue when compiling release notes v8.5.0 Feature:ML/AIOps ML AIOps features: Change Point Detection, Log Pattern Analysis, Log Rate Analysis labels Sep 17, 2022
@walterra walterra self-assigned this Sep 17, 2022
@walterra walterra added the ci:cloud-redeploy Always create a new Cloud deployment label Sep 17, 2022
@walterra walterra marked this pull request as ready for review September 19, 2022 09:50
@walterra walterra requested a review from a team as a code owner September 19, 2022 09:50
@elasticmachine
Copy link
Contributor

Pinging @elastic/ml-ui (:ml)

@walterra walterra added the bug Fixes for quality problems that affect the customer experience label Sep 19, 2022
(cp) => cp.fieldName === field && cp.fieldValue === value
);
if (stIndex === -1) {
delete currentItems[field];
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Deleting items while in the middle of looping over them can be dangerous. I appreciate there is no counter or index being used here, but perhaps adding them into a new set would be cleaner?
Something like:

fi.set = {};
Object.entries(currentItems).forEach(([field, value]) => {
  if (
    changePoints.find((cp) => cp.fieldName === field && cp.fieldValue === value) !==
    undefined
  ) {
    fi.set[field] = value;
  }
});
fi.size = Object.keys(fi.set).length;

But feel free to ignore if you think this is risk free.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point! I refactored the code to work the other way around, it nows builds a new set based on the field/values that match. Update in 684ef1c.

});
});
return p;
}, {} as FieldValuePairCounts);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this assertion isn't needed if you supply reduce with the type reduce<FieldValuePairCounts>(...

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 89767d7.

cpg.group.forEach((g) => {
const str = `${g.fieldName}$$$$${g.fieldValue}`;
fieldValuePairCounts[str] = fieldValuePairCounts[str] ? fieldValuePairCounts[str] + 1 : 1;
if (p[g.fieldName] === undefined) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

real nit pick, this code might be nicer to read if you expand g out to { fieldName, fieldValue }

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 4d06d86.

</>
</EuiToolTip>
),
render: (pValue: number | null) => pValue?.toPrecision(3) ?? NOT_AVAILABLE,
Copy link
Member

@jgowdyelastic jgowdyelastic Sep 20, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit, could this function be moved to a common utils area to reduce the duplication of the NOT_AVAILABLE string?
I see it's defined three times.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good suggestions, we'll pick this up in follow-up, there's quite some duplicate code in the components for the regular table, the grouped table and its expanded row now.

Copy link
Member

@jgowdyelastic jgowdyelastic left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added a few minor code comments, but other than that, I've tested this compared to the original behaviour. LGTM

Copy link
Contributor

@alvarezmelissa87 alvarezmelissa87 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tested and LGTM ⚡

@kibana-ci
Copy link
Collaborator

kibana-ci commented Sep 20, 2022

💚 Build Succeeded

Metrics [docs]

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id before after diff
@kbn/ml-agg-utils 39 40 +1

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id before after diff
aiops 759.5KB 760.5KB +1.0KB
Unknown metric groups

API count

id before after diff
@kbn/ml-agg-utils 58 59 +1

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @walterra

@walterra walterra merged commit 706d3de into elastic:main Sep 20, 2022
@kibanamachine kibanamachine added the backport:skip This commit does not require backporting label Sep 20, 2022
@walterra walterra deleted the ml-aiops-api-grouping-tweaks-2 branch September 20, 2022 15:37
@walterra walterra mentioned this pull request Sep 28, 2022
19 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport:skip This commit does not require backporting bug Fixes for quality problems that affect the customer experience ci:cloud-redeploy Always create a new Cloud deployment Feature:ML/AIOps ML AIOps features: Change Point Detection, Log Pattern Analysis, Log Rate Analysis :ml release_note:skip Skip the PR/issue when compiling release notes v8.5.0
Projects
None yet
6 participants