-
Notifications
You must be signed in to change notification settings - Fork 8.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[APM] ML jobs are limited to a single space #119975
Comments
Pinging @elastic/apm-ui (Team:apm) |
The module setup endpoint can take the boolean argument |
I am not entirely sure that ML jobs should be space agnostic. Spaces are used to segment access to certain data sources. Results from ML jobs contain a subset of the data that has been analysed. It is reasonable to think that a customer who wishes to segment (and restrict access to) their APM data in-app, will also want to segment (and restrict access to) their ML results. If this is the case, then the ML job should be space aware. |
Thanks for chiming in @sophiec20 . Nothing is decided yet, and things will most likely change as we start working on service groups (can't link to design doc since it's in a private repo but can send on slack). Either way, my current thoughts on this: Ideally I think the ML jobs should follow the APM index pattern settings. So by default all spaces will read from However, if the user changes the index patterns for a space, eg It is currently not possible to modify index pattern settings per space but it is something I expect us to add in the future.
Afaik spaces cannot be used to restrict access to APM data in a secure manner since the user will still be able access the raw data via Dev Tools or Discover (or by requesting ES directly). To restrict access admins will need to use document level security (which won't apply to ML jobs afaict) or ingest data to separate data stream namespaces and then restrict access on an index level. |
@peteharverson asked for my 2 cents on this-- It sounds reasonable to take some steps to avoid duplication of ML jobs. Keep in mind that the end user must have write access to the ML privileges in The "Share saved objects to spaces" flyout contains a space selector that checks for the user's privileges and presents them with an option to select spaces. The Security plugin exposes an endpoint to check these privileges. It sounds like APM could benefit from a component that behaves similarly -- perhaps it could just allow the user to select "This space" or "All spaces" for simplicity? I don't have all of the context on this, but based on what I know, here's what I propose: When creating an ML Job through APM:
If that sounds desirable, the Platform Security team can work on exposing a reusable control component for you to consume. Related: #49647 |
@jportner Thanks for chiming in!
I don't think this will provide a good user experience. Imagine the user has the following three spaces with separate index settings:
If an ML job is created in Space 1 and "All spaces" is selected, it will also apply to Space 3 which has a totally different data set. We will be showing ML anomalies detected on If the user instead creates an ML job in Space 1 and selects "This space" the job will be isolated to Space 1 and the user will have to create an identical job for Space 2 even though the data set is the same. Thus wasting resources, costing them more money, and requiring them to create the same job twice. Proposal The advantages of this is that implementation details are abstracted away from the user. A job created in one space is automatically available in another space if they have the same index settings, and not in spaces with different index settings. Let me know if we should zoom on this. |
Nit: let's please try not to use the term "space agnostic", that means something else entirely when it comes to Saved Objects (those are specific object types that exist outside of spaces), and using that term here might confuse other folks who read this thread. I'd prefer the term "shared to all spaces" 👍
That's an interesting idea. A few things stick out to me:
Ultimately I think that "APM ML jobs are always shared to all spaces" is a non-starter, our Product team has indicated that the multi-tenant use case is important and we want to use Spaces to be able to truly isolate tenants from one another. Alternatively the workflow could could be something like this:
This has a few advantages:
WDYT? |
That's a good point. We might have to do that to reduce complexity on our side.
Changing the index setting should restart the ML jobs with the new settings.
In general I think we need to try harder not to push complexity onto users. In this case we should just show the relevant ML anomalies - I don't see the benefit in showing a drop down with invalid results.
No, for exactly the reasons you mention. It wouldn't be trivial to implement and we should use exact matches only.
I would love to hear more about this, and what the thinking is around solution teams where the underlying data is inherently space-agnostic (I think I'm using the right term here). In our case APM data lives in data streams which is not space aware, so we have to decide what dimension to partition data by. In our case we were going to simply use the index setting for making APM data space aware. Thus, everything else in APM should follow this model. If we have multiple competing dimensions by which there is space awareness (index settings, ML jobs etc) it can easily become very confusing for the end user.
I think that could work! Even better if the user is not asked to decide between sharing and creating a new job. This again feels like we are bleeding implementation details through to them. Instead I think, if a matching job exists, it should be shared with the current space when they click "Create". |
So, Kibana's current authorization model demands that operators assign ES index privileges, ES cluster privileges, and Kibana privileges separately. I can appreciate that this isn't ideal for solutions, though. We have an open issue for supporting "composite features" in Kibana that automatically include the appropriate ES privileges: #96598 In Fleet's case, we are implementing some changes so that you cannot assign the Fleet privilege to a user in specific spaces, you can only assign it in * All spaces (#118001). But Fleet is an odd case and we expect this to be the exception, not the rule. We made that decision fully realizing that this prevents multi-tenant / MSP operators from allowing their users to use Fleet.
If the ML job existed in other spaces, though, would it change the settings there too? Or would you expect the user to go back to those other space(s) and create a new ML job? This idea of tying the ML job to the space's index setting sounds hairy, tbh.
100% agreed! We can always add additional customization in the future if we decide we need it. But it's much harder to take away a feature once we've released it.
Sounds reasonable, and that would be easy to do. These APM ML jobs almost feel like they should be "managed" saved objects, in that they really aren't intended to be changed by the end user -- is that right? Fleet (again) has a similar need, though I can't find an open issue for it, I think it's buried in one of their issues. |
Note for 8.1 we plan to add a 'Managed' badge for anomaly detection jobs to indicate when they have been deployed and managed by elastic - #120631. This will include adding warnings when the user edits, deletes, stops or starts these type of jobs. |
Problem
ML jobs created through APM is currently only displayed in the space where they were created.
This means that even though the user has the exact same 5 services across all their spaces, they need to create ML jobs in every one of them. This both leads to a bad user experience and a performance overhead since we need duplicate ML jobs.
Expectation
ML jobs should be space agnostic.
The text was updated successfully, but these errors were encountered: