-
Notifications
You must be signed in to change notification settings - Fork 4.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Agent] Inject index with sane defaults to configuration #16903
[Agent] Inject index with sane defaults to configuration #16903
Conversation
Pinging @elastic/ingest-management (Project:fleet) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think it's important that the config we have here is the a config that already provides value to the user. It's not only a sample. I compare to to running metricbeat with ./metricbeat
. All the system data is just there and works.
x-pack/agent/_meta/agent.yml
Outdated
- /var/log/hello1.log | ||
- /var/log/hello2.log | ||
datasources: | ||
- namespace: sample |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why not using the default
namespace here?
x-pack/agent/_meta/agent.yml
Outdated
- type: logs | ||
processors: | ||
streams: | ||
- dataset: sample.acccess |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I suggest if we don't have any good default logs here, we just skip the log parts / leave it commented out. I would suggest the system logs here (check logs system module)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
system module seems purely linux oriented so i left it out so our future windows test wont fail on something like this
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That is where in the future our constraints come into play. I think part of it work also on OS X
The samples you put in will also not work. Suggestion: Lets skip the logs part for now but have a logs sample based on the Linux logs inside but commented out. So users can easily edit it.
paths: /var/log/sample/error.log | ||
- type: system/metrics | ||
streams: | ||
- metricset: cpu |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Lets add a few more here like memory, disk.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can you change to the default namespace everywhere?
x-pack/agent/_meta/common.p2.yml
Outdated
- /var/log/hello1.log | ||
- /var/log/hello2.log | ||
datasources: | ||
- namespace: sample |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
- namespace: sample | |
- namespace: default |
x-pack/agent/_meta/agent.docker.yml
Outdated
- /var/log/hello1.log | ||
- /var/log/hello2.log | ||
datasources: | ||
- namespace: sample |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
- namespace: sample | |
- namespace: default |
- /var/log/hello1.log | ||
- /var/log/hello2.log | ||
datasources: | ||
- namespace: sample |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
- namespace: sample | |
- namespace: default |
go.sum
Outdated
@@ -214,6 +214,7 @@ github.com/eapache/queue v1.1.0 h1:YOEu7KNc61ntiQlcEeUIoDTJ2o8mQznoNvUhiigpIqc= | |||
github.com/eapache/queue v1.1.0/go.mod h1:6eCeP0CKFpHLu8blIFXhExK/dRa7WDZfr6jVFPTqq+I= | |||
github.com/eclipse/paho.mqtt.golang v1.2.1-0.20200121105743-0d940dd29fd2 h1:DW6WrARxK5J+o8uAKCiACi5wy9EK1UzrsCpGBPsKHAA= | |||
github.com/eclipse/paho.mqtt.golang v1.2.1-0.20200121105743-0d940dd29fd2/go.mod h1:H9keYFcgq3Qr5OUJm/JZI/i6U7joQ8SYLhZwfeOo6Ts= | |||
github.com/elastic/beats v7.6.1+incompatible h1:4iPVpFr8BSJW2fUVi+/tYXQ4v/IYVx0k2PPLTg8cfks= |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think this is weird?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
code LGTM, I haven't tested it.
Test looks OK to catch the problems.
- enabled: true | ||
metricset: info | ||
|
||
settings.monitoring: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
+1 for changing this.
@michalpristas lets add this change to the new changelog too, so we know when the new syntax has landed. |
yep i will after we merge changelog in |
settings.monitoring: | ||
use_output: monitoring | ||
|
||
management: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not in this PR, but thinking this + config should also be under settings?
inputs: | ||
- type: system/metrics | ||
streams: | ||
- metricset: cpu |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Had a quick chat with @michalpristas about this on Slack. The problem here is that this will send data to metrics-generic-default
but it should end up in metrics-system.cpu-default
. We probably have to add the dataset here at the moment and simplify it later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Configs LGTM.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM, lets get that merged in.
I've added the "need_backport" will merge after #16885 |
Jenkins test this |
[Agent] Inject index with sane defaults to configuration (elastic#16903) (cherry picked from commit 96e4506)
Migrated from: #15852
Build on top of: #15809
Added one extra rule for injecting index into inputs based on provided type and detected namespace/dataset type.
Other option was to use decorator but at the time of decorating there are already some pieces of config missing.
Draft until final configuration format is settled and mentioned PR is in, but READY for review.
It overwrites index specified in input.
Fixes: #15690
Fixes: #15691
cc @ruflin @ph