Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Agent] Inject index with sane defaults to configuration #16903

Merged
merged 10 commits into from
Mar 11, 2020
Merged
Show file tree
Hide file tree
Changes from 5 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions go.sum
Original file line number Diff line number Diff line change
Expand Up @@ -214,6 +214,7 @@ github.com/eapache/queue v1.1.0 h1:YOEu7KNc61ntiQlcEeUIoDTJ2o8mQznoNvUhiigpIqc=
github.com/eapache/queue v1.1.0/go.mod h1:6eCeP0CKFpHLu8blIFXhExK/dRa7WDZfr6jVFPTqq+I=
github.com/eclipse/paho.mqtt.golang v1.2.1-0.20200121105743-0d940dd29fd2 h1:DW6WrARxK5J+o8uAKCiACi5wy9EK1UzrsCpGBPsKHAA=
github.com/eclipse/paho.mqtt.golang v1.2.1-0.20200121105743-0d940dd29fd2/go.mod h1:H9keYFcgq3Qr5OUJm/JZI/i6U7joQ8SYLhZwfeOo6Ts=
github.com/elastic/beats v7.6.1+incompatible h1:4iPVpFr8BSJW2fUVi+/tYXQ4v/IYVx0k2PPLTg8cfks=
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is weird?

github.com/elastic/dhcp v0.0.0-20200227161230-57ec251c7eb3 h1:lnDkqiRFKm0rxdljqrj3lotWinO9+jFmeDXIC4gvIQs=
github.com/elastic/dhcp v0.0.0-20200227161230-57ec251c7eb3/go.mod h1:aPqzac6AYkipvp4hufTyMj5PDIphF3+At8zr7r51xjY=
github.com/elastic/ecs v1.4.0 h1:BGIUwWJhThRO2IQxzm7ekV9TMUGwZoYyevT5/1xmMf0=
Expand Down
16 changes: 10 additions & 6 deletions x-pack/agent/_meta/agent.docker.yml
Original file line number Diff line number Diff line change
Expand Up @@ -8,12 +8,16 @@ outputs:
username: elastic
password: changeme

streams:
- type: event/file
enabled: true
paths:
- /var/log/hello1.log
- /var/log/hello2.log
datasources:
- namespace: default
use_output: default
inputs:
- type: system/metrics
streams:
- metricset: cpu
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Had a quick chat with @michalpristas about this on Slack. The problem here is that this will send data to metrics-generic-default but it should end up in metrics-system.cpu-default. We probably have to add the dataset here at the moment and simplify it later.

- metricset: memory
- metricset: network
- metricset: filesystem

management:
# Mode of management, the agent support two modes of operation:
Expand Down
16 changes: 10 additions & 6 deletions x-pack/agent/_meta/agent.yml
Original file line number Diff line number Diff line change
Expand Up @@ -8,12 +8,16 @@ outputs:
username: elastic
password: changeme

streams:
- type: event/file
enabled: true
paths:
- /var/log/hello1.log
- /var/log/hello2.log
datasources:
- namespace: default
use_output: default
inputs:
- type: system/metrics
streams:
- metricset: cpu
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lets add a few more here like memory, disk.

- metricset: memory
- metricset: network
- metricset: filesystem

management:
# Mode of management, the agent support two modes of operation:
Expand Down
16 changes: 10 additions & 6 deletions x-pack/agent/_meta/common.p2.yml
Original file line number Diff line number Diff line change
Expand Up @@ -8,12 +8,16 @@ outputs:
username: elastic
password: changeme

streams:
- type: event/file
enabled: true
paths:
- /var/log/hello1.log
- /var/log/hello2.log
datasources:
- namespace: default
use_output: default
inputs:
- type: system/metrics
streams:
- metricset: cpu
- metricset: memory
- metricset: network
- metricset: filesystem

management:
# Mode of magement, the agent support two modes of operation:
Expand Down
16 changes: 10 additions & 6 deletions x-pack/agent/_meta/common.reference.p2.yml
Original file line number Diff line number Diff line change
Expand Up @@ -8,12 +8,16 @@ outputs:
username: elastic
password: changeme

streams:
- type: event/file
enabled: true
paths:
- /var/log/hello1.log
- /var/log/hello2.log
datasources:
- namespace: default
use_output: default
inputs:
- type: system/metrics
streams:
- metricset: cpu
- metricset: memory
- metricset: network
- metricset: filesystem

management:
# Mode of management, the agent currently only support the following mode:
Expand Down
16 changes: 10 additions & 6 deletions x-pack/agent/agent.docker.yml
Original file line number Diff line number Diff line change
Expand Up @@ -8,12 +8,16 @@ outputs:
username: elastic
password: changeme

streams:
- type: event/file
enabled: true
paths:
- /var/log/hello1.log
- /var/log/hello2.log
datasources:
- namespace: default
use_output: default
inputs:
- type: system/metrics
streams:
- metricset: cpu
- metricset: memory
- metricset: network
- metricset: filesystem

management:
# Mode of management, the agent support two modes of operation:
Expand Down
16 changes: 10 additions & 6 deletions x-pack/agent/agent.reference.yml
Original file line number Diff line number Diff line change
Expand Up @@ -13,12 +13,16 @@ outputs:
username: elastic
password: changeme

streams:
- type: event/file
enabled: true
paths:
- /var/log/hello1.log
- /var/log/hello2.log
datasources:
- namespace: default
use_output: default
inputs:
- type: system/metrics
streams:
- metricset: cpu
- metricset: memory
- metricset: network
- metricset: filesystem

management:
# Mode of management, the agent currently only support the following mode:
Expand Down
16 changes: 10 additions & 6 deletions x-pack/agent/agent.yml
Original file line number Diff line number Diff line change
Expand Up @@ -13,12 +13,16 @@ outputs:
username: elastic
password: changeme

streams:
- type: event/file
enabled: true
paths:
- /var/log/hello1.log
- /var/log/hello2.log
datasources:
- namespace: default
use_output: default
inputs:
- type: system/metrics
streams:
- metricset: cpu
- metricset: memory
- metricset: network
- metricset: filesystem

management:
# Mode of magement, the agent support two modes of operation:
Expand Down
4 changes: 2 additions & 2 deletions x-pack/agent/dev-tools/cmd/fakewebapi/action_example.json
Original file line number Diff line number Diff line change
Expand Up @@ -3,9 +3,9 @@
"success": true,
"actions": [
{
"type": "POLICY_CHANGE",
"type": "CONFIG_CHANGE",
"data": {
"policy": {
"config": {
"id": "default",
"outputs": {
"default": {
Expand Down
18 changes: 9 additions & 9 deletions x-pack/agent/pkg/agent/application/action_store.go
Original file line number Diff line number Diff line change
Expand Up @@ -33,14 +33,14 @@ func newActionStore(log *logger.Logger, store storeLoad) (*actionStore, error) {
return &actionStore{log: log, store: store}, nil
}

var action actionPolicyChangeSerializer
var action actionConfigChangeSerializer

dec := yaml.NewDecoder(reader)
if err := dec.Decode(&action); err != nil {
return nil, err
}

apc := fleetapi.ActionPolicyChange(action)
apc := fleetapi.ActionConfigChange(action)

return &actionStore{
log: log,
Expand All @@ -53,7 +53,7 @@ func newActionStore(log *logger.Logger, store storeLoad) (*actionStore, error) {
// any other type of action will be silently ignored.
func (s *actionStore) Add(a action) {
switch v := a.(type) {
case *fleetapi.ActionPolicyChange:
case *fleetapi.ActionConfigChange:
// Only persist the action if the action is different.
if s.action != nil && s.action.ID() == v.ID() {
return
Expand All @@ -69,12 +69,12 @@ func (s *actionStore) Save() error {
return nil
}

apc, ok := s.action.(*fleetapi.ActionPolicyChange)
apc, ok := s.action.(*fleetapi.ActionConfigChange)
if !ok {
return fmt.Errorf("incompatible type, expected ActionPolicyChange and received %T", s.action)
}

serialize := actionPolicyChangeSerializer(*apc)
serialize := actionConfigChangeSerializer(*apc)

reader, err := yamlToReader(&serialize)
if err != nil {
Expand All @@ -98,7 +98,7 @@ func (s *actionStore) Actions() []action {
return []action{s.action}
}

// actionPolicyChangeSerializer is a struct that add YAML serialization, I don't think serialization
// actionConfigChangeSerializer is a struct that add YAML serialization, I don't think serialization
// is a concern of the fleetapi package. I went this route so I don't have to do much refactoring.
//
// There are four ways to achieve the same results:
Expand All @@ -108,14 +108,14 @@ func (s *actionStore) Actions() []action {
// 4. We have two sets of type.
//
// This could be done in a refactoring.
type actionPolicyChangeSerializer struct {
type actionConfigChangeSerializer struct {
ActionID string `yaml:"action_id"`
ActionType string `yaml:"action_type"`
Policy map[string]interface{} `yaml:"policy"`
Config map[string]interface{} `yaml:"config"`
}

// Add a guards between the serializer structs and the original struct.
var _ actionPolicyChangeSerializer = actionPolicyChangeSerializer(fleetapi.ActionPolicyChange{})
var _ actionConfigChangeSerializer = actionConfigChangeSerializer(fleetapi.ActionConfigChange{})

// actionStoreAcker wraps an existing acker and will send any acked event to the action store,
// its up to the action store to decide if we need to persist the event for future replay or just
Expand Down
14 changes: 7 additions & 7 deletions x-pack/agent/pkg/agent/application/action_store_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -57,10 +57,10 @@ func TestActionStore(t *testing.T) {

t.Run("can save to disk known action type",
withFile(func(t *testing.T, file string) {
actionPolicyChange := &fleetapi.ActionPolicyChange{
actionConfigChange := &fleetapi.ActionConfigChange{
ActionID: "abc123",
ActionType: "POLICY_CHANGE",
Policy: map[string]interface{}{
ActionType: "CONFIG_CHANGE",
Config: map[string]interface{}{
"hello": "world",
},
}
Expand All @@ -70,7 +70,7 @@ func TestActionStore(t *testing.T) {
require.NoError(t, err)

require.Equal(t, 0, len(store.Actions()))
store.Add(actionPolicyChange)
store.Add(actionConfigChange)
err = store.Save()
require.NoError(t, err)
require.Equal(t, 1, len(store.Actions()))
Expand All @@ -82,12 +82,12 @@ func TestActionStore(t *testing.T) {
actions := store1.Actions()
require.Equal(t, 1, len(actions))

require.Equal(t, actionPolicyChange, actions[0])
require.Equal(t, actionConfigChange, actions[0])
}))

t.Run("when we ACK we save to disk",
withFile(func(t *testing.T, file string) {
actionPolicyChange := &fleetapi.ActionPolicyChange{
actionConfigChange := &fleetapi.ActionConfigChange{
ActionID: "abc123",
}

Expand All @@ -98,7 +98,7 @@ func TestActionStore(t *testing.T) {
acker := newActionStoreAcker(&testAcker{}, store)
require.Equal(t, 0, len(store.Actions()))

require.NoError(t, acker.Ack(context.Background(), actionPolicyChange))
require.NoError(t, acker.Ack(context.Background(), actionConfigChange))
require.Equal(t, 1, len(store.Actions()))
}))
}
4 changes: 2 additions & 2 deletions x-pack/agent/pkg/agent/application/fleet_gateway_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -193,10 +193,10 @@ func TestFleetGateway(t *testing.T) {
{
"actions": [
{
"type": "POLICY_CHANGE",
"type": "CONFIG_CHANGE",
"id": "id1",
"data": {
"policy": {
"config": {
"id": "policy-id"
}
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -14,24 +14,24 @@ import (
"github.com/elastic/beats/v7/x-pack/agent/pkg/fleetapi"
)

type handlerPolicyChange struct {
type handlerConfigChange struct {
log *logger.Logger
emitter emitterFunc
}

func (h *handlerPolicyChange) Handle(ctx context.Context, a action, acker fleetAcker) error {
h.log.Debugf("HandlerPolicyChange: action '%+v' received", a)
action, ok := a.(*fleetapi.ActionPolicyChange)
func (h *handlerConfigChange) Handle(ctx context.Context, a action, acker fleetAcker) error {
h.log.Debugf("handlerConfigChange: action '%+v' received", a)
action, ok := a.(*fleetapi.ActionConfigChange)
if !ok {
return fmt.Errorf("invalid type, expected ActionPolicyChange and received %T", a)
return fmt.Errorf("invalid type, expected ActionConfigChange and received %T", a)
}

c, err := config.NewConfigFrom(action.Policy)
c, err := config.NewConfigFrom(action.Config)
if err != nil {
return errors.New(err, "could not parse the configuration from the policy", errors.TypeConfig)
}

h.log.Debugf("HandlerPolicyChange: emit configuration for action %+v", a)
h.log.Debugf("handlerConfigChange: emit configuration for action %+v", a)
if err := h.emitter(c); err != nil {
return err
}
Expand Down
Loading