Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor: Introduce a Global Registry #7392

Merged
merged 9 commits into from
Jun 26, 2018
Merged

Conversation

ph
Copy link
Contributor

@ph ph commented Jun 21, 2018

Motivation:

In the current libbeat implementation and also inside the actual beat we are defining registries for each
types of feature that we want to expose. This add duplication to the project, the global registry
is a way to keep the flexibility of multiple features and reduce the duplication code only to take
care of the type satefy.

Also all features now use an init function to make the plugin register with their specific registry.
This PR is a step forward to remove that pattern and use a global variable in the package to identify
the feature. This change will allow a beat author to build a beat with only a specific set of feature.

Example: Build with only ES out and not Logstash and kafka, this could reduce the size of some beats.

This PR is written in a backward compatible way, to make the init and the new feature works both at
the same time.

Instead of using an init function you will the following to expose the feature.

// Feature exposes a spooling to disk queue.
var Feature = queue.Feature("spool", create, feature.Beta)

Each new type of feature require to implement two things for type satefy:

  • A factory method to assert the correct type at runtime.
  • A sugar method like the queue.Feature, for type satefy at compile time.

Example:

// Feature creates a new type of queue.
func Feature(name string, factory Factory, stability feature.Stability) *feature.Feature {
	return feature.New(Namespace, name, factory, stability)
}

// FindFactory retrieves a queue types constructor. Returns nil if queue type is unknown
func FindFactory(name string) Factory {
	f, err := feature.Registry.Find(Namespace, name)
	if err != nil {
		return nil
	}
	factory, ok := f.Factory().(Factory)
	if !ok {
		return nil
	}

	return factory
}

How it will look like for building beats with a minimal set of plugins:

b := MustBundle(
  MustBundle(docker.Feature),
  MustBundle(dissect.Feature),
  MustBundle(elasticsearch.Feature, logstash.Feature),
)

feature.RegisterBundle(b)

Caveats:

we still expose the methods and the registry as global, but this is a step to to isolate a registry
per beat.

TODO:

  • Move all plugins to the feature concept.
  • Move the external plugin to the feature concept.
  • Update the auto generation tools.
  • Changelog
  • Developper changelog
  • Meta issue for migration
    • codec
    • output
    • processor
    • autodiscover
  • How to discover features, we could generate a feature list automatically.

Ref: #7093

ph added 2 commits June 21, 2018 12:04
*Motivation:*

In the current libbeat implementation and also inside the actual beat we are defining registries for each
types of feature that we want to expose. This add duplication to the project, the global registry
is a way to keep the flexibility of multiple features and reduce the duplication code only to take
care of the type satefy.

Also all features now use an init function to make the plugin register with their specific registry.
This PR is a step forward to remove that pattern and use a global variable in the package to identify
the feature. This change will allow a beat author to build a beat with only a specific set of feature.

Example: Build with only ES out and not Logstash and kafka, this could reduce the size of some beats.

This PR is written in a backward compatible way, to make the init and the new feature works both at
the same time.

Instead of using an init function you will the following to expose the feature.
```golang

// Feature exposes a spooling to disk queue.
var Feature = queue.Feature("spool", create, feature.Beta)
```

Each new type of feature require to implement two things for type satefy:

- A factory method to assert the correct type at runtime.
- A sugar method like the `queue.Feature`, for type satefy at compile time.

*Example:*

```golang
// Feature creates a new type of queue.
func Feature(name string, factory Factory, stability feature.Stability) *feature.Feature {
	return feature.New(Namespace, name, factory, stability)
}

// FindFactory retrieves a queue types constructor. Returns nil if queue type is unknown
func FindFactory(name string) Factory {
	f, err := feature.Registry.Find(Namespace, name)
	if err != nil {
		return nil
	}
	factory, ok := f.Factory().(Factory)
	if !ok {
		return nil
	}

	return factory
}

```

How it will look like for building beats with a minimal set of plugins:

```
b := MustBundle(
  MustBundle(docker.Feature),
  MustBundle(dissect.Feature),
  MustBundle(elasticsearch.Feature, logstash.Feature),
)

feature.RegisterBundle(b)

```

*Caveats:*

we still expose the methods and the registry as global, but this is a step to to isolate a registry
per beat.
@ph ph changed the title Introduce a Global Registry Refactor: Introduce a Global Registry Jun 21, 2018
@ph
Copy link
Contributor Author

ph commented Jun 21, 2018

As a side effect this gives us the opportunity to make sure a beats doesn't load anything experimental or beta.

b := MustBundle(
  MustBundle(docker.Feature),
  MustBundle(dissect.Feature),
  MustBundle(elasticsearch.Feature, logstash.Feature),
)

feature.RegisterBundle(b.Filter(feature.Stable))
feature.RegisterBundle(b.Filter(feature.Stable, feature.Beta))

@ph
Copy link
Contributor Author

ph commented Jun 22, 2018

In my previous comment, I meant execute not load, the features will still get compiled in.

}
}
return NewBundle(merged), nil
}
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will change this, I will remove the switch case.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The following might be better.

diff --git a/libbeat/feature/bundle.go b/libbeat/feature/bundle.go
index 174c2f973..9886b68b5 100644
--- a/libbeat/feature/bundle.go
+++ b/libbeat/feature/bundle.go
@@ -1,22 +1,26 @@
 package feature
 
-import "fmt"
+// Groupable allows features and Bundle to be merged together.
+type Groupable interface {
+	// Features returns a slice of features.
+	Features() []Featurable
+}
 
 // Bundle defines a list of features available in the current beat.
 type Bundle struct {
-	Features []Featurable
+	features []Featurable
 }
 
 // NewBundle creates a new Bundle of feature to be registered.
 func NewBundle(features []Featurable) *Bundle {
-	return &Bundle{Features: features}
+	return &Bundle{features: features}
 }
 
 // Filter creates a new bundle with only the feature matching the requested stability.
 func (b *Bundle) Filter(stabilities ...Stability) *Bundle {
 	var filtered []Featurable
 
-	for _, feature := range b.Features {
+	for _, feature := range b.features {
 		for _, stability := range stabilities {
 			if feature.Stability() == stability {
 				filtered = append(filtered, feature)
@@ -27,29 +31,16 @@ func (b *Bundle) Filter(stabilities ...Stability) *Bundle {
 	return NewBundle(filtered)
 }
 
-// MustBundle takes existing bundle or features and create a new Bundle with all the merged Features,
-// will panic on errors.
-func MustBundle(features ...interface{}) *Bundle {
-	b, err := BundleFeature(features...)
-	if err != nil {
-		panic(err)
-	}
-	return b
+// Features returns a slice of features.
+func (b *Bundle) Features() []Featurable {
+	return b.features
 }
 
-// BundleFeature takes existing bundle or features and create a new Bundle with all the merged
-// Features,
-func BundleFeature(features ...interface{}) (*Bundle, error) {
+// MustBundle takes existing bundle or features and create a new Bundle with all the merged Features.
+func MustBundle(features ...Groupable) *Bundle {
 	var merged []Featurable
 	for _, feature := range features {
-		switch v := feature.(type) {
-		case Featurable:
-			merged = append(merged, v)
-		case *Bundle:
-			merged = append(merged, v.Features...)
-		default:
-			return nil, fmt.Errorf("unknown type, expecting 'Featurable' or 'Bundle' and received '%T'", v)
-		}
+		merged = append(merged, feature.Features()...)
 	}
-	return NewBundle(merged), nil
+	return NewBundle(merged)
 }
diff --git a/libbeat/feature/bundle_test.go b/libbeat/feature/bundle_test.go
index df293cc4b..e675a7f09 100644
--- a/libbeat/feature/bundle_test.go
+++ b/libbeat/feature/bundle_test.go
@@ -16,25 +16,25 @@ func TestBundle(t *testing.T) {
 
 	t.Run("Creates a new Bundle", func(t *testing.T) {
 		b := NewBundle(features)
-		assert.Equal(t, 3, len(b.Features))
+		assert.Equal(t, 3, len(b.Features()))
 	})
 
 	t.Run("Filters feature based on stability", func(t *testing.T) {
 		b := NewBundle(features)
 		new := b.Filter(Experimental)
-		assert.Equal(t, 1, len(new.Features))
+		assert.Equal(t, 1, len(new.Features()))
 	})
 
 	t.Run("Filters feature based on multiple different stability", func(t *testing.T) {
 		b := NewBundle(features)
 		new := b.Filter(Experimental, Stable)
-		assert.Equal(t, 2, len(new.Features))
+		assert.Equal(t, 2, len(new.Features()))
 	})
 
 	t.Run("Creates a new Bundle from specified feature", func(t *testing.T) {
 		f1 := New("libbeat.outputs", "elasticsearch", factory, Stable)
 		b := MustBundle(f1)
-		assert.Equal(t, 1, len(b.Features))
+		assert.Equal(t, 1, len(b.Features()))
 	})
 
 	t.Run("Creates a new Bundle with grouped features", func(t *testing.T) {
@@ -49,6 +49,6 @@ func TestBundle(t *testing.T) {
 			MustBundle(f3, f4),
 		)
 
-		assert.Equal(t, 4, len(b.Features))
+		assert.Equal(t, 4, len(b.Features()))
 	})
 }
diff --git a/libbeat/feature/feature.go b/libbeat/feature/feature.go
index 0931bbe53..1d2621982 100644
--- a/libbeat/feature/feature.go
+++ b/libbeat/feature/feature.go
@@ -59,6 +59,11 @@ func (f *Feature) Stability() Stability {
 	return f.stability
 }
 
+// Features allows to Bundle a Feature with a bundle of features.
+func (f *Feature) Features() []Featurable {
+	return []Featurable{f}
+}
+
 // Equal return true if both object are equals.
 func (f *Feature) Equal(other Featurable) bool {
 	// There is no safe way to compare function in go,

@ruflin
Copy link
Member

ruflin commented Jun 25, 2018

@ph As we currently generate the imports we guarantee that we don't forget about including an output for example or a new Metricbeat module. Would be nice if we could find a method to still verify this with this new method.

"reflect"
)

//go:generate stringer -type=Stability
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about the 'generate stringer' command next to the type declaration or in a separate 0gen.go file. The generated file already contains the type name. Having the generate command next to the type declaration makes it easier to find when navigating the code. Like:

// Comment...
// go:generate stringer -type=Stability
type Stability int

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was added automatically when you call generate stringer -type=Stability in the feature directory :)


//go:generate stringer -type=Stability

// Registry is the global plugin registry, this variable is mean to be temporary to move all the
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

s/mean/meant/

Equal(other Featurable) bool

String() string
}
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

'X returns x' is kind of redundant. Plus godoc will not generate any docs for the methods. Instead the type will be shown in the docs as is in code. Rather then stating the obvious, describing how Featurable interacts with the Bundle and Registry would be more helpful. What's the purpose of Namespace. Why do I need to implement 'equal'?

// Equal return true if both object are equals.
func (f *Feature) Equal(other Featurable) bool {
// There is no safe way to compare function in go,
// but since the method are global it should be stable.
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

s/method/function pointers/

Beta
Experimental
Undefined
)
Copy link

@urso urso Jun 25, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice trick:

const (
  Undefined Stability = iota
  Stable
  Beta
  Experimental
)

This way the 'zero value' of Stability becomes Undefiend -> force developer to use a constant.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point, I have added undefined at the end, will make the change.

feature.Name(),
)

r.namespaces[feature.Namespace()][feature.Name()] = feature
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Instead of calling a function that might potentially generate a string on every call (it's all interfaces), let's introduce some local vars: ns, name := feature.Namespace(), feature.Name().


f, found := r.namespaces[feature.Namespace()][feature.Name()]
if found {
if feature.Equal(f) {
Copy link

@urso urso Jun 25, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can remove Equal from the public interface by providing a helper function checking for equal:

func featuresEq(f1, f2 Featurable) bool {
  type comparable interface { Equal(other Featurable) bool }
  if cmp, ok := f1.(comparable); ok {
    return cmp.Equal(f2)
  }
  return false
} 

return nil
}

// Find returns a specific Find from a namespace or an error if not found.
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Find searches for a Feature by the namespace-name pair.

Alternative name: Lookup

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm OK for using Lookup instead.

return nil
}

return factory
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Using reflection, this pattern can be simplified to:

func FindFactory(name string) (f Factory) {
  if err := feature.Registry.Lookup(Namespace, name, &f); err != nil {
    // log error?
  }
  return
}

Using reflaction based function calling support, we could even implement an 'Instantiate/Invoke' method in the registry (maybe too much magic, though).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think I would prefer to stay away from reflexion as much as possible in the context of plugin, this force user to make the choice.

Copy link
Contributor Author

@ph ph Jun 26, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've created a private bundleable interface implemented by both Feature and Bundle.

I've kept the Feature and Bundle type public, since we will interact with it for creating new Features and filtering / merging bundle.

@ph
Copy link
Contributor Author

ph commented Jun 26, 2018

@ruflin Nothing prevents us to use code generation to generate an include with the new format, there is a point in the description for Update the auto generation tools

@urso urso merged commit e380bf9 into elastic:master Jun 26, 2018
@ph
Copy link
Contributor Author

ph commented Jun 26, 2018

I've moved the TODO list to a meta issue at #7423

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants