[Metricbeat] ReporterV2 migration meta-issue #10774
Comments
sayden
added
meta
Metricbeat
|
Do we need to regenerate the |
|
Correction, some of them are changing https://github.com/elastic/beats/pull/10817/files#diff-5a9cef679c0694a2ace62a1ffc7159ad |
Guide to migration
(python-env) metricbeat ➜ make python-env
(python-env) metricbeat ➜ source build/python-env/bin/activate
(python-env) metricbeat ➜ MODULE=etcd make test-module
# Navigate to the Metricset being migrated and
(python-env) metricbeat ➜ cd module/etcd/store
(python-env) metricbeat ➜ go test ./...
(python-env) metricbeat ➜ go test -tags=integration -data ./...If everything is green, continue to the next step:
func (m *MetricSet) Fetch() (common.MapStr, error) {
content, err := m.http.FetchContent()
if err != nil {
return nil, err
}
return eventMapping(content), nil
}Will end up looking like this: // Fetch methods implements the data gathering and data conversion to the right
// format. It publishes the event which is then forwarded to the output. In case
// of an error set the Error field of mb.Event or simply call report.Error().
func (m *MetricSet) Fetch(reporter mb.ReporterV2) error {
content, err := m.http.FetchContent()
if err != nil {
return err
}
reporter.Event(mb.Event{
MetricSetFields: eventMapping(content),
})
}
func TestFetch(t *testing.T) {
compose.EnsureUp(t, "postgresql")
f := mbtest.NewEventFetcher(t, getConfig())
event, err := f.Fetch()
if !assert.NoError(t, err) {
t.FailNow()
}
t.Logf("%s/%s event: %+v", f.Module().Name(), f.Name(), event)
assert.Contains(t, event, "checkpoints")
assert.Contains(t, event, "buffers")
assert.Contains(t, event, "stats_reset")
//continue...So look for the line f := mbtest.NewReportingMetricSetV2Error(t, getConfig())
events, errs := mbtest.WriteEventsReporterV2Error(f)
if len(errs) > 0 {
t.Fatalf("Expected 0 error, had %d. %v\n", len(errs), errs)
}
assert.NotEmpty(t, events)
event := events[0].MetricSetFields
// test continue untouched...
func TestData(t *testing.T) {
f := mbtest.NewReportingMetricSetV2Error(t, getConfig())
if err := mbtest.WriteEventsReporterV2Error(f, t, ""); err != nil {
t.Fatal("write", err)
}
}
func TestFetchEventContent(t *testing.T) {
absPath, err := filepath.Abs("../_meta/test/")
response, err := ioutil.ReadFile(absPath + "/storestats.json")
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(200)
w.Header().Set("Content-Type", "application/json;")
w.Write([]byte(response))
}))
defer server.Close()
config := map[string]interface{}{
"module": "etcd",
"metricsets": []string{"store"},
"hosts": []string{server.URL},
}
f := mbtest.NewEventFetcher(t, config)
event, err := f.Fetch()
if err != nil {
t.Fatal(err)
}
t.Logf("%s/%s event: %+v", f.Module().Name(), f.Name(), event)
}So the approach is the same than f := mbtest.NewReportingMetricSetV2Error(t, config)
events, errs := mbtest.ReportingFetchV2Error(f)
if len(errs) > 0 {
t.Fatalf("Expected 0 error, had %d. %v\n", len(errs), errs)
}
assert.NotEmpty(t, events)
t.Logf("%s/%s event: %+v", f.Module().Name(), f.Name(), events[0])
}
# Replace etcd with the module you are testing
(python-env) metricbeat ➜ MODULE=etcd make test-module
# Navigate to the Metricset being migrated and
(python-env) metricbeat ➜ cd module/etcd/store
(python-env) metricbeat ➜ go test ./...
(python-env) metricbeat ➜ go test -tags=integration -data ./...Everything is OK if
{
"@timestamp": "2017-10-12T08:05:34.853Z",
"agent": {
"hostname": "host.example.com",
"name": "host.example.com"
},
"etcd": {
"leader": {
"followers": {},
"leader": "8e9e05c52164694d"
}
},
"event": {
"dataset": "etcd.leader",
"duration": 115000,
"module": "etcd"
},
"metricset": {
"name": "leader"
},
"service": {
"address": "127.0.0.1:2379",
"type": "etcd"
}
}So, at the end, just 2 to 4 `files are uploaded in the PR, like here #10817 |
Yes! Some of the changes are expected. Like adding the |
|
Sorry, clicked the wrong button... |
|
Examples and tutorials have been updated after merging #11106 |
|
|
|
Guide updated again to use 45074a3 |
|
Thanks to everybody for helping with this migration. Good work! |
After migrating all metricsets (elastic#10774) to the Reporter interfaces, the EventFetcher and EventsFetcher interface can be removed.
Using
grepI managed to find this modules/metricset usingFetch() (common.Mapstr, error),Fetch () ([]common.Mapstr, error),Run(reporter mb.PushReporter)orRun(reporter mb.PushReporterV2)docker/event (it uses mb.PushReporterV2)graphite/server (it uses mb.PushReporter)http/server (it uses mb.PushReporterV2)kubernetes/event (it uses mb.PushReporter)The text was updated successfully, but these errors were encountered: