Early event encoding obscures raw event content in the event log

[cmacknz]

Looking at recent data in the event log, we are logging the Go representation of the event which now includes the JSON encoded event as an opaque blob. For the log messages below, we need to handle the encoded part correctly in the event before logging it so we can actually see the contents. Currently the logs look like the example below where the problematic part is the EncodedEvent:(*elasticsearch.encodedEvent)(0xc003933700)} key.

{"log.level":"warn","@timestamp":"2024-08-13T14:04:46.708Z","log.logger":"elasticsearch","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/outputs/elasticsearch.(*Client).applyItemStatus","file.name":"elasticsearch/client.go","file.line":489},"message":"Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), Meta:null, Fields:null, Private:interface {}(nil), TimeSeries:false}, Flags:0x0, Cache:publisher.EventCache{m:mapstr.M(nil)}, EncodedEvent:(*elasticsearch.encodedEvent)(0xc003933700)} (status=400): 

We need to log the actual JSON content of the event instead of the reference to it. Possibly we need a type used for logging representing this field as json.RawMessage or similar.

We have tests that attempt to verify that the actual event data is in the event log and it did not catch this, possibly because the event content is also contained in the error message from Elasticsearch.

beats/filebeat/tests/integration/event_log_file_test.go

Lines 124 to 131 in 7c82c86

	strData := string(data)
	eventMsg := "not a number"
	if !strings.Contains(strData, eventMsg) {
	t.Errorf("expecting to find '%s' on '%s'", eventMsg, eventsLogFile)
	t.Errorf("Contents:\n%s", strData)
	t.FailNow()
	}

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)