Add simplified integration framework

[rdner]

That can be used for running Beats binaries in integration tests.

This framework has a simplified API comparing to the existing one and uses a more efficient way to search for logs in the output of the command.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
  - [ ] I have made corresponding changes to the documentation
  - [ ] I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
  - [ ] I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Motivation

Recently I needed to write a few integration tests that run the Filebeat binary and I found the existing integration framework very cumbersome. It requires a lot of boilerplate code and the purpose of any framework is to illuminate that.

Since we're going to eventually migrate all the integration tests written in Python, we need to have a very simple tool for writing integration tests that should cover most of the test-cases.

Both frameworks (old and new) can co-exist for now.

Pros

This design has a few advantages over the previous one:

• It does not use log files for inspecting/matching the expected logs. Instead it connects directly to stdout/stderr and matches all the output expectations in memory line by line as they arrive. Which makes it extremely efficient at expecting thousands of log lines (e.g. confirming each line of a file gets ingested).
• The test suite kills the process immediately once the defined expectations are met, no more polling with intervals.
• It runs the binary that we ship to our customers instead of a custom binary (debatable, I know but I think we should test what we ship)
• It has a call-chain interface which is more compact
• It supports testing cases when a Beat crashes with errors (not sure if the old framework supported that)
• It has very detailed output for debugging a test failure
• It's generic and in theory can be used with any Beat not just Filebeat (yet to be tested)
• Can be extended and specialized for each Beat, see the example with Filebeat in this PR.

For example, test similar to the one written in the older framework:

https://github.com/elastic/beats/blob/main/filebeat/tests/integration/filebeat_test.go#L37-L94

Can be replaced with this:

func TestFilebeat(t *testing.T) {
	messagePrefix := "sample text message"
	fileCount := 5
	lineCount := 128
	configTemplate := `
filebeat.inputs:
  - type: filestream
    id: "test-filestream"
    paths:
      - %s
# we want to check that all messages are ingested
# without using an external service, this is an easy way
output.console:
  enabled: true
`
	// we can generate any amount of expectations
	// they are light-weight
	expectIngestedFiles := func(test Test, files []string) {
		// ensuring we ingest every line from every file
		for _, filename := range files {
			for i := 1; i <= lineCount; i++ {
				line := fmt.Sprintf("%s %s:%d", messagePrefix, filepath.Base(filename), i)
				test.ExpectOutput(line)
			}
		}
	}

	ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
	defer cancel()

	generator := NewJSONGenerator(messagePrefix)
	path, files := GenerateLogFiles(t, fileCount, lineCount, generator)
	config := fmt.Sprintf(configTemplate, path)
	test := NewTest(t, TestOptions{
		Config: config,
	})

	expectIngestedFiles(test, files)

	test.
		ExpectEOF(files...).
		ExpectStart().
		Start(ctx).
		Wait()
}

Additionally, it also includes validation that:

• every file has been read until EOF
• every line from each file was ingested

Another example, this time we expect Beat to crash:

func TestFilebeat(t *testing.T) {
	ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
	defer cancel()

	// path items are required, this config is invalid
	config := `
filebeat.inputs:
  - type: filestream
    id: "test-filestream"
output.console:
  enabled: true
`
	test := NewBeatTest(t, BeatTestOptions{
		RunBeatOptions: RunBeatOptions{
			Beatname: "filebeat",
			Config:   config,
		},
	})

	test.
		ExpectStart().
		ExpectOutput("Exiting: Failed to start crawler: starting input failed: error while initializing input: no path is configured").
		ExpectStop(1).
		Start(ctx).
		Wait()
}

Cons

• This framework does not have all the functionality of the previous one. Only essential functions that cover most of the test-cases. It can be easily extended, every defined interface is an extension point.
• It heavily relies on the terminal output of a Beat, we can configure both console logging and file logging if necessary but it's outside of the scope of this framework.

Current functionality

Basic Assertions

• Assert an output line that contains a defined string
• Assert a list of output lines in a defined order that contain a given list of strings
• Assert an output line that matches a regular expression
• Assert a list of output lines in a defined order that match a given list of regular expressions
• Assert that the process started
• Assert that the process exited by itself with a certain exit code

Filebeat-specific Assertions

• Assert all files on the list have been read to EOF

Reporting

• Print out all defined expectations of the test
• Print last N lines of the output

Config

• Add additional arguments to the command to run the binary
• Pass a config file (e.g. filebeat.yml)

[mergify]

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @rdner? 🙏.
For such, you'll need to label your PR with:

• The upcoming major version of the Elastic Stack
• The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-8./d is the label to automatically backport to the 8./d branch. /d is the digit

[mergify]

backport-8.x has been added to help with the transition to the new branch 8.x.
If you don't need it please use backport-skip label and remove the backport-8.x label.

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)