[Filebeat] - validate files before harvesting

[VihasMakwana]

Current Issue

• This issue exists is we enable prospector.scanner.symlinks.
• If we enable prospector.scanner.symlinks and try to ingest a symlink in symlink -> non-regular file manner, the irregular file will still be ingested.
• This shouldn't happen and we should validate the source file as well.

Describe the enhancement:

• Currently, we validate a file here and only harvest a regular file (i.e. NOT directories, pipes, sockets, device etc).
• I propose to move this step before harvesting it i.e. here.

Describe a specific use case for the enhancement or feature:

• With [filebeat][filestream] Enable status reporter for filestream input #40121 open, we will soon have status reporter.
• Our filestream input will remain degraded unless the user changes fixes the symlink and source file.
• In my opinion, we should have never ingested the symlink in the first place as the source file is an irregular file.

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[pierrehilbert]

@rdner / @belimawr could you please have a look here to share your thoughts?

[belimawr]

@VihasMakwana, I'm a bit confused here... We check if a file is a regular file right before opening it, which is indeed after resolving the symlink. But every single file we open for reading is will go through the validation at

beats/filebeat/input/filestream/input.go

Lines 293 to 296 in 032a4cf

	err = checkFileBeforeOpening(fi)
	if err != nil {
	return nil, nil, false, err
	}

which checks the file mode

beats/filebeat/input/filestream/input.go

Lines 323 to 329 in 032a4cf

	func checkFileBeforeOpening(fi os.FileInfo) error {
	if !fi.Mode().IsRegular() {
	return fmt.Errorf("tried to open non regular file: %q %s", fi.Mode(), fi.Name())
	}
	return nil
	}

So even if the fileWatcher finds a file under a symlink that is not a regular file and returns it, filestream.openFile will still validate it by calling checkFileBeforeOpening and won't be harvested if it is not a regular file.

Regarding reporting the input status as degraded in this case, I believe it is the correct behaviour. The user has configured Filestream to ingest something that is not a regular file, thus the user should be notified of their error and the input should stay degraded until this is fixed.

We just need to make sure the message returned to the user is clear enough so the can understand and act on it.

[VihasMakwana]

@belimawr thanks for sharing your thoughts. I agree with you.

Closing this.