Skip to content

Commit

Permalink
docs: filestream review updates (#25568)
Browse files Browse the repository at this point in the history
  • Loading branch information
bmorelli25 authored May 6, 2021
1 parent e2bfa10 commit afbdaa9
Show file tree
Hide file tree
Showing 3 changed files with 17 additions and 16 deletions.
8 changes: 4 additions & 4 deletions filebeat/docs/inputs/input-filestream-file-options.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -127,7 +127,7 @@ The files affected by this setting fall into two categories:
* Files that were harvested but weren't updated for longer than `ignore_older`

For files which were never seen before, the offset state is set to the end of
the file. If a state already exist, the offset is not changed. In case a file is
the file. If a state already exists, the offset is not changed. If a file is
updated again later, reading continues at the set offset position.

The `ignore_older` setting relies on the modification time of the file to
Expand Down Expand Up @@ -185,9 +185,9 @@ certain criteria or time. Closing the harvester means closing the file handler.
If a file is updated after the harvester is closed, the file will be picked up
again after `prospector.scanner.check_interval` has elapsed. However, if the file
is moved or deleted while the harvester is closed, {beatname_uc} will not be able
to pick up the file again, and any data that the harvester hasn't read will be lost.
to pick up the file again, and any data that the harvester hasn't read will be lost.

The `close.on_state_change.*` settings are applied asynchronously
The `close.on_state_change.*` settings are applied asynchronously
to read from a file, meaning that if {beatname_uc} is in a blocked state
due to blocked output, full queue or other issue, a file that would be
closed regardless.
Expand Down Expand Up @@ -349,7 +349,7 @@ that should be removed based on the `clean_inactive` setting. This happens
because {beatname_uc} doesn't remove the entries until it opens the registry
again to read a different file. If you are testing the `clean_inactive` setting,
make sure {beatname_uc} is configured to read from more than one file, or the
file state will never be removed from the registry.
file state will never be removed from the registry.

[float]
[id="{beatname_lc}-input-{type}-clean-removed"]
Expand Down
17 changes: 9 additions & 8 deletions filebeat/docs/inputs/input-filestream-reader-options.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -145,14 +145,15 @@ The maximum number of bytes that a single log message can have. All bytes after
[float]
===== `parsers`

This option expects a list of parsers the log line has to go through.
This option expects a list of parsers that the log line has to go through.

Avaliable parsers:
- `multiline`
- `ndjson`
Available parsers:

* `multiline`
* `ndjson`

In this example, {beatname_uc} is reading multiline messages that consist of 3 lines
and encapsulated in single-line JSON objects.
and are encapsulated in single-line JSON objects.
The multiline message is stored under the key `msg`.

["source","yaml",subs="attributes"]
Expand Down Expand Up @@ -188,7 +189,7 @@ decoding only works if there is one JSON object per message.
The decoding happens before line filtering. You can combine JSON
decoding with filtering if you set the `message_key` option. This
can be helpful in situations where the application logs are wrapped in JSON
objects, as with like it happens for example with Docker.
objects, like when using Docker.

Example configuration:

Expand All @@ -214,7 +215,7 @@ structure. For example, `{"a.b.c": 123}` would be expanded into `{"a":{"b":{"c":
This setting should be enabled when the input is produced by an
https://github.com/elastic/ecs-logging[ECS logger].

*`add_error_key`*:: If this setting is enabled, {beatname_uc} adds a
*`add_error_key`*:: If this setting is enabled, {beatname_uc} adds an
"error.message" and "error.type: json" key in case of JSON unmarshalling errors
or when a `message_key` is defined in the configuration but cannot be used.

Expand All @@ -226,7 +227,7 @@ occur.

*`document_id`*:: Option configuration setting that specifies the JSON key to
set the document id. If configured, the field will be removed from the original
json document and stored in `@metadata._id`
JSON document and stored in `@metadata._id`

*`ignore_decoding_error`*:: An optional configuration setting that specifies if
JSON decoding errors should be logged or not. If set to true, errors will not
Expand Down
8 changes: 4 additions & 4 deletions filebeat/docs/inputs/input-filestream.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -10,11 +10,11 @@ beta[]
++++

Use the `filestream` input to read lines from active log files. It is the
new, improved alternative to the `log` input. It comes various improvements
new, improved alternative to the `log` input. It comes with various improvements
to the existing input:

1. Checking of `close_*` options happens out of band. Thus, if an output is blocked
{beatname_uc} is able to close the reader and it avoids keeping too many files open.
1. Checking of `close_*` options happens out of band. Thus, if an output is blocked,
{beatname_uc} can close the reader and avoid keeping too many files open.

2. Detailed metrics are available for all files that match the `paths` configuration
regardless of the `harvester_limit`. This way, you can keep track of all files,
Expand All @@ -24,7 +24,7 @@ even ones that are not actively read.
aggregate the contents into a multiline event.

4. Some position updates and metadata changes no longer depend on the publishing pipeline.
If a the pipeline is blocked some changes are still applied to the registry.
If the pipeline is blocked some changes are still applied to the registry.

5. Only the most recent updates are serialized to the registry. In contrast, the `log` input
has to serialize the complete registry on each ACK from the outputs. This makes the registry updates
Expand Down

0 comments on commit afbdaa9

Please sign in to comment.