Allow chunking of stdout output files.

[karinushka]

Motivation and Context

When Horust runs for longer periods of time and the supervised programs generate a lot of output, it would be beneficial to introduce a simple log rotation system. A full-blown solution would have Horust have a complicated log rotation and compression system - but that would be an overkill. A much simpler (and chosen) approach is to just chunk the output file and then let the user customize the rotation with a small shell script.

Description

This change adds a parameter stdout-rotation-size which allows setting the maximum size of the stdout output file in bytes. For example: if this is set to 10240 (10 kb) and the stdout is redirected to /tmp/out.txt, then as soon as this file grows to 10kb it will be closed and a new file /tmp/out.txt.1 will be created and the output will continue there.
Having these separate output files allows compressing the, for example with gzip.

How Has This Been Tested?

The changes are tightly bound to spawning threads, duping descriptors and creating files - so it is very hard to test them with unit tests. But they were tested with a simple configuration file:

command = "nfsstat -v -s -Z3"
stdout = "/tmp/foo-stdout.log"
stdout-rotate-size = 2048
working-directory = "/tmp/"

This correctly created a bunch of /tmp/foo-stdout* files after some time, each 2kb large.

Types of changes

• New feature (non-breaking change which adds functionality)
• Documentation (a change to man pages or other documentation)

Checklist:

• I have updated the documentation accordingly.
• I have added tests to cover my changes.

[FedericoPonzi]

Thanks a lot for this! Can you take a look at the CI output please?

[karinushka]

Yup, I somehow lost the ending brace of a function when staging the diff. This should be fixed now.

[FedericoPonzi]

I've added some comments, overall looks very good thanks a lot!

I agree unit-tests are complicated to write for this case but It should be possible to write an integration test instead.

You could create a unit test using this as a base like in your test:

command = "nfsstat -v -s -Z3"
stdout = "/tmp/foo-stdout.log"
stdout-rotate-size = 2048
working-directory = "/tmp/"

the log redirection is tested here: https://github.com/FedericoPonzi/Horust/blob/master/tests/section_general.rs#L51 but if you take a look at the other tests you should be able to quickly write an integ test for this case given you already have the service code.

[FedericoPonzi]

Can you also add the default value and how it behaves with the default? Also, it might be more convenient if these were kilobytes or even megabytes? I don't think users need bytes granulation level for logs, it depends on the use case but I would probably use 100MB as a default for my horust configs.

Actually, should we set the default to some sane value? I think in the majority of the use cases, they won't want to set this to 0. Maybe we should put 50 or 100 megs as default, and let users pick 0 if they really want to. What do you think? (I'm aware it's a change of behavior, but it feels like a desirable change).

[karinushka]

This would break the logic of having simple config files like:

stdout = "STDOUT"

Not specifying the stdout-rotate-size would default it to some value and this contradicts the STDOUT, which can't be chunked. So this behavior change would break a bunch of integration tests and potentially a bunch of existing user configs.

[karinushka]

but if you take a look at the other tests you should be able to quickly write an integ test for this case given you already have the service code.

Ah cool, you already have a bunch of test harnesses!
Added a test which spawns a simple writer to stdout and then checks whether its output is chunked into expected file names.

Please take another look 😸

Also if you prefer having a "clean" PR with fewer commits, I can squash my fixes into the initial commits and force re-push it.

[FedericoPonzi]

Can you also add the new parameter to the sample service? https://github.com/FedericoPonzi/Horust/blob/31c829792b86d28d220084d2c5f8620108984d9c/example_services/sample_service.toml this is used as source for horust --sample-service command.
You can also squash the commits, otherwise I'll squash them from the github ui when merging.

[karinushka]

Adjusted the sample service - stderr now goes to stderr, while stdout goes to a rotated file.

[FedericoPonzi]

The deserialize test will also need to be updated:

Horust/src/horust/formats/service.rs

Line 672 in 31c8297

fn test_should_correctly_deserialize_sample() {

[karinushka]

The deserialize test will also need to be updated:

Yup, just noticed and fixed it 😄

[FedericoPonzi]

Thanks a lot for your contribution!