Reduce debug.log file size by folding consecutive identical errors

[OrenAudeles]

Summary

Bugfixes "Reduce debug.log file size by folding consecutive identical errors"

Purpose of change

Should Fix #50017 without requiring the user to modify file permissions. File size of debug.log is unbounded and in the worst case may grow uncontrollably until storage is full.

Describe the solution

This PR does two things to mitigate the issue.

1. Folds identical errors as much as possible by tracking the unique information of an error (filename, line, function, error message ) and increments a counter if these are identical AND the newest error is within a 100ms window following the previous error. 100ms is a lot of time for a computer, so if the threshold is exceeded it is probably unique enough to deserve its own error line.
2. Kills program execution at 10k consecutive identical errors. Combined with the 100ms threshold between errors if this threshold is reached something has probably gone horribly wrong and the program has not been able to recover gracefully. 10k was picked to reduce the chance for a false positive.

Describe alternatives you've considered

Mostly readability alternatives for folding. Considered not adding in the killswitch, but if the program is repeatedly erroring on the same line without end the user will end up forcing execution to halt anyway. Considered using a lower value for the killswitch.

Testing

Added calls to debugmsg in loops to see how the folding looks. Used debug menu to force crash the game to make sure the folding is output properly.

Additional context

https://media.discordapp.net/attachments/675502002230394880/890078801176965160/unknown.png

[Qrox]

if the program is repeatedly erroring on the same line without end the user will end up forcing execution to halt anyway

That is not necessarily true. For example, during a player activity, a debug message may be triggered repeatedly and may exceed 10 times a second, but the game does not necessarily freeze (the user can ignore the message and continue playing).

[OrenAudeles]

That is not necessarily true. For example, during a player activity, a debug message may be triggered repeatedly and may exceed 10 times a second, but the game does not necessarily freeze (the user can ignore the message and continue playing).

okay, I'll grant that as plausible. Part of why I made the threshold so high is to minimize the chance for false positives. Two ways forward if false positives are encountered. Raise the threshold by another factor of 10 to make it less likely, and tighten to temporal threshold from 100ms to something shorter.

If a player does get caught by the killswitch they should probably file a bug report. 10k repetitions of the exact same error in a short period of time is a problem that needs solving on our end.

[Qrox]

Maybe we can just ask the player to file a bug report in that case instead of crashing the game. (By showing another debug message, for example)

[OrenAudeles]

Players are more likely to file reports when their game crashes. If they can ignore it they will ignore it. If it generates too many false positives instead of catching the absurd loops that were bloating the debug log by gigabytes then it can be dealt with then.

I could also change the folding timeout mechanism from its current 100ms between messages to time threshold over all repetitions which may be able to handle this better. 10k+ identical errors over a few seconds may be more indicative of an unrecoverable loop than what is likely to occur with player_activity generated errors.

[kevingranade]

I agree crashing is excessive, just escalating to a popup with clear directions to report the issue is as far as we should go.

[Qrox]

Calling debugmsg recursively here might have unintended side-effects. I suggest prepending this line to the original message instead. Might need to add a new flag to debug_error_prompt and use it in the message key to force display the error when the original error was ignored. This also has the benefit of showing the actual repeating error in the debug message.

[OrenAudeles]

Probably the correct move. I'll modify to prepend the original message.
The excessive repetitions error doesn't serve much use going into debug.log but it would be useful as an error prompt. The excessive repetitions will become apparent when reading the log and an entry for that purpose doesn't add information.
An unintended consequence for calling debugmsg here I can see at the moment is for a degenerate case where a single error is repeated millions of times. This will generate folded repetition groups of

[ERROR] fold begin
error fold 10k
[ERROR] fold end
excessive repetition error

and do so a large number of times. It will save some space in debug.log but it will become less readable as a result. It would be better to just output the prompt and skip directly logging it.

The simplest change would be to move this test down to prompt generation and make the test
if( ( rep_folder.repeat_count % repetition_folder::repetition_threshold ) == 0 ) { ... }
and generate a standard ignorable prompt or a forced prompt. If there is prompt forcing via flag would need to alter both debug_error_prompt and replay_buffered_debugmsg_prompts to handle the flag.
A forced, unignorable prompt would interrupt the player frequently with the prompt in a degenerate case and is likely to generate some kind of bug report here, or questions on discord or reddit which can then be used to further prompt a bug report.

Next commit:

1. Change repetition error into prompt generation so as to not prematurely end existing folds (primary PR goal)
2. Add in original error message to prompt for readability

For now I will not add in forced prompts. For now if the user wants to ignore the error they will be able to. Can be future work in a future PR if it isn't immediately required.
Ended up adding in forced prompts to ensure that the user does see the prompt once per set of excessive repetitions.