Clamp VT parameter values to 32767 #5200
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary of the Pull Request
This PR clamps the parameter values in the VT
StateMachine
parser to 32767, which was the initial limit prior to PR #3956. This fixes a number of overflow bugs (some of which could cause the app to crash), since much of the code is not prepared to handle values outside the range of ashort
.References
#3956 - the PR where the cap was changed to the range of
size_t
#4254 - one example of a crash caused by the higher range
PR Checklist
Detailed Description of the Pull Request / Additional comments
The DEC STD 070 reference recommends supporting up to at least 16384 for parameter values, so 32767 should be more than enough for any standard VT sequence. It might be nice to increase the limit to 65535 at some point, since that is the cap used by both XTerm and VTE. However, that is not essential, since there are very few situations where you'd even notice the difference. For now, 32767 is the safest choice for us, since anything greater than that has the potential to overflow and crash the app in a number of places.
Validation Steps Performed
I had to make a couple of modifications to the range checks in the
OutputEngineTest
, more or less reverting to the pre-#3956 behavior, but after that all of the unit tests passed as expected.I manually confirmed that this fixes the hanging test case from #5160, as well as overflow issues in the cursor operations, and crashes in
IL
andDL
(see #4254 (comment)).