Add UTF-8 support #19
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This will simplify editing by allowing the use of Int indexes instead of having to use String.Index. Rename EditState.currentBuffer to .text, since it's a String, which is generated from .buffer. Redefine how .cursorPosition is computed, considering non-ASCII characters to occupy two positions (works with Linux and Windows terminals command line editing). This was experimentally determined. Change return type of LineNoise.readCharacter() to Character?. This allows more complex characters to be input, such as emoji, which may occupy up to 4 UTF-8 bytes per character. Implement a simple UTF-8 decoding state machine by inspecting the first input byte. If its value is < 128, then it's an ASCII character, which can be immediately returned. Otherwise the number of high order bits set in the byte is used to compute the number of expected bytes needed to assemble the full character. These are buffered in a Data object, and when complete, returned as a Character. Change LineNoise.output(character:) to correctly convert the character into the proper number of bytes to pass to write, using its .utf8 property. Change LineNoise.output(text:) to correctly convert a String to the proper number of bytes using its .utf8 property. Change LineNoise.getCursorXPosition(inputFile:outputFile:) to use [Character] for its local character buffer. Change LineNoise.handleCharacter(_:editState:) to use Character as its first parameter type, and use .asciiValue to compare with the various control characters in the ControlCharacters enum. This works because .asciiValue returns nil if the character isn't ASCII, instead of silently truncating to an ASCII character. Change LineNoise.insertCharacter(_:editState:) to use Character as its first parameter type. Change unit tests to use .buffer and .text properties consistently. In some cases, it makes sense to write strings to .text, and in other cases array values to .buffer.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is my attempt to add support for UTF-8 input. This generalizes the reading and writing of characters and strings, using the Character type instead of representing characters as UInt8.
I've tested it on Arch Linux primarily, over SSH and directly using a KDE terminal application. All the unit tests pass, and terminal display and editing seem to work. It's possible with more work to simplify the buffer indexing, since it's now using an internal buffer of [Character] as the intermediate representation. This would allow using Int for character indexing, instead of having to use buffer.startIndex, .endIndex. However, I've left that unchanged for now, as that would be needed when using array slices.