x/term: create subrepo for terminal, console, pty and tty related packages

[davecheney]

Following the lead of #13058 towards smaller, more focused, sub repositories, I would like to propose the creation of a sub repo for packages related to handling the interaction with tty like things. This includes

• posix ttys and ptys
• serial termios control, like terminals and rs232 devices.
• windows cmd console

Much of the code for this exists in various packages, some in other sub repos like x/crypto/ssh/terminal, others has been duplicated incompletely by package authors with a specific use case; most times without a comprehensive story for windows.

Various informal proposal have been made, like #12853, relating to posix termios.

Other proposals like #12101 for the creation of an x/sys/terminal group of packages have been rejected on the grounds that sys is too low level, which also dovetails into the desire for more smaller sub repos, rather than larger ones.

There is also the the long standing question of why the code to disable terminal echo lives in the x/crypto/ssh package.

With the exception of the name of the repository, which I'm sure will require debate, is there general support for a sub repository along the lines outlined in this proposal ?

[mattn]

👍 x 256

[sbinet]

yes, please. especially the pty feature (which currently lives under github.com/kr/pty) which is always a bit subtle to handle right.

[rsc]

I am not sure about this one: terminal handling may not be significant enough for its own subrepo.

In general the subrepos seem to be mirroring our top-level standard library directories (sync, net, crypto, ...). It's certainly true that the ssh terminal code does not belong in crypto.

That said, I don't know where else to put it either. Let's see if any other suggestions arrive.

[gdamore]

yes! I am willing to help here. I'd like some higher level primitives, and frankly expressing things in terms of POSIX termio functions would be really nice. It could even be possible to express Windows console APIs in such terms.

In addition to just enabling / disabling raw & cooked mode (and echo/noecho), I see specific needs around:

• baud rate determination (used for real serial lines, but even console programs need it -- for example curses type apps need to know "baud" rate when calculating padding character delays required in terminfo strings
• tcflush & tcdrain
• serial port manipulation (stop bits, parity, etc.)

I'd far rather this all be expressed in terms of POSIX termios than some other way, at least to applications. (Using ioctls or whatever under the hood is fine.)

[bradfitz]

@rsc wrote:

In general the subrepos seem to be mirroring our top-level standard library directories (sync, net, crypto, ...).

Perhaps in that spirit we create golang.org/x/os then, for operating-system specific packages. I have a bunch of code myself in various repositories I'd love to centralize, for things that differ between Linux/Mac/Windows. Terminals seem like it could go under x/os.

[distributed]

Yes, please. I'm the author of github.com/distributed/sers, a way too noncomprehensive serial port library. I basically wrote it because I needed to set "non standard" baud rates to speak with hardware.

I would be glad to help out, should the proposed subrepo become reality.

[kim-racktop]

I have some preliminary code I wrote for the Posix tc functions that I have tested on Illumos. I could contribute that if people think it would be useful. I created a branch of x/sys in my local copy and have the code in x/sys/term.

[peterh]

With respect, I disagree that there isn't enough terminal handling to warrant its own subrepo. The termcap/terminfo parser alone would probably be a decent size, in addition to all the other things already mentioned. Plus all the os-specific versions of the interfaces for output (colours/styles/cursor positioning) and input (Ctrl-, Alt-, Fn-, Arrow-, etc keys) on non-Unix-like OSes.

I looked at the links in the proposal, and (like my own liner) none appear to support terminfo (or... is there a Plan 9 equivalent?). That's probably because everybody independently figured it wasn't worth the effort to write a terminfo parser for a single project. If the library and the effort are going to be shared, I'd be more likely to help.

[gdamore]

If we go beyond just termios, into terminfo parsing and so forth, yes, this
would be quite a bit bigger.

I wrote a fairly complete terminfo parser in tcell see
github.com/gdamore/tcell

(Its a little special purpose, because I don’t pick up all terminfo
keywords for storage, because most terminfo capabilities are useless in
modern terminals. But I have a full parser and expander, including a
complete implementation of TParm and TGoto.)

The low level half of tcell (basically everything in the base tcell
package) is probably what you’d want if you’re going to be doing rich
terminal work. This API may still be a little higher level than we’d like
to express in the core Go library though. (For example it has fairly rich
color mapping support, mapping any RGB value into the supported colors of
the terminal, including best-match color on 256 color terminals.) The
entirety of the base tcell package is about 7KLOC, plus a few KLOC for the
preparsed embedded terminfo database. (Basically I precompile the most
likely terminals into the code in an attempt to minimize dependencies on
external database files.)

On Fri, Oct 30, 2015 at 11:52 AM, Peter Harris notifications@github.com
wrote:

With respect, I disagree that there isn't enough terminal handling to
warrant its own subrepo. The termcap/terminfo parser alone would probably
be a decent size, in addition to all the other things already mentioned.
Plus all the os-specific versions of the interfaces for output
(colours/styles/cursor positioning) and input (Ctrl-, Alt-, Fn-, Arrow-,
etc keys) on non-Unix-like OSes.

I looked at the links in the proposal, and (like my own liner
https://github.com/peterh/liner) none appear to support terminfo (or...
is there a Plan 9 equivalent?). That's probably because everybody
independently figured it wasn't worth the effort to write a terminfo parser
for a single project. If the library and the effort are going to be shared,
I'd be more likely to help.

—
Reply to this email directly or view it on GitHub
#13104 (comment).

[frederich]

I quite agree like Brad x/os.

[gdamore]

One problem I have with x/os, is that if we think of this functionality as
“os-specific”, then it becomes more likely that there is drift between the
interfaces.

That is, I agree that the implementation is OS-specific. But having
OS-specific APIs is highly undesirable, and is devastating for
application portability. (Indeed, this problem is exactly why things like
POSIX and the standard C APIs exist …)

So while the location is not terribly important to me — especially if it is
just base functionality like termio handling, if the location means that
its likely that the APIs will not be compatible across different ports, I
consider that a failure.

On Fri, Oct 30, 2015 at 12:50 PM, Jens Frederich notifications@github.com
wrote:

I quite agree like Brad x/os.

—
Reply to this email directly or view it on GitHub
#13104 (comment).

[rthornton128]

+1. I also agree that functionality needs to be as OS agnostic/unified as possible akin to the majority of the standard library.

Would a package like this also include higher level constructs akin to curses panel, menu and form or would that be a community extension with the standard library package being solely low level?

[davecheney]

Would a package like this also include higher level constructs akin to curses panel, menu and form or would that be a community extension with the standard library package being solely low level?

I would vote no to that. I think we'll be struggling with the scope as it is.

However, I would hope this package would provide all the raw materials needed to construct such a library.

[gdamore]

Indeed. I could use that functionality and I believe the work I'm doing in tcell today goes substantially towards addressing the other higher level needs. (With views, widgets and so forth.)

Sent from my iPhone

On Oct 30, 2015, at 10:35 PM, Dave Cheney notifications@github.com wrote:

Would a package like this also include higher level constructs akin to curses panel, menu and form or would that be a community extension with the standard library package being solely low level?

I would vote no to that. I think we'll be struggling with the scope as it is.

However, I would hope this package would provide all the raw materials needed to construct such a library.

—
Reply to this email directly or view it on GitHub.

[techtonik]

It would be also nice if the package exposed performance metrics // tests both for terminal and for windows-like console.

[davecheney]

@adg @rsc ping

The x/time and x/sync repos have been approved, what are your thoughts on adding an as yet to be finalised, repository for terminal/serial/tty type packages ?

[adg]

I think x/os is probably the way forward.

[davecheney]

@adg golang.org/x/os sounds good to me !

[kim-racktop]

As long as we strive for an API that is consistent across the different OS's, sounds ok to me.

[gdamore]

Yes please!

That would NOT be consistent with the other stuff in x/os however. Mostly
x/os seems to be a kind of dumping ground for OS-specific functionality,
rather than any effort to provide a unified portable interface. For this
reason, I’m much more in favor of an x/term (or probably better, x/tty,
which would help keep the scope from ballooning into terminfo, curses, or
anything like that).

On Wed, Nov 4, 2015 at 11:47 AM, Kim Shrier notifications@github.com
wrote:

As long as we strive for an API that is consistent across the different
OS's, sounds ok to me.

—
Reply to this email directly or view it on GitHub
#13104 (comment).

[adg]

That would NOT be consistent with the other stuff in x/os however. Mostly
x/os seems to be a kind of dumping ground for OS-specific functionality,
rather than any effort to provide a unified portable interface.

Some unix-isms crept in for historical reasons, but on the whole the os package is in fact a platform-agnostic interface to the operating system.

[techtonik]

Also, x/term and x/tty are already unixisms.

[davecheney]

I agree, but this that is unavoidable.

I'm sure there will also be packages that only make sense for the Windows command subsystem.

I hope that in addition to these operating system specific packages we can develop high level ones

For example, console control providing cursor control for text mode interfaces, Garrett has shown that this is possible for Windows with the same high level api.

Similarly I want to see a high level serial Comms package that operates across Linux, Mac and Windows. It's exported api should be agnostic, but will probably call on operating system specific packages to provide the implementation. See my github.com/pkg/term package.

On 5 Nov 2015, at 19:07, anatoly techtonik notifications@github.com wrote:

Also, x/term and x/tty are already unixisms.

—
Reply to this email directly or view it on GitHub.

[techtonik]

@davecheney Windows console is more powerful/responsive in terms of user experience, because it provides a separate channel for controlling console window object (independent orthogonal API from C standard streams), while in terminal it is all garbled into one stream, which is slow and inefficient (and requires complex state machine to maintain). So decide carefully based on data if you need this Comms port and why - maybe there is a better way..

[rsc]

Proposal accepted. Assigning to @adg for subrepo creation.

[gdamore]

Sounds perfectly reasonable I think. Are we at consensus finally?

Sent from my iPhone

On Nov 12, 2015, at 11:55 AM, Dave Cheney notifications@github.com wrote:

We're only creating one subrepo for this code. Subdirectories like
x/term/tty and x/term/prompt are fine (and expected).

Yes, this was the design I had in mind.

—
Reply to this email directly or view it on GitHub.

[kim-racktop]

I am not exactly sure what we have consensus on other than the repo name. I do think we should factor out things like the Posix tc* routines into their own subdirectory/package. I agree that something like a full curses replacement is outside the scope of this repository. I am not exactly sure where we should draw the line between these two extremes. What I am hoping to see is that whatever we put into the packages inside this repo isolates me from needing cgo or needing os specific code.

[davecheney]

Hi Kim,

A cursors interface is out of scope, both by the original proposal, and also because none of the underlying plumbing exists.

I suggest we work on that plumbing and we'll probably find the natural boundary around the level of a type that represents the attributes of a tty or console; its echo state, its raw dimensions, possibly its speed (if necessary). Anything above that is probably out of scope for this package

[alexbrainman]

Here alexbrainman/winapi@2657113 are some serial port related APIs.

Alex

[techtonik]

@cznic about performance of Linux terminal vs Windows console. As I said, it is my perception that Gnome terminal (which is default on Ubuntu) is slow as hell. And quick search reveals that Linux terminals are indeed slow https://unix.stackexchange.com/questions/41225/can-a-terminal-emulator-be-as-fast-as-tty-1-6

As for your "windows console slow" the first thing I've got is this https://stackoverflow.com/questions/14295570/why-is-console-animation-so-slow-on-windows-and-is-there-a-way-to-improve-spee and there user runs "CLS" external process for every frame. As for https://groups.google.com/forum/#!msg/golang-codereviews/hq63SCACMJo/KL97UzWeEQAJ which says The Windows console is extreemly slow so the extra syscalls won't be noticable. phrase, then there are two points:

1. there are no any reference to this claim
2. the context in which it was mentioned is that Go doesn't use WinAPI's WriteConsole which probably should be used if you want performance on Windows

If you have any idea how to measure FPS of Linux terminal vs Windows console, I'll be happy to run tests on my machine. Please also note that stdout.print() performance is not the only parameter for measuring speed - keyboard response equally matters (if not even more important).

[techtonik]

It sounds like this repository will contain code for (1) dealing with consoles/ttys/ptys, (2) termcap-style escape sequence generation and parsing, and (3) code for configuring serial ports. It's a bit of an odd mix, but it's certainly a common grouping historically. Is that list correct, @davecheney?

I'd say 2 and 3. Because I believe that dealing with windows console is impossible in the same way as ttys and ptys.

[techtonik]

Right. GetConsoleMode() and SetConsoleMode() are the functions I'd
probably want to use when building this Windows analog.

WinAPI contains stateless getch() and kbhit() functions that are more clean and I think more useful. See this implementation in Python
https://bitbucket.org/techtonik/python-pager/src/2936adb19c9407b8e890949f9e88bf4f0461a524/pager.py?at=default&fileviewer=file-view-default#pager.py-190

Still there is a big problem how to get exact keycode indifferent of language layout and encoding. Something kind of readKey() function. For some reason I think it is impossible on Linux, but it would be nice if Go could provide an interface for that once kernel is patched to forward that information to active user space process.

[techtonik]

And JFYI, I am working on Windows client for LXC (yes, Linux Containers) and we've just committed ANSI colors support there with the help of external package that parses them and now I am looking how to read arrow keypresses to forward to remote server - lxc/incus#1317

[gdamore]

Um. I did this. (Get a key etc.) Windows actually has a pretty sane API,
and it does map reasonably well. The console modes can be used to
construct a fair facsimile of POSIX termios, and for richer console I/O
there are the ConsoleOutputCharacter(), ConsoleOutputAttribute(),
ReadConsoleInput(), etc.

These don’t play entirely perfectly well with simple input/output
(stdin/stdout), but using them its perfectly possible to build a curses or
termbox API. This has indeed been done multiple times.

I recommend you have a look at my tcell package, this file in particular:
https://github.com/gdamore/tcell/blob/master/console_win.go

On Tue, Nov 17, 2015 at 3:24 PM, anatoly techtonik <notifications@github.com

wrote:

And JFYI, I am working on Windows client for LXC (yes, Linux Containers)
and we've just committed ANSI colors support there with the help of
external package and now I am looking how to read arrow keypresses to
forward to remote server - lxc/incus#1317
lxc/incus#1317

—
Reply to this email directly or view it on GitHub
#13104 (comment).

[gopherbot]

CL https://golang.org/cl/17020 mentions this issue.

[adg]

The term repo is now created.

[techtonik]

Great! =) Now how can I track all issues there? (and also, is it already possible to use it right from the start in Go 1.5.x?)

[adg]

We still use a centralized bug tracker. Related bugs should have the prefix 'x/term'.

[techtonik]

Okay. Github doesn't allow to subscribe to that. So the only strategy I see is to create all the issues myself. =0

[Julio-Guerra]

• posix ttys and ptys
• serial termios control, like terminals and rs232 devices.

@davecheney @adg I think this package shouldn't expose a unix-like TTY interface but should rather build something smarter and cleaner on top of it. By separating concerns, it should then be even easier to port this package on non unix-like OSes. TTY and PTY are low-level POSIX/*NIX interfaces that should go to x/sys/unix like termios partly is already (the structure only so far)...

In my opinion, this package should only provide an abstraction based on the canonical-mode API (line disciplines + terminal capabilities? + readline?). And make the most out of go's beautiful IO stack: this package should see a serial device as nothing more than a stream of bytes ([]byte) with hardware or software control-flow management.

It is commonly admitted that the TTY API is a messy melting pot of distinct concerns:

• line disciplines.
• serial device management including data transmission, control-flow/handshaking...
• non-canonical/raw mode for blocking/polling/timed reading modes.

Of course it made sense at the time it was built (the 80's is synonym of high space and time constraints), but only time constraints require special kernel attention nowadays (i.e. save time/be fast by letting a kernel handling special cases like clocking a serial line transmission or handling C-z signal asap)...

I even think the 100% user-land implementation in pure go could be experienced given the CPU/serial device speed ratio. This is for example what GNU readline does.