Problems with socket connections

[JoeNemo]

I am running clangd as an LSP server over a socket to remote server using socat. Has anybody else been able to get this to work. Running clangd locally is working perfectly fine, so I am pretty sure eglot and jsonrpc are installed correctly.

The connection to the remote server dies in the initialization handshake. That is, the LSP client (eglot) has sent the first JSON-RPC method and there is a TCP reset before a reply can get back to eglot.

EDIT

The proxied connection is now sending the initialize method reply, but jsonrpc is not parsing it. I have added a lot of tracing. I am a lisp programmer from the 80's and 90's so this is pretty easy to trace.

As I investigate this further, the proxied stream to clang is showing the Content-Length header with lines terminated by LF, whereas the subprocess connection to clangd is showing CR LF, which matchers the regexp in jsonrpc.el. I am tracing the proxy and I see the 0x0D, 0x0A pairs flowing through the TCP buffers, so I am really confused. Do emacs buffers do strange things with CR's and LF's. I feel like I am very close, but there's some configuration I am missing.

EDIT2 finally got a localhost packet capture in Wireshark showing property formed JSON-RPC headers. Something in Emacs process or buffer is losing some data.

clang-remote.zip

[joaotavora]

As I investigate this further, the proxied stream to clang is showing the Content-Length header with lines terminated by LF, whereas the subprocess connection to clangd is showing CR LF, which matchers the regexp in jsonrpc.el. I am tracing the proxy and I see the 0x0D, 0x0A pairs flowing through the TCP buffers, so I am really confused. Do emacs buffers do strange things with CR's and LF's. I feel like I am very close, but there's some configuration I am missing.

You are on to something, or at least so I recall, but you need to post a fool-proof reproduction recipe, so I (or anyone else) can help you with this. When you open an issue, don't you see an issue-reporting template? If you follow it, it should guide you to what things are generally desired.

Is either clangdor your Emacs process running under Windows?

[JoeNemo]

Sorry about not using the issue template. And yes, Emacs is running under Windows. Clangd is running under windows, too. But as a sub-process of a java program acting as a proxy. I will make a new issue that with these details. Meanwhile, I have a patch for jsonrpc.el that eliminates the problem, but I really dislike it. The RegExp used to match the Content-length header can be modified to also tolerate headers that have just a newline. I wish that I knew how to see the exact bytes flowing into the network processes buffer from TCP, and any magic that Emacs is doing relative to that.

`

 (setq expected-bytes
                        (or
			 (and (search-forward-regexp
			       "\\(?:.*: .*\r\n\\)*Content-Length: \
*\\([[:digit:]]+\\)\r\n\\(?:.*: .*\r\n\\)*\r\n"
			       (+ (point) 100)
			       t)
			      (string-to-number (match-string 1)))
                         (and (search-forward-regexp
			       "\\(?:.*: .*\n\\)*Content-Length: \
*\\([[:digit:]]+\\)\n\\(?:.*: .*\n\\)*\n"
			       (+ (point) 100)
			       t)

		      (string-to-number (match-string 1)))))

[joaotavora]

Somehwere in the process creation options there should be something to account for line endings, I seem to recal.

I will make a new issue that with these details.

No don't. Or maybe don't? We already have this conversation to track it.

[JoeNemo]

Ok, thanks. I have now displayed the buffer that holds the data associated with open-network-stream. The CR's are stripped there, and they are pressent when looking at Wireshark packet capture. So I am suspecting that the C code in Emacs internals is being "clever" with how the bytes make it from the underlying socket into a buffers.c buffer. What is interesting is that the pathway from running the LSP server as a subprocess, (inferior process) does not go down exactly the same path and the CR's are present.

This is sort of fun for me to figure out. I haven't programmed in LISP since 1992, but it's all coming back. If you know of any options to network/process/buffer management in Emacs whose behaviors vary by parameters or ifdef-ed macros in the C code that might help me. I would be proud to submit a patch to Emacs or one of its packages. I have been an Emacs user since 1985.

[JoeNemo]

I have a fix now that makes sense (at least to me). I don't seem to have rights to create a bug fix branch for the eglot repo, but maybe I am not understanding github.

Anyway, inside eglot.el there's a function called eglot--connect and it has a few different ways it makes a process to do JSON-RPC IO with the LSP server. One way is using open-network-stream at https://github.com/joaotavora/eglot/blob/master/eglot.el#L1045.

If this uses the same coding as the call to make-process a few lines down, things seem to work pretty well. Here's what I changed:

`
 `(:process ,(lambda ()
                               (let ((network-process (apply #'open-network-stream
							     readable-name nil
							     (car contact) (cadr contact)
							     (cddr contact))))
				 (set-process-coding-system network-process 'utf-8-emacs-unix)
				 network-process))))`

I think I need this would need to be tested against other network-oriented LSP implementations. I only used clangd, so it's hard to know. If this hasn't been noticed yet, it's probably that it is rare to use Windows and network-attached LSP servers.

[joaotavora]

Fantastic !yes, we're nearing a patch with the set-process-coding-system but this has to be well tested to be sure it doesn't break other language servers. Don't worry too much about PR (GitHub pull request) creation. If you want and you're comfortable with it, just post a diff of the patch. If you prefer and you like email-style bug-reporting (I do :-) ), you can also send email to bug-gnu-emacs@gnu.org with your latest conclusions and CC me in.

[joaotavora]

I think I need this would need to be tested against other network-oriented LSP implementations. I only used clangd, so it's hard to know. If this hasn't been noticed yet, it's probably that it is rare to use Windows and network-attached LSP servers.

Just read this part of your comment. Yes, I completely agree with the two things you say there.

Also note that the (cddr contact) coupled with open-network-stream 's docstrings indicates that you could theoretically fix this "user-side" by crafting an entry for eglot-server-programs with the suitable keyword parameters for the coding system.

Although I do prefer the patch idea. The clause below, which also uses network streams, would probably need it, too.

[JoeNemo]

Hi, I haven't forgotten this, but had to put this down for a few days to finish a (paying) work issue first.