Feature request - Serial over TCP

[mjkl-gh]

As promised:

I'm happy to look at any feature PRs, and your feature sounds useful. I certainly think the ability to view the serial output remotely, including in the UI would be a nice feature to have for debugging and monitoring. Others have expressed an interest in taking this further with "remote terminals" too.

In fact, I've attempted to do this in the past with limited success and there's some work in various branches which never made it's way into master. This thread has details of some of my attempts:

#123

I know that @proddy's project has developed a unified remote terminal over telnet/serial successfully, see project:

https://github.com/proddy/EMS-ESP

And I also know of projects such as the following which aim to support capture and transmission of Serial over TCP and/or WebSockets:

https://github.com/ayushsharma82/WebSerial
https://github.com/jeelabs/esp-link
https://github.com/JoaoLopesF/RemoteDebug

The second example (ESP-LINK) probably does the closest thing to what I was hoping to achieve.

I never reached an ideal solution, especially on the esp8266 which has such limited RAM. Failures such as dropped messages and buffer overflows were all too common for me, especially over WebSockets which always seems to have a very low transmission rate with the ESPAsyncWebServer. It worked much better on esp32 where there's more RAM.

I'm admittedly a C++ developer of only intermediate skill, and someone with more experience may be able to have a better crack at it, or at very least make code which fails more gracefully. I know what I wanted out of this feature, and I wonder how that fits in with what you are attempting to do. For me the key points were:

• Abstraction for logging - so logging can be redirected as required (freeing the Serial for use as a terminal if required).
• Persistent rolling logging to the FS - so a limited amount (configurable) of logs may be viewed later
• Secure - No insecure open ports

Perhaps this is too much of a wish list to be sensible... what are your thoughts/goals for this feature?

Many thanks for all the pointers. Some stuff I've already seen, some looks very interesting. I will definetly look into it

Unfortunately, I would class myself as a C++ developer with just enough knowledge to know what not to start on. Anyhow, my goals are somewhat similar to esp-link. As in create a simple serial to tcp bridge (and back). My ideas were:

• create a serialservice that opens a Serial.begin on a (configurable for esp32) pin.
• Allow for clients to be registered, returning a buffered stream object .
• create a stream_server that takes a stream object and transmits it over tcp

In my final application I want to utilize an m5stickc to read out my dutch utilities smart meter. All these smart meters come with a serial port which can be used to feed data into a smarthome application. I already have some working prototypes at friend's and family's houses.

The reason for requiring a serialservice is that I eventually also want to create an MQTT service for automatically logging the data as sensors to home assistant using autoconfigure. More,over include some internal sensors such as battery level. The reason for this is that some meters only supply 100ma from their ports so they need some efficiency tweaking.

[rjwats]

Thanks for this. I understand what you are looking to do, and I can certainly see the benefits of such a pattern and I think your buffered stream approah sounds good.

Are you thinking of this as an API extension to the framework or a feature which may be configured within the UI in addition to the APIs to support clients of the serial stream?

Regardless, finding a decent approach here could certainly help with the other requirements such as remote logging. I'll be slow to reply during the working week - generally evenings only.

[proddy]

I never really saw the need for pushing Serial over WiFi to be honest. In my view it would be easier to collect the data, perform any crunching, normalizing and push out the enriched data to the Web UI (via a socket), MQTT (as payload) or dump into a remove syslog (as log lines). Like Rick I hit the end of ESP8266 and struggled with its limited heap memory and fragmentation so switched exclusively now to the ESP32 which is a blessing I can tell you. I'm slowing working on tweaking esp8266-react to use esp32_https_server which brings another level of security with signed certs.

btw @mjkl-gh I did a small project for the p1meter based on esp8266-react which I never finished, including the circuit design which I can share.

[rjwats]

@proddy I think abandoning support for 8266 for certain features is probably a good idea for the framework. I can see why you would do that for given all the features you pack into your project.

Not wanting to get too off-topic on this thread. As it wasn't looking like TLS support is going to come to AsyncTCP any time soon - development there is more or less dead. Investigating a different set of libraries for the Web Server and MQTT (which also uses AsyncTCP and therefore can't support TLS either) was very high on my list - intrested to hear how you get on using an alternative WebServer lib - especially when it comes to timing critical stuff. I'm wondering a re-write on idf might be a better long term solution - their WebServer and MQTT implementations support TLS.

[rjwats]

If this feature is too much for the esp8266, we should be prepared to say it's ESP32 only.

[proddy]

If this feature is too much for the esp8266, we should be prepared to say it's ESP32 only.

yup, agree. how about a spanking new repo called esp32-react and I'll help you port it over to IDF, using the standard libraries. If we run into limitations we can always switch to any of the alternatives out there for web & mqtt. It's been on my todo list for a while also.

[mjkl-gh]

I never really saw the need for pushing Serial over WiFi to be honest. In my view it would be easier to collect the data, perform any crunching, normalizing and push out the enriched data to the Web UI (via a socket), MQTT (as payload) or dump into a remove syslog (as log lines).

Totally agree in the case of an endproduct or endgoal. In that case serial over tcp usually doesn't make sense if that's all you want you're probably better off programming it with esp-link and call it a day. A project like that doesn't need a nice interface as it will probably be used only once.

However, as there are a ton of serial devices in the world it is a good point to get started. Using ser2net one can use an esp as a virtual serial device on a Linux server across the building. And first explore the serial connection using python libraries for serial devices out there before moving to a (self-written) C++ implementation.

In case it wasn't clear. ( I didn't mention it in this issue). This is less of a Feature request and more of a "Hey I'm developing this, maybe you're interested too." After that it became apparent you guys already had some ideas on this.

---

On the esp32 part. I'm all for it. I must admit the only esp8266 I've got came in devices (tuya smart bulbs etc etc). I myself haven't tried IDF yet what are the advantages?

[rjwats]

I've not used it either other than a bit of tinkering. I'm imagining a few of the advantages might be:

• Developed & supported by Espressif
• Actively maintained
• The first SDK to receive bug fixes and new features (Arduino gets slow trickle-down updates)
• SDK provided client/server implementations
• SDK provided peripheral interface implementations

Feature wise, it's really complete: https://docs.espressif.com/projects/esp-idf/en/latest/esp32/api-reference/index.html

However, it's not written like Arduino code where the primary objective is being multi-system and developer friendly. The SDK provides a more fully featured but lower level abstraction. So that probably would put many people off and would immediately reduce the ecosystem of libraries and tools at your disposal - so there's a significant trade off.

[rjwats]

@proddy Yes, I think a collaborative effort and a clean(ish) slate would be good if going down the IDF route. I'm planning to spend some time this week learning the SDK and working with some of the SDK's examples. I see no reason why the SDK provided server, client and peripheral utilities wouldn't be sufficient. I'm not sure however if there will be many alternatives if working with IDF directly as the majority of the community maintained libraries are built for the Arduino core.

Send me an email if you are interested in discussing this - perhaps we could have a chat this weekend?