Problem uploading files

[Benemerit]

Hello bozimmerman

First, congratulations on the "invention". It has been wonderful to be able to follow your project and create a modem for the Amstrad PPC (and the 486), and to be able to connect to an endless number of BBS.
I am writing to you because I have been trying many times to upload a file to the Amstrad BBS (amstrad.simulant.uk), but I am not able to.
I have tried two applications (Kermit and Telix), and in the first I saw that the upload protocol was not supported by the BBS. But in the second application (telix) I have seen the possibility of uploading via 4 protocols: zmodem, xmodem, ymodem and ymodem-g. I have tried all 4, but there is no way to upload a binary file. In all of them the upload starts, but suddenly, randomly, it is cut off, it remains blocked, and after a few seconds it gives an error. I have tried multiple configurations and even changing the FLOW, but nothing, there is no way.

I thought I read that there is some problem with uploading files.

The board is "Lolin V3 ESP8266 ESP-12F". As I say, I connect and navigate the BBS well (many of them). I even download files well, but the upload, no matter how many times I have tried, I can never finish it successfully.

I thought I read something similar in other forums with your firmware, but I'm not 100% sure.

Does it occur to you what I might be doing wrong and/or what I can try?

Thank you!

[bozimmerman]

Try using ATD to connect to the BBS instead of ATDT. Upcoming/later versions will not have this confusing issue by default.

[Benemerit]

THanks for your reply Bozimmerman.

I always use ATD . And i have a connection in the connection book configured, and telnet translation is off.
I have tried everything I could. I even thought that it could be the machine, due to low memory (although it wouldn't make sense, but oh well), and I tried again with the 486 and a more modern software for Win 3.11, in case I had more options and it gave me a clue.

File uploads always fail me. I have tried several types of files, in case it is something specific to syncronet for Linux (it is the amstrad.simulant.uk:464 BBS).

It's as if the chain it sends is either not complete, or has too much... I don't know.

I have uploaded a small video of the climb: https://www.youtube.com/watch?v=5TZz6gAz1u0

[bozimmerman]

Thanks, I'll try that BBS and see if I can reproduce.

[frnno967]

Not to pile on, but I have also had this issue with the Zimodem firmware for a few years now. Bo, this file transfer problem is what I mentioned to you that day of your talk at VCF Southwest 2023 if you recall. it just seems to either drop characters or somehow corrupt the data stream when the data starts flowing fast. Jim Drew mentioned that he thought it was an underlying problem with the espressif library code, but not sure how much, if any, of that you have implemented in Zimodem.
And it seems to be independent of file transfer protocol, as it doesn't work with any protocol I've tried. Thanks

[bozimmerman]

Well, it's definitely not an always-everywhere-broken type thing, as I was uploading files to my BBS just last night using Novaterm and X-Modem w/o issues. Maybe it's a baud rate issue, or flow control, or the size of the packets. shrug Won't know until I can reliably recreate it over and over again.
I did talk to Jim Drew once, but he explained that the issue he found was with Incoming packets (which would break downloads more than uploads of course). Unfortunately, I don't think Jim Drew's code is open source, so I wouldn't be able to learn anything from it anyhow. If I'm wrong on that, please LMK.

[Benemerit]

I don't know if it can help debug this error.
The truth is that it is frustrating, because the firmware is great in everything. Being able to connect to the old BBS from the old machines is wonderful.
So:
With XON/XOFF it usually increases more, but in the end it is always cut at almost 80% in the same way.
With RTS/CTS the rise is usually cut at around 10, 20%, just like without any flow control.
I have also tried with the ATS45 command but still the same.
I've tried so many things that I don't even know anymore... I think they're going to kick me out of the bbs XD
With x-modem, the BBS recognizes that something has arrived but always, in all cases, it thinks that a file of 128 bytes size has been uploaded. So it doesn't rise well either.
And with y-modem (with and without G), nothing, the file does not upload well either.
I have also done tests at 1200, 9600, 19200, 38400... etc... Slower but still, it cuts.

[Benemerit]

HI,

If it can help, i have take a new video trying to upload with my PPC and telix sofware with the modem set at 2400 Bauds (8n1).
https://www.youtube.com/watch?v=Rgi0ZSiADG8

[bozimmerman]

Thanks. I'm writing a bunch of local tests today that might hopefully reproduce the issue.

[bozimmerman]

Well, I added a test script (test.py) that does various upload and download simulations between the modem and a local socket server. For the most part, it seems reliable, but there are indeed times when the network packets are either slow to be sent or slow to arrive (always < 1 sec delay). This might be normal behavior, but I'll see if I can affect it, as it's the only issue I found.

Our BBS file transfer protocols were intended to go over phonelines where both sides are sending/receiving at the same speed, so that when, e.g. the 5th byte in a data grouping arrives, the receiver is guaranteed that the 6th will be immediately available right after it. TCP/IP packets are more -- irregular, and can split data up in different ways. I suspect that if the network protocols on online BBSes were coded in such a way as to assume more flexibility in timeouts, this would make them more reliable over the internet.

Anyway, I'll experiment and see if I can do anything to influence how the packets are assembled and/or split apart.

P.S. Most of my testing is being done on ESP8266 modems. What chip do yall use?

[Benemerit]

ESP8266, this (Lolin V3 ESP8266 ESP-12F):
https://cdn.shopify.com/s/files/1/1509/1638/files/NodeMCU_Lua_Lolin_V3_Modul_mit_ESP8266_12E_Datenblatt.pdf?342081239282763366

[djchez01]

Bo, I had reported a similar issue with being unable to upload binary files a few months back. I wonder if this is related. I was unable to do any tracking at the time and the issue expired. Is there any update on the testing you were going to do in your post in this thread on July 8?

[jgoerzen]

Hello! First, thanks for Zimodem -- it is fantastic!

Next, I am encountering this issue as well. I suspect it may be related to flow control or buffer size, exacerbated by slow upload performance.

I have been experimenting with Kermit, because it gives me a ton of control over all these things.

I'll note that Xmodem has a very small packet (128 bytes) and doesn't use sliding windows, so in general there is never data flowing in both directions at once (unlike ZModem or the more modern versions of kermit).

So let's look at my testcase. I am running a kermit server over TCP, without any customizations:

kermit -Y
set host /server * 12345

Then, connecting to the Zimodem:

kermit -Y
set line /dev/ttyUSB0
set parity none
set speed 57600
set flow rts
set carrier-watch off      [to permit me to talk to the modem before DCD is raised]
c                 [to talk to the modem]
ATD"192.[redacted]:12345"
CONNECT 57600

The default here starts with a small packet length (90 bytes, which grows up to 9K with successes) and a window size of 30 (so 30 packets can be in flight). This has the same symptoms described above -- occasionally some bytes make it, but things bog down and stop.

I'll first try no sliding windows: set window-size 1.

It let 5 packets through, but as Kermit's slow-start algorithm started raising packet sizes, by the time it got to 2K packets, 100% of them failed.

I wonder.... that happens to be just above the 1500 MTU / 1460 MSS. Perhaps there is a problem with splitting into fragments? Let's set that theory.

First, packets of 1400 bytes with window size of 1 should work:

set window-size 1
set send packet-length 1400

Nope, still failed after about 5 packets.

Let's try a smaller packet size:

set window-size 1
set send packet-length 1000

This works perfectly! Kermit reports a sending packet length of 1011 for some reason (not sure exactly why it's bigger than 1000 but it is). Oddly, watching the lights on the modem, there is considerable delay between the time the SD light is on (for packet transmission) and the ACK is received by the RD side).

By looking at a tcpdump on the receiving end, I have arrived at what I suspect is behind the very slow upload speed:

Every single byte of upload goes in its own TCP/IP packet.

That's a lot of processing overhead, network overhead, and size overhead (every byte uploaded sends 41 bytes of data across the network) Probably Zimodem needs a sort of buffer to add stuff too when it's receiving it at line rate.

However, this doesn't explain the reliability problem. My guess is that Zimodem is failing to exert flow control on upload, so some internal (perhaps 1K?) buffer is being overflowed by protocols that will send more than 1K of data at once (that includes pretty much anything supporting sliding windows, including Zmodem, and modern Kermit in its default configuration) It is also possible that the underlying buffer is smaller (say, 256 bytes) and Zimodem is able to keep up until the 1K barrier because the line rate of even 1-byte packets across my LAN is fast enough that 57600bps doesn't overwhelm it immediately.

Let me just see if I can rule out sliding windows as the problem. I should be able to reduce the packet size and increase the window size to make sure we still have less than 1K in the buffer. Let me try:

set window-size 4
set send packet-length 250

And yes, that still works.

One approach to solving this one-byte-per-packet problem is taken in ser2net; see the chardelay options beginning at https://github.com/cminyard/ser2net/blob/90e35f79bf811bf9b361b555f3decaea42456786/ser2net.yaml.5#L414 .

So I think there are two problems here:

1. Zimodem is sending packets with a single byte of payload on upload, causing slow performance
2. Zimodem isn't dropping CTS when its transmit buffer is full, causing buffer overrun somewhere

This also implies that Zimodem isn't going to work with PPP, because transmits will almost always exceed that 1K limit.

Thanks!