unhandled exception somewhere in outbound

[celesteking]

v2.8.25

Getting this unhandled exception inside outbound on stderr:

Error: read ECONNRESET
    at TCP.onStreamRead (internal/stream_base_commons.js:205:27) {
  errno: 'ECONNRESET',
  code: 'ECONNRESET',
  syscall: 'read',
  source: 'tls'
}

And in syslog:

May 26 18:00:50 out1 haraka[2378]: [INFO] [187B320F-821D-485D-8ACE-F32C18A026BA.2.1] [outbound] [6F3694DA-6C48-4EDE-8D95-35BEEFF924C8] Looking up A records for: d191392a.ess.barracudanetworks.com
May 26 18:00:50 out1 haraka[2378]: [INFO] [187B320F-821D-485D-8ACE-F32C18A026BA.2.1] [outbound] [6F3694DA-6C48-4EDE-8D95-35BEEFF924C8] Attempting to deliver {xx} to: 209.222.82.144:25 (0) (4)
May 26 18:00:50 out1 haraka[2378]: [INFO] [-] [core] [outbound] [outbound::25:209.222.82.144:xx:50] dispense() clients=1 available=0
May 26 18:00:50 out1 haraka[2378]: [INFO] [-] [core] [outbound] acquired socket DFF88964-CC5C-493A-8AB3-847CE5963EA5 for outbound::25:209.222.82.144:xx:50
May 26 18:00:50 out1 haraka[2378]: [INFO] [187B320F-821D-485D-8ACE-F32C18A026BA.2.1] [outbound] [DFF88964-CC5C-493A-8AB3-847CE5963EA5] initiating delivery
May 26 18:01:03 out1 haraka[2378]: [CRIT] [-] [core] Error: read ECONNRESET
May 26 18:01:03 out1 haraka[2378]: [CRIT] [-] [core]     at TCP.onStreamRead (internal/stream_base_commons.js:205:27)
May 26 18:01:03 out1 haraka[2378]: [NOTICE] [-] [core] Shutting down

So I'm wondering where that's happening. Thanks to source: 'tls' it looks like it's coming from either here or there, but where does it end up?

Update: Note, that I would usually receive [WARN] [-] [core] [outbound] Socket got an error while shutting down: Error: read ECONNRESET -- happening very frequently.

[msimerson]

Is your node version up-to-date? If it's a TLS issue, updating node (and thus openssl) often resolves it.

[celesteking]

of crse it is - v12.16.1

[baudehlo]

Any chance it's related to #2743 (which I note never got fixed)?

…

On Wed, May 27, 2020 at 3:39 PM celesteking ***@***.***> wrote: of crse it is - v12.16.1 — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#2807 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAFBWY4UXDYHM7JO563MIVLRTVUAJANCNFSM4NMMRQQQ> .

[celesteking]

It's unrelated to that, but might be related to the code/logic quality overall in that file. I just need to understand where it might be happening as we've got safenets everywhere.
btw, it happened again this night.

[msimerson]

Is this still an issue?

[celesteking]

I don't have 99cb9f9 applied, but there haven't been any crashes in the last month. Can't repro, I guess. And even if I were able to, I'd apply that patch before reopening.

[tabascoterrier]

I think I'm experiencing the same issue - I get occasional crashes with the log message:

haraka[259003]: {"level":"CRIT","uuid":"-","origin":"core","message":"Error: read ECONNRESET"}
haraka[259003]: {"level":"CRIT","uuid":"-","origin":"core","message":"    at TCP.onStreamRead (node:internal/stream_base_commons:217:20)"}

It's very inconsistent - sometimes weeks and a few million outbound mails go past with no issues, and then there are occasions where there have been tens of restarts in a day.

I was previously on Node v18, and just upgraded to the lastest v20 to see if that helped - but no, after several days it happened again.

[msimerson]

If you can narrow it down to a particular server, that would be useful. Exceptional issues like this are why the no-tls-hosts feature exists.

[tabascoterrier]

I appreciate it doesn't give you much to go on :) I've been trawling logs, looking at adjacent email events but have yet to find a particular smoking gun server. I'm beginning to suspect it's the MS 365 MXs occasionally doing wierd things, but I don't have what you'd call strong evidence.

I'll keep looking at it though, and I'll let you know if I can be more specific. I might try playing with netem or something similar locally to simulate some terrible network connections to see if I can reproduce it that way.

[lnedry]

Node v20.15.0
Haraka 3.0.3
Debian 12

I've had the same issue occur 11 times in the last 7 months.

What should I look for in the logs? Is there any additional logging that I can do to help track this down?