Expired LE cert on server renewed - agents don't connect

[D4V3M0NK]

• MC2 v1.1.30 on Ubuntu 20.04.6

Greetings Simon and the MC2 community as a whole - I trust that this finds you all well and healthy...

We use Lets Encrypt certificates (with SANs) on our proxy servers and then utilize the certUrl configuration option within MC2 (we have to have encrypted traffic behind the proxy servers too) to have the agents double check the front end and back end certs upon connection. We then have an automated process to renew the certs every 85 days, update the proxy servers and then update the file that certUrl points to. All has been working without issue for a couple of years.

We have a cert that will be expiring within the next couple of days. When I start up the MC2 server it states the following:

Loaded web certificate from "file:///path/to/myexternalcert.crt", host: "host.domain.com"
  SHA384 cert hash: 192cc6d2de69....
  SHA384 key hash: f345c0b0b6...

(Note the cert hash...) and all the agents connect, happy as Larry.

Now we generate a new LE cert, apply it to the proxy servers and as soon as we do that, remotes start falling off of the MC2 server's UI and the server begins to post messages about the agents certificate not matching:

Agent bad web cert hash (Agent:192cc6d2de != Server:f345c0b0b6 or 34ec5ba5b9),
Agent reported web cert hash:192cc6d2de69...

This makes sense - the agents have been using the "192cc6d2de" hash cert up until now and it appears that the proxy servers new LE cert has a SHA384 cert hash starting with f345c0b0b6. We've not yet updated the myexternalcert.crt file...

So, I now update myexternalcert.crt with the new cert and restart MC2, but to my surprise I'm getting the exact same hash being reported...

Loaded web certificate from "file:///path/to/myexternalcert.crt", host: "host.domain.com"
  SHA384 cert hash: 192cc6d2de69....
  SHA384 key hash: a7de60e49...

(I am assuming that this is the hash on the certUrl). The server loads, but none of the agents connect, they certainly don't come "live" within the interface and I no longer get the (Agent:192cc6d2de != Server:f345c0b0b6 or 34ec5ba5b9) messages... I got to thinking that perhaps they're cached on the agent for a short period of time, but I've left this new certificate loaded for at least 30 minutes, and still no joy. I've even checked openssl x509 -noout -dates -in myexternalcert.crt and the certificate is good until early next year, so I know that the cert has the latest information - but the fact that MC2 is still reading and showing the same SHA384 hash makes me a little confused.

The key hash does - however - change...

Can anyone point me in a direction to further my investigation? Unfortunately I have no access to the remotes so I can't see what they're reporting...

[si458]

you have an outstanding issue which shows you have an issue with lets ecnrypt creating a newer RSA key then meshcentral supports
#6266
have you checked/compares the new key to the old key? any values different?

[D4V3M0NK]

Hey @si458 ... yes, I've already reviewed that, but these are RSA keys (we're asking specifically for them) as opposed to ECDSA...

[si458]

`

Additionally, thanks for the info on the hash generation @si458 - however, what happens when there's multiple certificate start/stops within the certificate file? Does MC2 cater for that?

it only looks for the very FIRST certificate from the looks of it

MeshCentral/certoperations.js

Lines 556 to 567 in 1e2d736

	} else if (u.protocol == 'file:') {
	// Read the certificate from a file
	obj.fs.readFile(url.substring(7), 'utf8', function (err, data) {
	if (err) { func(url, null, hostname, tag); return; }
	var x1 = data.indexOf('-----BEGIN CERTIFICATE-----'), x2 = data.indexOf('-----END CERTIFICATE-----');
	if ((x1 >= 0) && (x2 > x1)) {
	func(url, Buffer.from(data.substring(x1 + 27, x2), 'base64').toString('binary'), hostname, tag);
	} else {
	func(url, data, hostname, tag);
	}
	});
	} else { func(url, null, hostname, tag); }

[D4V3M0NK]

Ok, agentsignLock - is that a config set on the agent itself, or on the server? I certainly don't see it in my config:

{
  "$schema": "http://info.meshcentral.com/downloads/meshcentral-config-schema.json",
  "settings": {
    "cert": "host.domain.com",
    "WANonly": true,
    "_trustedproxy": "CIDR of proxy servers in all applicable regions",
    "trustedproxy": ["10.100.30.0/24","10.100.40.0/24"],
    "cookieIpCheck": false,
    "plugins": { "enabled": true }
  },
  "domains": {
    "": {
      "title": "Title",
      "title2": "Remote Support Services",
      "TitlePicture": "bg2.png",
      "_welcomePicture": "bg.png",
      "welcomePicture": "bg.jpg",
      "welcomePictureFullScreen": true,
      "welcomeText": "{blurb}",
      "passwordRequirements": {
              "force2factor": true,
              "min": 12,
              "upper": 1,
              "lower": 1,
              "numeric": 1,
              "nonalpha": 1,
              "oldPasswordBan": 24
      },
      "autocomplete": false,
      "orphanAgentUser" : "admin@domain.com",
      "certUrl": "file:///path/to/myexternalcert.crt",
      "_ignoreAgentHashCheck": true
    }
  }
}

[D4V3M0NK]

what you need to do is check via the remote device what SSL is being served by visits the web panel in there web browser

The remotes are headless so no browsers available, but I could get the certificate using the command line - but I don't think that'll help much, I'm 100% sure that they'll see the one on the proxy server, that's it. AFAIK, there's no way of "mimicking" the agent to get the cert from the MC2 certificate file...?

[D4V3M0NK]

I installed the MC2 agent on my system using the older certificate and it connected without issue. With the newer certs installed, it also no longer connects, so mimicks my remotes, but I can see what the agent.log has, which isn't overly helpful (at least to me)

[D4V3M0NK]

I've now removed my system and reinstalled - the installation downloaded and installed what it needed to, but still no connection to the MC2 server...

[D4V3M0NK]

I've no idea if this has anything to do with anything, but earlier as I was inspecting the differences between the two certs, I did notice the following when reviewing the structure of the certificates in the web browser and the location of the Certificate Signature Algorithm:

• working cert (expires 2410): Certificate > Subject Public Key Info > Extensions > Certificate Signature Algorithm: PKCS #1 SHA-256 With RSA Encryption
• not working cert (expires 2501): Certificate > Subject Public Key Info > Certificate Signature Algorithm: PKCS #1 SHA-256 With RSA Encryption

Note that the Extensions section is no longer in the newer cert, but placed one level up in the structure.

[D4V3M0NK]

Also just found out how to log meshagent connections, so added controlChannelDebug=1 and restarted the agent. I see this in the /usr/local/mesh_services/meshagent/meshagent.log:

meshcore/agentcore.c:4299 (0,0) Attempting to connect to Server...
microstack/ILibParsers.c:10978 (0,0) Connecting to: wss://host.domain.com:443/agent.ashx
microstack/ILibParsers.c:10978 (0,0) Control Channel Connection Established [13]...
microstack/ILibParsers.c:10978 (0,0) Sending Authentication Data...
microstack/ILibParsers.c:10978 (0,0) Control Channel Disconnected [13]...
meshcore/agentcore.c:4299 (0,0) Attempting to connect to Server...
microstack/ILibParsers.c:10978 (0,0) Connecting to: wss://host.domain.com:443/agent.ashx
microstack/ILibParsers.c:10978 (0,0) Control Channel Connection Established [13]...
microstack/ILibParsers.c:10978 (0,0) Sending Authentication Data...

[D4V3M0NK]

I stop the service and then run it as a command line, and get a little more in stdout and the log file:
stdout:

Attempting to connect to Server...
AutoRetry Connect in 12442 milliseconds
Connecting to: wss://host.domain.com:443/agent.ashx
Control Channel Connection Established [13]...
Sending Authentication Data...
MeshAgent_AgentMode_IPAddressChanged_Handler(2)
Control Channel Disconnected [13]...
Mesh Server Connection Error [13]
Attempting to connect to Server...
AutoRetry Connect in 20009 milliseconds

log file:

microstack/ILibParsers.c:10978 (0,0) Control Channel Connection Established [13]...
microstack/ILibParsers.c:10978 (0,0) Sending Authentication Data...
microstack/ILibParsers.c:10978 (0,0) Control Channel Disconnected [13]...
meshcore/agentcore.c:3726 (0,0) Connection Error [0x15ee958, 5, [13]]...

microstack/ILibParsers.c:10978 (0,0) PLATFORM_TYPE: 1
microstack/ILibParsers.c:10978 (0,0) Running as Service: 0
meshcore/agentcore.c:4296 (0,0) Attempting to connect to Server...
meshcore/agentcore.c:4299 (0,0) Attempting to connect to Server...

[D4V3M0NK]

Ok @si458 - another element that hasn't helped is that our proxy servers weren't refreshing in the manner we were expecting, leading us in having mismatches between front end and back end certificates. Now that we're aware of this, we're able to keep an eye on ensuring they don't throw up red herrings...

So, I have now less than 24 hours to sort this out and I have a couple more comments / thoughts:

1. Is there any way to (temporarily) completely remove all certificate checking, just to get my remotes back online, allowing me my extended access to them? I'm nigh on convinced that the agents are hanging onto the (prior) cert, even though agentSignLock is not configured (which according to the full config is still set to false by default). FWIW, setting ignoreAgentHashCheck: true doesn't help my situation.
2. We have been able to connect two completely new (test) systems to the MC2 server, with the new cert loaded. This begs the question: why can we add new systems and not have the old (production) ones connect (again, supposition is that the production systems are hanging onto the older cert...)
3. Is there any way of removing the cert that the production systems use locally?

[si458]

Try

1. set tlsoffload: true
2. Set trustedproxy: 'reverproxyiphere'
3. Set certurl: 'https://mydomainnamehere.com'
4. restart meshcentral
5. set ur reverse proxy to talk to http://meshcentralip:443 NO HTTPS or TLS!

[D4V3M0NK]

Messages crossed!! I can't remove TLS from behind the proxy and - as you'll see from my latest edited comment - two new test systems are able to be installed and connect without issue

[si458]

@D4V3M0NK sadly the only option is the ignoreAgentHashCheck: false

This tells meshcentral to ignore any agent certificate checking.

If u are setting this and it's STILL not working, then the must be something wrong with ur reverse proxy talking to meshcentral

So only other thing to do is run meshcentral in a full debug mode node node_modules/meshcentral --debug and watch the logs and see if anything jumps out

[D4V3M0NK]

@si458 Putting MC2 into full debug mode, I get a ton of information - hits from my remotes - so there can't be an issue with the proxy.

A sample (IPs coerced for security) - again ignoreAgentHashCheck doesn't appear to have any effect, whether it be set or not. Does this give you any clue as to what's going on? (I wasn't sure if the "upgrade:websocket" followed by "connection: upgrade" meant anything in particular?)

$ node node_modules/meshcentral --debug
Missing Modules: archiver@7.0.1
Installing modules [ 'archiver@7.0.1' ]
NPM Command Line: /home/ubuntu/.nvm/versions/node/v20.15.0/bin/node /home/ubuntu/.nvm/versions/node/v20.15.0/bin/npm install --save-exact --no-audit --omit=optional --no-fund archiver@7.0.1
Missing Modules: archiver@7.0.1
Installing modules [ 'archiver@7.0.1' ]
NPM Command Line: /home/ubuntu/.nvm/versions/node/v20.15.0/bin/node /home/ubuntu/.nvm/versions/node/v20.15.0/bin/npm install --save-exact --no-audit --omit=optional --no-fund archiver@7.0.1
MAIN: Core module windows-amt is 636085 bytes.
MAIN: Core module linux-amt is 582008 bytes.
MAIN: Core module linux-noamt is 481472 bytes.
MAIN: Core module windows-recovery is 137824 bytes.
MAIN: Core module linux-recovery is 82212 bytes.
MAIN: Core module windows-agentrecovery is 61977 bytes.
MAIN: Core module linux-agentrecovery is 6365 bytes.
MAIN: Core module windows-tiny is 6305 bytes.
MAIN: Core module linux-tiny is 6305 bytes.
MeshCentral HTTP redirection server running on port 80.
AUTHLOG: Server listening on 0.0.0.0 port 80.
MeshCentral v1.1.30, WAN mode.
MAIN: Core module windows-amt is 636085 bytes.
MAIN: Core module linux-amt is 582008 bytes.
MAIN: Core module linux-noamt is 481472 bytes.
MAIN: Core module windows-recovery is 137824 bytes.
MAIN: Core module linux-recovery is 82212 bytes.
MAIN: Core module windows-agentrecovery is 61977 bytes.
MAIN: Core module linux-agentrecovery is 6365 bytes.
MAIN: Core module windows-tiny is 6305 bytes.
MAIN: Core module linux-tiny is 6305 bytes.
DISPATCH: AddEventDispatch [ 'server-shareremove' ]
DISPATCH: AddEventDispatch [ '*' ]
DISPATCH: DispatchEvent [ '*' ]
MAIN: Server started
MeshCentral Intel(R) AMT server running on host.domain.com:4433.
AUTHLOG: Server listening on 0.0.0.0 port 4433.
HTTPS: Server listening on 0.0.0.0 port 443.
MeshCentral HTTPS server running on host.domain.com:443.
Loaded web certificate from "file:///home/ubuntu/mq9/meshcentral-data/myexternalcert.crt", host: "host.domain.com"
  SHA384 cert hash: 192cc6d2de69c43399e7a4ec4f65f47e2cc2ab0fe35c0e5bdd01426b2203ceef68792e1109d7958521724b4deb54773b
  SHA384 key hash: a7de60e49fbd9d101d8172490c298fefdcd5ef76e2b9f74d3db49c470a794e9610668530325a70559d905ea227b4437b
HTTPHEADERS: GET /agent.ashx/.websocket {
  host: 'host.domain.com',
  'x-real-ip': '11.21.31.41',
  'x-forwarded-for': '11.21.31.41',
  'cf-connecting-ip': '11.21.31.41',
  'x-forwarded-host': 'host.domain.com:443',
  'x-forwarded-proto': 'https',
  upgrade: 'websocket',
  connection: 'upgrade',
  'sec-websocket-key': 'JVT3FMR9BwG+uRQ2C4GRBA==',
  'sec-websocket-version': '13'
}
WEBREQUEST: (11.21.31.41) /agent.ashx/.websocket
AGENT: New agent at 11.21.31.41:48546
HTTPHEADERS: GET /agent.ashx/.websocket {
  host: 'host.domain.com',
  'x-real-ip': '10.20.30.40',
  'x-forwarded-for': '10.20.30.40',
  'cf-connecting-ip': '10.20.30.40',
  'x-forwarded-host': 'host.domain.com:443',
  'x-forwarded-proto': 'https',
  upgrade: 'websocket',
  connection: 'upgrade',
  'sec-websocket-key': 'vs1vsVCrkcCvfjkfZYHbUg==',
  'sec-websocket-version': '13'
}
WEBREQUEST: (10.20.30.40) /agent.ashx/.websocket
AGENT: New agent at 10.20.30.40:53734
AGENT: Verified agent connection to fvtGX@pYPCkkfRsIMvn5lGNKXDO9BZtePr1YR4KEhI007RCdQnFJDTyiRbslZYUn (11.21.31.41:48546).
HTTPHEADERS: GET /agent.ashx/.websocket {
  host: 'host.domain.com',
  'x-real-ip': '1.2.3.4',
  'x-forwarded-for': '1.2.3.4',
  'cf-connecting-ip': '1.2.3.4',
  'x-forwarded-host': 'host.domain.com:443',
  'x-forwarded-proto': 'https',
  upgrade: 'websocket',
  connection: 'upgrade',
  'sec-websocket-key': 'tvvQXkZGI5nBLSFZ8R1j6g==',
  'sec-websocket-version': '13'
}
WEBREQUEST: (1.2.3.4) /agent.ashx/.websocket
AGENT: New agent at 1.2.3.4:39098
HTTPHEADERS: GET /agent.ashx/.websocket {
  host: 'host.domain.com',
  'x-real-ip': '101.201.301.401',
  'x-forwarded-for': '101.201.301.401',
  'cf-connecting-ip': '101.201.301.401',
  'x-forwarded-host': 'host.domain.com:443',
  'x-forwarded-proto': 'https',
  upgrade: 'websocket',
  connection: 'upgrade',
  'sec-websocket-key': 'USj0cDNGgVyCd7ySPbkR4g==',
  'sec-websocket-version': '13'
}
WEBREQUEST: (101.201.301.401) /agent.ashx/.websocket
AGENT: New agent at 101.201.301.401:39104
HTTPHEADERS: GET /agent.ashx/.websocket {
  host: 'host.domain.com',
  'x-real-ip': '100.200.300.400',
  'x-forwarded-for': '100.200.300.400',
  'cf-connecting-ip': '100.200.300.400',
  'x-forwarded-host': 'host.domain.com:443',
  'x-forwarded-proto': 'https',
  upgrade: 'websocket',
  connection: 'upgrade',
  'sec-websocket-key': 'dXZQDRkh9INLa/YB/Cw+7g==',
  'sec-websocket-version': '13'
}
WEBREQUEST: (100.200.300.400) /agent.ashx/.websocket
AGENT: New agent at 100.200.300.400:48606
AGENT: Verified agent connection to hAtXQl0fJECC8kCfOUSsmOh1qL1qjiy@IjWtiyXxgH5YrD40bx27TP90jlTJWfDg (10.20.30.40:53734).
AGENT: Verified agent connection to qFikB1waKwtkEXpZbhmeAtD5knyXWMpdnTxvqrUhJx8chq2Y$AzrpYTesPzn7rhW (101.201.301.401:39104).
HTTPHEADERS: GET /agent.ashx/.websocket {
  host: 'host.domain.com',
  'x-real-ip': '10.20.30.40',
  'x-forwarded-for': '10.20.30.40',
  'cf-connecting-ip': '10.20.30.40',
  'x-forwarded-host': 'host.domain.com:443',
  'x-forwarded-proto': 'https',
  upgrade: 'websocket',
  connection: 'upgrade',
  'sec-websocket-key': 'INIyehDog02gqNpXc+ZgFw==',
  'sec-websocket-version': '13'
}
WEBREQUEST: (10.20.30.40) /agent.ashx/.websocket
AGENT: New agent at 10.20.30.40:53764
HTTPHEADERS: GET /agent.ashx/.websocket {
  host: 'host.domain.com',
  'x-real-ip': '10.20.30.40',
  'x-forwarded-for': '10.20.30.40',
  'cf-connecting-ip': '10.20.30.40',
  'x-forwarded-host': 'host.domain.com:443',
  'x-forwarded-proto': 'https',
  upgrade: 'websocket',
  connection: 'upgrade',
  'sec-websocket-key': 'Jy6AlTSIe++WdMhczPscvw==',
  'sec-websocket-version': '13'
}
WEBREQUEST: (10.20.30.40) /agent.ashx/.websocket
AGENT: New agent at 10.20.30.40:53766
AGENT: Verified agent connection to fw$nOH7eiuMmm91D3Wm0G3XBYtQvI$k@miIE6IosoTaiB7BLOgbTqqNCab7J8dli (1.2.3.4:39098).
AGENT: Verified agent connection to 9zWWq58LZK@O2YML$XGNa8ugE711VohLB0dKPXcy3FswCNlngsTXlSJWrdP6yQml (10.20.30.40:53764).
AGENT: Verified agent connection to qlXFPiU$RKllQIc@D7aK$pLSyOZwSsF8z2JWelN2bxW$4ILvxWdbhs1bzPXNZ9Zv (10.20.30.40:53766).
AGENT: Verified agent connection to BhEn5oshxP0isKD5tP9FRtAP0o5EtkLFXu9GZGSNhGSEH76W6lZu7vcCKATYu97K (100.200.300.400:48606).
DISPATCH: DispatchEvent [
  'node//BhEn5oshxP0isKD5tP9FRtAP0o5EtkLFXu9GZGSNhGSEH76W6lZu7vcCKATYu97K',
  '*',
  'mesh//FDWOKKvfQmV8BGbZj9OvYakz4OVHCoq1CbCMYz3zYnetxw2iX5Op9tjQ476x4kam'
]
DISPATCH: DispatchEvent [
  'node//BhEn5oshxP0isKD5tP9FRtAP0o5EtkLFXu9GZGSNhGSEH76W6lZu7vcCKATYu97K',
  '*',
  'mesh//FDWOKKvfQmV8BGbZj9OvYakz4OVHCoq1CbCMYz3zYnetxw2iX5Op9tjQ476x4kam'

[si458]

I also see Verified agent connection to so 4 hosts connected without any issues and showed online

So maybe the hosts that can't connect have a dns issue maybe or a certificate issue? I would try connecting to those machines that can't connect with the new cert and use something like curl -v https://mydomain.com and see what ssl gets returned

[cajund]

Hi @si458,

I'm working with @D4V3M0NK on this while he is on a plane. Thanks for your help thus far.

On our proxy, we are seeing this:

2024/10/18 18:34:44 [error] 20839#20839: *87816 connect() failed (111: Connection refused) while connecting to upstream, client: 208.89.120.247, server: meshcentral.domain.com, request: "GET /agent.ashx HTTP/1.1", upstream: "[https://10.100.100.61:443/agent.ashx](https://10.100.100.61/agent.ashx)", host: "meshcentral.domain.com"

Does this offer any clues? It seems that this would reinforce the theory of a cert issue. Also, as best as I can tell, the CPU is thrashing on this machine.

Please keep in mind that new machines added to the sever ping in and connect with no problems. It's the already set up machines that are not connecting.

Thanks.

[cajund]

OK, seeing connection refused when the server isn't running. So nothing new there.

[cajund]

Some addition info (I'm nit certain these are all part of the same request):

Successful:

HTTPHEADERS: GET /agent.ashx/.websocket {
  host: 'meshcentral.domain.com',
  'x-real-ip': '71.163.31.63',
  'x-forwarded-for': '71.163.31.63',
  'cf-connecting-ip': '71.163.31.63',
  'x-forwarded-host': 'meshcentral.domain.com:443',
  'x-forwarded-proto': 'https',
  upgrade: 'websocket',
  connection: 'upgrade',
  'sec-websocket-key': '9QxuDwVcbd1W60yF4ZlSSp==',
  'sec-websocket-version': '13'
}
WEBREQUEST: (71.163.31.63) /agent.ashx/.websocket
AGENT: New agent at 71.163.31.63:58758
AGENT: Verified agent connection to o4E68J2F2JfIGDP6e8OqamA2eKbAmCR5OEEkDUDQCk2BcDjS4XYrMYCsrquJzFzh (71.163.31.63:58758).
DISPATCH: DispatchEvent [
  'node//o4E68J2F2JfIGDP6e8OqamA2eKbAmCR5OEEkDUDQCk2BcDjS4XYrMYCsrquJzFzh',
  '*',
  'mesh//FDWOKKvfQmV8BGbZj9OvYakz4OVHCoq1CbCMYz3zYnetxw2iX5Op9tjQ476x4kam'
]
DISPATCH: DispatchEvent [
  'node//o4E68J2F2JfIGDP6e8OqamA2eKbAmCR5OEEkDUDQCk2BcDjS4XYrMYCsrquJzFzh',
  '*',
  'mesh//FDWOKKvfQmV8BGbZj9OvYakz4OVHCoq1CbCMYz3zYnetxw2iX5Op9tjQ476x4kam'
]
DISPATCH: DispatchEvent [
  'node//o4E68J2F2JfIGDP6e8OqamA2eKbAmCR5OEEkDUDQCk2BcDjS4XYrMYCsrquJzFzh',
  '*',
  'mesh//FDWOKKvfQmV8BGbZj9OvYakz4OVHCoq1CbCMYz3zYnetxw2iX5Op9tjQ476x4kam'
]
AGENT: Agent disconnect o4E68J2F2JfIGDP6e8OqamA2eKbAmCR5OEEkDUDQCk2BcDjS4XYrMYCsrquJzFzh (71.163.31.63:49852) id=29

Failed:

HTTPHEADERS: GET /agent.ashx/.websocket {
  host: 'meshcentral.domain.com',
  'x-real-ip': '152.129.43.1',
  'x-forwarded-for': '152.129.43.1',
  'cf-connecting-ip': '152.129.43.1',
  'x-forwarded-host': 'meshcentral.domain.com:443',
  'x-forwarded-proto': 'https',
  upgrade: 'websocket',
  connection: 'upgrade',
  'sec-websocket-key': '/SLXr+JtS7LBqGdahLn/Hw==',
  'sec-websocket-version': '13'
}
WEBREQUEST: (152.129.43.1) /agent.ashx/.websocket
AGENT: New agent at 152.129.43.1:58682
AGENT: Verified agent connection to bWQLwto5UDXUj3onj9@Ov0Bp$PNzMmkYcvSo2UbN4TAt23m6eOaXyWa8tquCuxr8 (152.129.43.1:58682).
AGENT: Agent disconnect u@ncsNXyDWo@OuGyLbwwhBH905AUKA8goFBklmSeybtL@mPLXymGvsOuwvZr7bIy (152.129.43.1:51248) id=Unknown

These guys are definitely timing out, as the failures tend to arrive in the log later on. So it doesn't seem there are any clues in this log.

We are going to try to reboot one of the production agents. Then we will try a reinstall of the agent. But this seems pretty extreme.

Is anyone on your end available for real-time assistance? We are under the gun.

Thanks,

[si458]

im literally not sure, ur config.json above looks correct, so im not sure why its not letting some comps connect and some not?
sadly im ill at the moment (flu time) so im having to go bed early so i cant stay awake,
im happy to look into it in the morning tho for you!
please send me an email (check github profile) and i can possibly connect into ur environment and have alook

[cajund]

OK, thank you, feel better...

[si458]

@cajund partner said do it tonight instead of 2moz as forgot about things, so if u email me or discord me (si458), I can have a look with u, might need remote access someway somehow

[cajund]

Hi All, while this continues to remain a mystery, we are out of the woods. The solution was to reinstall the agents. But it was a bit of a juggling act:

The issue was not with MC or with the Proxy, it seems. But, we were able to determine that switching back and forth between the two certs caused the agents to register and deregister with the cert. Supplemented with the fact that newer agents would connect as expected, we decided to do the following:

• Switch to the old cert
• Upload a script set to run at a specific time (say 10 minutes hence).
• Switch back to the new cert
• Wait

The script would uninstall and reinstall the agent at a specific time to register with MC via the new cert. This indicates that the agents were "caching" some kind of reference to the old certificate, and wasn't responding to the fact that the certificate had changed. This behavior also survived a restart of the machine intended to clear out any memory artifacts (one note: I'm 100% confident that the restart test took place, so don't rely too much on that - they said they restarted it...).

We stand ready to assist in any troubleshooting, so please reach out with any questions. Many thanks again to @si458 for his invaluable assistance and going above and beyond with this matter. Couldn't have resolved with without you.

Cheers!