localtest.me connection issue on ARM Mac

[krachtstefan]

Hi Timo. Thank you so much for making this proxy publicly available. I used your example and it works great. I still have one general question that I could not figure out myself.

I'm not very familiar with Caddy, but in my understanding, I somehow have to configure it on my local machine to make this all work?

So I installed Caddy via brew, played around a bit, set up the neon proxy as described in your Readme and everything worked eventually. However, after a couple of hours, the npm run test fails with an error that tells me the host is not available anymore.

/Users/sk/Desktop/local-neon-http-proxy-main/node_modules/@neondatabase/serverless/index.js:1533
stringify(ye),headers:le,...D})}catch(oe){let $=new Ee(`Error connecting to data\
                                                ^
NeonDbError: Error connecting to database: fetch failed
    at execute (/Users/sk/Desktop/local-neon-http-proxy-main/node_modules/@neondatabase/serverless/index.js:1533:49)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async file:///Users/sk/Desktop/local-neon-http-proxy-main/sample.mjs:16:18 {
  code: null,
  sourceError: TypeError: fetch failed
      at Object.fetch (node:internal/deps/undici/undici:11576:11)
      at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
      at async execute (/Users/sk/Desktop/local-neon-http-proxy-main/node_modules/@neondatabase/serverless/index.js:1532:43)
      at async file:///Users/sk/Desktop/local-neon-http-proxy-main/sample.mjs:16:18 {
    cause: Error: getaddrinfo ENOTFOUND db.localtest.me
        at GetAddrInfoReqWrap.onlookup [as oncomplete] (node:dns:108:26) {
      errno: -3008,
      code: 'ENOTFOUND',
      syscall: 'getaddrinfo',
      hostname: 'db.localtest.me'
    }
  }
}

Node.js v18.18.1

Can you give me some basic instruction on what I have to do with my local caddy setup (like e.g. does the deamon have to run?). I also wonder why caddy is used in the first place, when the configured neon fetch endpoint is not even using https.

Sorry for asking dumb questions, I appreciate any help :)

[TimoWilhelm]

Hi Stefan, if you are using the Dockerfile or image from the GitHub Container Registry, you shouldn't need to install Caddy manually. It's included in the container image.

The reason Caddy is used as a reverse proxy is that the neon-proxy only accepts HTTPS connections with the self-signed certificate provided during startup.

To avoid having to accept the self-signed cert on your local machine for development, Caddy trusts it and proxies the connection over a plain HTTP connection while preserving the host header for neon.

It seems like the local proxy stopps working after some time in your case. Do you get any errors in your container logs?

[krachtstefan]

Thanks for your quick answer. I just retried from scratched and used the Dockerfile and the Image from the registry as well. Both now have this error (I wonder how I got this working in the first place :D)

There are indeed some errors in the logs.

Might this be a Mac/Apple Silicon issue?

[TimoWilhelm]

Thanks for the logs. I see the same error block in my logs about invalid region but this seems to be related to some AWS S3 functionality that should not prevent the usage of the proxy. The only difference I can see is the OpenTelemetry error gives a os error 111 in my logs instead of os error 99 in your logs.

OpenTelemetry trace error occurred. error sending request for url (http://localhost:4318/): error trying to connect: tcp connect error: Connection refused (os error 111)

Unfortunately I don't have access to an ARM-based machine to test if this is the reason it's not working but it is strage that it worked partially before.

Could you maybe try to clean up the neon_control_plane schema in your Postgres DB to see if that helps?

[krachtstefan]

Funny, that you say that. I was not aware that the DB was also used by the proxy itself. I found a small bug that can lead to state where the schema was not created.

Unfortunately, it still throws the same error.

[krachtstefan]

Update: I think it all boils down to the fact that my machine cannot resolve db.localtest.me. I'm not sure how or why my machine should be able to do that because I don't understand how Caddy is working. I would naivly think, that they just point everyting to 127.0.0.1.

But anyway, when I change the sample.mjs to the following the npm test works for me

import { neon, neonConfig } from '@neondatabase/serverless';

const connectionString = 'postgres://postgres:postgres@db.localtest.me:5432/main';

neonConfig.fetchEndpoint = (host) => {
  const protocol = host === 'db.localtest.me' ? 'http' : 'https';
  const port = host === 'db.localtest.me' ? 4444 : 443;
  const usedHost = host === 'db.localtest.me' ? 'localhost' : host;
  return `${protocol}://${usedHost}:${port}/sql`; // resolves to 'http://localhost:4444/sql' instead of 'http://db.localtest.me:4444/sql'
};

const sql = neon(connectionString);
const [result] = await sql`SELECT * FROM NOW()`;

console.log(result);

When I got the setup working the first time, I'm pretty sure that I hardcoded this in a similar way while I was debugging.

This is good enough for me, because it eventually works. Feel free to close this issue, except you want to investigate any further.

And again, thanks for this repo and for answering my questions so quickly!

[TimoWilhelm]

The localtest.me domain is not related to Caddy. It's a domain that should point all wildcard entries to 127.0.0.1 and it is used as the cert subject. I got that from the original issue this repo is based on.

The neon proxy does some SNI stuff and doesn't accept localhost from my testing.

The caddy reverse-proxy already sets the upstream host header but unfortunately it seems to be required to use the cert domain in the connection string, otherwise the proxy returns NeonDbError: invalid hostname: Common name inferred from SNI ('localhost') is not known.

This is kind of brittle but unfortunately I haven't found a workaround so far.

I wonder what causes the DNS to fail on your setup for that domain...

By the way, I tried publishing a multi-platform image for AMD64 + ARM64. It's not going to solve this issue and I mainly did it to understand how that is done with GitHub Actions 😅 It would be very helpful if you could give that a try sometime from ghcr.io/timowilhelm/local-neon-http-proxy:mutli-platform-image and let me know if it works at all.

[krachtstefan]

Thanks for clarifying. I just implied that Caddy owns that domain because it was linked on their website.

I finally got it sorted. The issue indeed was that db.localtest.me could not be resolved on my machine. My DNS was configured to be used from my router (good old FRITZ!Box), which is configured to be used from my provider (1&1). I changed it to 8.8.8.8 and 8.8.4.4 on my machine, restarted it. Everything works. The only thing I can think of is that my internet provider does not resolve all the localtest.me entries (readme.localtest.me resolves by the way...) for security reasons? I could imagine that there might be some phishing scenarios with this domain.

For anyone else having a similar issue, you want to make sure that nslookup db.localtest.me resolves to 127.0.0.1.

ghcr.io/timowilhelm/local-neon-http-proxy:mutli-platform-image works like a charm ♥️

[flexchar]

Just wanted to mention that if one is using Chrome, all domains ending with .localhost tld resolves to 127.0.0.1. So no need for host files etc. :)