Server Typescript Migration

[wolfendale]

Changes:

This migrates all serverside JavaScript files to TypeScript. I've tried to change as little as possible, but there are some changes to make types work out and some changes to code that must have been wrong / broken.

This also updates some eslint and TypeScript settings, and resolves some eslint errors, also adds dist to .gitignore

I have removed the webfiles projects from the scope of TypeScript compilation for now as they should likely be their own compiled projects

eslint

• Separates eslint configuration into 3 parts:
  1. eslint for server JavaScript files
  2. eslint for server TypeScript files (this extends 1. but includes extra TypeScript specific rules)
  3. eslint for projects in the webfiles directory
• Addresses all remaining eslint errors, most of which were simple
• A few of the eslint errors related to potential bugs that I've provided a fix for:
  1. this commit adds offset as a skip parameter to the mongo query, and updates the link sent as a response so that pagination should work properly. I assume this is what was intended
  2. src/utils.js was using sharp without importing it, so I've added an import for that

I have also disabled eslint for all projects in the webfiles directory temporarily as the eslint settings that were being applied were likely inappropriate. I have also included a separate eslint configuration file for these projects with an initial attempt as a sensible configuration. However, there are some questions before I can finish this:

1. Are these all one project or should they really have their own separate eslint configs? One reason I'm thinking this is that they all appear to use a variety of globals that I'm not sure are applicable to every one of the projects.
2. What version of ES should we be aiming for in each of these projects?

TypeScript

• Adds required types modules to allow the project to compile properly
• Changes TypeScript config to exclude webfiles sources

Tasks

• I have read and agreed to the Code of Conduct.
• I have read and complied with the contributing guidelines.
• What I'm implementing was an approved issue.
• I have tested all of my changes.

[jonbarrow]

First round of review by me. I will likely go over it again, it's late and I've done a lot of stuff in other repos as well. Going to ping @CaramelKat for her review as well.

Overall looks pretty good 👍 just a few nitpicks and some actual issues/suggestions/clarifications.

I will also note: in the checklist you say you haven't tested these changes. Is there a reason for that? Ideally this should all be tested, on console, before being merged.

Also; this has made me realize that this is a decently sized repository with a lot of moving parts, especially when trying to juggle support for 3 different platforms at the same time. It's resulted in some pretty janky code, dead code left over that isn't clear if it can be removed or not, and an overall not great experience working in the code base.

This is especially present in the first question asked here:

Are these all one project or should they really have their own separate eslint configs? One reason I'm thinking this is that they all appear to use a variety of globals that I'm not sure are applicable to every one of the projects.

This is a valid question. It feels like 3 different projects shoved into one box, and it's not going super well imo. I unfortunately don't have an answer at this time since currently they are one project, but I'd like to possibly change that.

What version of ES should we be aiming for in each of these projects?

For the consoles, I'd say ES5 at the latest. NWFX targets ES3, but I didn't have any issues when using ES5 in some testing. That being said, NWFX is a simpler code base than this.

For the browser, latest always imo.

We should really consider a better way to structure this project, or just separate it out entirely into 3 different ones and bite the bullet of some duplicated backend code. We already duplicate some backend logic as it is, since the miiverse-api server has pretty similar logic to some of these routes.

[wolfendale]

I will also note: in the checklist you say you haven't tested these changes. Is there a reason for that? Ideally this should all be tested, on console, before being merged.

Yeah, this absolutely needs testing. I am happy to do that but I don't know the best way to set this up for local testing - especially on console

[jonbarrow]

I am happy to do that but I don't know the best way to set this up for local testing - especially on console

All you'd need in this case is:

• An instance of the account server
• An instance of this server
• Probably an instance of the friends server but that can likely be skipped (though it should probably be done for the sake of testing, since some parts of this rely on the friends server)

@MatthewL246's Docker project can set this kind of thing up super easily https://github.com/MatthewL246/pretendo-docker

In my case when doing local testing all I do is:

• Hook the console up to a proxy server (I use Charles personally, but we also maintain a special version of mitmproxy for these consoles here https://github.com/PretendoNetwork/mitmproxy-nintendo)
• Disable Pretendo in Inkay (so that requests go to Nintendo)
• Use the proxy server redirection feature to send Nintendo's requests to my local server

However there's a variety of ways to connect locally. I mostly do it this way so that I can avoid setting up SSL certificates just for local testing (redirect from https (Nintendo) to http (local)).

You'll then need to create some server documents on the account server for the friends server (NEX) and Miiverse (service), and make some local PNIDs on your instance for testing.

That being said that only really works for connecting to most services locally. Apps like the eShop, NNID account settings, and importantly Miiverse are actually all just browser apps in a web view. They roll their own SSL, rather than using the system SSL, so they need additional patches. You'll need a custom build of https://github.com/PretendoNetwork/Inkay for this for 2 reasons:

1. When connecting to Pretendo is disable (like described above) then Inkay will not attempt to load the Miiverse patches
2. You need to use a different certificate than what is provided by Inkay so that Miiverse will load from your local server

I'll have to ping @CaramelKat about the exact steps for this (she uses a custom build iirc). This should probably all be added to our developer documentation at some point.

[MatthewL246]

Hello! It's going to be a little bit of work to get my project working with this PR, since I use some patches to the source code and they'll all be invalid now with all the file renames. I can get that done pretty soon today.

The readme has full instructions for setup and connecting to your own server, including the custom Inkay patches Jon mentioned.

[jonbarrow]

Hello! It's going to be a little bit of work to get my project working with this PR, since I use some patches to the source code and they'll all be invalid now with all the file renames. I can get that done pretty soon today.

To be clear my suggestion was not to use your project for this PR, as in the juxtaposition-ui server. The suggestion was to use your project to get an instance of the other servers up (account and friends) without much hassle

[MatthewL246]

I'm not sure I'm understanding your proposal. I can see that Juxt depends on account, friends, Mongo, and Redis. Are you proposing running Juxt on the host and connecting it to the already-set-up containerized services with port forwarding? That doesn't really make sense to me? Nginx and mitmproxy run inside containers too. and I don't think there's a way to make them connect to a server that's running on the host? I see you can set every container to use network_mode: "host", which does solve some things, but it still seems weird to mix containers with non-containers.

I am aware though that my project is not great for actively developing on the servers, mainly because using patches messes up the diffs, but it should be fine just for testing in this case.

[jonbarrow]

I'm not sure I'm understanding your proposal. I can see that Juxt depends on account, friends, Mongo, and Redis. Are you proposing running Juxt on the host and connecting it to the already-set-up containerized services with port forwarding?

I'm not really sure where the disconnect here is? I thought the proposal was pretty straight forward

• juxtaposition-ui server running on the host
• Other services running in their containers

That doesn't really make sense to me?

I'm not sure why? It makes a ton of sense to me. If all that's needed is to test the juxtaposition-ui server, then there's no real need to spend time setting up all the rest of the infrastructure yourself. That's the entire point of Docker containers like these, to quickly spin up instances of the servers without needing to manually setup the environment.

Working in this way makes a ton of sense, imo. There's no need to set up the environment to run servers you aren't actually working on, when all you need is for them to be available locally. Mixing containers and non-containers is extremely common in my experience when doing dev work, I very often spin up containers for certain dependencies (especially those which require specialized setups to work, such as Cassandra and SSSL, the latter of which requires it's own custom build of nginx), and then just work on and run what I'm actually working on in the host.

This is a very common way of working in my experience.

Nginx and mitmproxy run inside containers too, and I don't think there's a way to make them connect to a server that's running on the host?

I am aware though that my project is not great for actively developing on the servers, mainly because using patches messes up the diffs, but it should be fine just for testing in this case.

It seems I may have been mistaken about the usefulness of your project with relation to development work, then. I was under the impression that your project would allow for spinning up quick instances of servers for this very use case, where you may not necessarily want to setup the full server yourself but you need a local instance of it running for what you are working on.

[MatthewL246]

I see. There was a disconnect in how we thought about developing the services using my project. I had assumed that one would work on everything as containerized all the time, including what they're currently working on (commonly known as "dev containers"). Probably using a bind mount instead of rebuilding the containers every time? I don't know which strategy is better, we can discuss that another time.

Anyway, to work on the services using the setup you're describing, you'd need to add network_mode: "host" and delete networks: internal: in the compose.yml for all of the containers you're using. It's certainly possible to have it work like you want, it's just not how I set it up because I didn't think that way.

Edit: Okay, it's not that quite that simple. Quite a few things need to be reconfigured so all the services use different ports and nginx uses the right upstreams.

Edit 2: It's definitely possible to set this up as you described. The trick is setting the Nginx upstream of the relevant server to http://host.docker.internal:<port>. Better and more documented support for developing like this is on the roadmap.

[CaramelKat]

Overall this works great, using the couple changes I highlighted here I was able to get the server running locally with success using the desktop, Wii U and 3DS clients. The biggest issue left here is building. Right now, the dist folder structure looks like this:

\dist
|--\config.json
|--\src
  |- everything else...

This prevents tsc-alias from actually working correctly, since it can't find the files it needs to fix. This also causes issues with the current package.json, since it won't be able to find the right place to start from (which is a small problem, that can easily be updated). We are also missing the views directory, but that can be easily copied, like we already do on the account server here.

The bigger issue is until we move away from using the config.json file, it will create this structure with the src folder. A temp solution would be to move the config file into the src directory and update the imports. This would block the creation of the src folder and allow tsc-alias to run correctly, though it's less than ideal. Personally, I think that would be okay just to get us over the finish line so we can move to a setup like we have in the account server with .env for our configs, but I personally think that would be out of scope for this PR. I'd like @jonbarrow's input though, since I know this is one of the larger repositories that are getting moved and I know that a bigger rewrite is likely going to be happening regardless here in the not too distant future.

Other than that, great job. For not having an environment to test this in before, this was remarkably water tight in my own local testing. I hope to have a better guide for setting up local testing this weekend, but it depends on how things go.

[CaramelKat]

Since I've confirmed this is working in test environments, my vote is to move this into dev to keep testing and stop blocking additional development. @jonbarrow since you still had an open review on this, do you have any objections to that?