[BACKLOG] Change monorepo tool from Nx to Turborepo

[saenyakorn]

Problem

TL;DR Nx is an over-engineer tool that makes everything harder. Use Turborepo instead.

To create a new app under Nx monorepo, you must use Nrwl CLI to generate it. For example, creating a Next.js app using @nrwl/next. Moreover, Nx does not use workspace feature of yarn or pnpm but uses their workspace management tools (e.g. nx.json, project.json), which is not a standard for developing Typescript/Javascript apps. It consumes lots of time to understand what it does, which means it's hard to create a new app in this repo. So, we'll deliver our new product late.

One of the crucial problems is shared node_modules and single package.json. For now, to build the Docker image for each app. We have to build the target app outside Docker images before copying the files to its since we don't know which packages are used in the app. That's a really hacked technique. It may cause a problem when the local and the image environment do not match.

In addition, node_modules for just the Nx library is like a black hole for some reason.

Nx is tough to integrate the python app into this repo. Since creating a new app needs their CLI, and there's no official CLI for python. I did not try it yet, but it would be F hard to integrate the python app into this repo now.

But Nx provided some good things for us.

• Nx has a Remote Caching feature that lets some commands faster than usual (e.g. build, lint, test). Because of it, we can reduce workflow time usage by half.
• Nx has affected command, which can detect affected packages from changes between base and head.

So, since Nx makes our life harder. I want to propose the new monorepo tools, Turborepo by Vercel

Why Turborepo?

Turborepo is a monorepo management tool that fully utilizes yarn or pnpm workspace features.

Turborepo is simple! The thing you need when creating a new app is just package.json, which contains a proper name, version, and some scripts. Each app will have its own node_modules. It may be hard to clear all node_modules in the project, but I think it's not a big problem.

Since Turborepo fully utilizes the workspace feature of Node package manager. You can run the app with simple commands like yarn workspace app1 dev or pnpm --filter app1 dev. But using that command will not use the Turborepo feature; pipeline.

Turborepo can run multiple tasks concurrently using the pipeline feature. You can read more about how it works here. https://turbo.build/repo/docs/core-concepts/monorepos/running-tasks#most-tools-dont-optimize-for-speed

Note: pipeline is defined In turbo.json. It's a bit hard to understand, but only the maintainer has to define this. Then it's not a common problem.

Turborepo also provides official instructions for building Docker images in their docs. You can follow it here. https://turbo.build/repo/docs/handbook/deploying-with-docker

In their docs, they also mention how to version the apps and packages using the different tools, which is great for us to make a proper process for versioning apps and packages.

Notes

I have been playing Turborepo recently for building Docler images and versioning the app. I'll write about them later when I fully prepare the demo.

Task Description

Step 1

• Remove Nx and replace it with Turborepo
• Change the imported path
• Test that all apps can build Docker images properly

Step 2

• Migrate other outside apps into this repo, including the Python app
• Versioning every app in this repo using Changeset (Or other tools)

This task might affect other issues such as

• [Refactor] Migrate common components to @libs/react-ui #443
• [BACKLOG] Migrate cugetreg-scraper to this repo #455
• [BACKLOG] Split admin backend into a separate service #403
• Etc.

Additional Context

My Demo

With Docker

• TODO

With Versioning

• https://github.com/saenyakorn/turborepo-versioning-demo
• Demo for Nx versioning, Thanks @tithanayut, https://github.com/tithanayut/cugetreg-playground

Related Teams

• Frontend
• Backend
• Data
• Design
• Infra
• QA

Task Advisors

No response

[bombnp]

I believe @leomotors had some experience with Turbo too

[saenyakorn]

Turborepo versioning with Changeset is complete now! (I guess) https://github.com/saenyakorn/turborepo-versioning-demo

[PongDev]

From my direct experience, Turbo repo is more easily to use than nx a lot. It's also has well document too.

[leomotors]

🤔

[saenyakorn]

Here's my full example of Turborepo versioning using Changeset. I also added building Docker workflow for deploy published app. You can fork the repo and play with it.