Research /core: Evaluate available solutions for SSR of dynamic remote modules in Next.js

[webcrumbs-community]

As we're diving into the new terrain of server-side loading with Next.js, we're also on the lookout for existing methods or libraries that could be beneficial for our issue #103.

Related to discussion #91

What Needs to Be Done?

The task is simple but crucial: compile a list of existing methods and libraries that enable server-side loading of remote modules in Next.js. Think of this as the scaffolding for our issue #103.

Why Is This Important?

Why do the legwork, you ask? Well, this research sets the stage for a more efficient and effective implementation of server-side loading in our upcoming experiments.

Skills Needed

• Research savvy
• A knack for JavaScript and Next.js
• The art of summarizing and documenting your treasure finds

Requirements

• Create a list of methods that allow for server-side loading in Next.js.
• Add in any relevant libraries or tools that catch your eye.
• Sum it all up: the good, the bad, and the "hmm, interesting."
• Include your findings as a Markdown file at /research

Here's another chance to make an impact on the future of WebCrumbs. Take a peek at our GitHub repo and let's get this research train moving!

[aditya4196]

@webcrumbs-community
I just tested the experiment-1, it looks good (the way it loads plugin-1 and plugin-2)
I would like to take this issue. But anyone else also wants to join in, i'm ready to work as a team on it. Since its better to have brainstorming to understand different alternatives.

[webcrumbs-community]

GREAT, @aditya4196! This one is super important. I was (and still am) stuck with it when I wrote the #91. Let me know if you'd like to discuss any idea. Cheers!

[aditya4196]

I would definitely like to connect and discuss on this idea through a call or something.
I have done some research on alternatives
I found couple of doubts in experiment-1

1. Every plugin needs to be hosted which would wear out the ports available
2. Im not sure if you can remove or add a plugin dynamically (you would need a plugin manager for this)

I would say, if you can unassign me and assign it to someone who can research and do some documentation because im trying to implement a method which i have in mind and would focus less on documenting multiple altenatives

[webcrumbs-community]

Hey, @aditya4196!

Thanks for the insights. Let’s touch base on this later.

Regarding the ports issue: it's just a dev thing. In production, we'll have a CI/CD setup similar to these workflows. I'll push some code soon to clarify this setup.

On dynamic plugin management: yeah, lots to do. Experiment-1 is kinda flexible in the sense that it does load plugins based on given strings (that can potentially be called from a db in the future), but server-side management could be better for SEO and core vitals. I’m confident we’ll crack it with the brains we have here.

I'll reassign the research and documentation to someone else as you focus on your method. Keep us posted on your progress!

Catch you later,
Cheers!

[aditya4196]

Sure, Im on it.

[aditya4196]

So the solution which I was trying was using the next/dynamic using dynamicImports
The way wordpress does is when you try to load a plugin, it gets the entire codebase of that plugin into the root project which the user is building, then dynamically calls the plugin modules in the root project.
But it does not seem to be an ideal approach for the plugin based development for react modules

Following the #91, I did some research on micro-frontend architecture which is said to be a better way to for plugin based development. There are many open source libraries for micro-frontend architecture

Tailor (Zalando Tailor):
GitHub Repository: https://github.com/zalando/tailor

Piral (Piral Framework):
GitHub Repository: https://github.com/smapiot/piral

qiankun (乾坤):
GitHub Repository: https://github.com/umijs/qiankun

Podium:
GitHub Repository: https://github.com/spotify/podium

Incase anyone is picking this story, these links should be helpful
Im going to understand one each and try to implement to get a better understanding of these libraries

[webcrumbs-community]

This is great, @aditya4196. Did not know about qiankun. I'll take a look as well. Thanks.

[rovoskarl]

I have used qiankun before and it is not suitable. There will be performance problems.
If you want to load a remote module, the dependency cannot be shared, which is bad

[webcrumbs-community]

I see. Thanks. I think the way to go is module federation. But we're having a hard time loading it dynamically on the server side. What do you think?

[aditya4196]

Module federation looks to be a better approach for plugin based architecture.
Im just unsure about sharing same plugins across users.
Im trying to compare it to how wordpress does it at every step.

[webcrumbs-community]

Got it. I'm thinking about a center registry (similar to npm) that host all the remoteEntry.js of the plugins, at registry.webcrumbs.org. So, what we see today as localhost:800x, would eventually become registry.webcrumbs.org/plugin-name/v-1/remoteEntry.js, or something like that. Then, as long as we have a database with all the plugins and urls, it should work, theoretically. But to avoid rebuild, this would be loaded dynamically (run time), i.e., without registering all plugins in each website every time a plugin is updated or included. This is being done in experiment-1. But as is it is loading on the client-side, which is bad for UX and SEO. We may find the solution at one of the module federation examples the team placed in this GitHub repo, I hope.

[aditya4196]

Makes sense.
Thanks for clarifying it!!

[webcrumbs-community]

Sure!! Not server side, but I'm saving here the idea to explore using import maps from SystemJS together with lazy load, for loading the 'database of all the plugins and urls'. Just crossed my mind. To be studied. Cheers!

[aditya4196]

Hmm, just read about it. SystemJS is also a dynamic module loader which support dynamic imports and bindings
Let me read about it too.
Actually since no one is taking, let me take this task.
I will research all the different libraries which provides dynamic server-side loading of modules. I already did some research, I can create that list

[webcrumbs-community]

Great, @aditya4196! Let us know how we can help. This one is key.

[aditya4196]

Got it!
Since this one is the crucial one. I would try to keep this thread lively based on what i found and if any doubts before dumping in the documentation

[aditya4196]

I'm sorry for the delay in getting this task done. I was working on the database schema.
I was also trying to do some research on each approach I have found. Im trying to come up with questions which would help me narrow down the needed points.
I would be posting those questions by tomorrow

[aditya4196]

So for the central registry for all plugins solution in future, the plugins would be rendered at the client side for every app a user creates.

Would it be through the url endpoint in the remoteEntry.js?
or
It would be like downloading the plugin codebase into the user created app from the plugin registry through the remoteEntry.js?

If yes for the first one, I'm a bit unsure how the plugin instance management be handled. And other issue which is a drawback of module federation i.e, Network Latency at client side

For the second one, I will make sure to put up something together by tomorrow

[webcrumbs-community]

Don't worry, @aditya4196. This one is a challenge and will invite important discussions. I'm also taking a look, and @rovoskarl may have inputs too.

So, ideally it wouldn't be downloaded locally (can be cached for speed) but it would compose in the server by the users app at runtime. Challenge.

[webcrumbs-community]

I'm sharing this on server side rendering micro frontends. It talks about caching too:

https://youtu.be/QD2BvPfNc6c?si=c_MI7PcUjOG1JGxM

[aditya4196]

Regarding composing the plugins in the server by the users app. That would make that particular plugin as a central point of expose to multiple users right?
Or is it that whenever the user request a plugin instance, a new container will spun up for that plugin instance which will be exposed to that user site?

[webcrumbs-community]

The envisioned scenario:

• a central registry make available static plugins remoteEntry.js but does not serve them
• each user fetch a copy of the remoteEntry.js from the same central registry
• each user load the plugin dynamically from the remoteEntry.js url and compose them at runtime on the server side hydrating it at the client side

I know of the challenges. It would have to be:

• dynamically loaded (we won't have all plugins at build time, although we may have the central registry)
• server-side rendered
• client-side hydrated

[aditya4196]

Got it.
Thanks for clarifying it.
I was kind of doubtful about the plugin instance management so had to ask so many questions 😅

[webcrumbs-community]

No, please ask. This is really the difficult point! Ow, did you receive the newsletter?

[aditya4196]

I missed to sign up for it.

[webcrumbs-community]

I think you may like it. I can send you again, if you want.

[aditya4196]

Sure!!

[webcrumbs-community]

Nice! After you subscribe here https://www.webcrumbs.org/ let me know and I'll resend it just for you

[aditya4196]

Thanks.
I just submitted my email.

[webcrumbs-community]

Hi @aditya4196, I've committed an initial SSR setup in the new branch experiment-3. This is meant to serve as a creative prompt for our ongoing research into the best SSR strategies. It's not set in stone, just a concept to help us think outside the box. Please review the README.md and we can use this as in our discussion on potential paths forward and the pros and cons of different approaches. Let me know your thoughts. Cheers!

[aditya4196]

Im gonna try it out tomorrow morning and will update you

[github-actions]

⚠️ This issue has not seen any activity in the past 2 months so I'm marking it as stale. I'll close it if it doesn't see any activity in the coming week.

[aditya4196]

Sorry @webcrumbs-community for lack of attendance. I was gonna deassign it as I was stuck in an interview process for past 2 months. But Im gonna be free from that process early next week and then Im gonna update on this task!!

[pachicodes]

Awesome @aditya4196 ! Thanks for coming back.
I hope the interview process went ok.
Let me know if you need help with anything.