NodeServices - multiple node workers?

[caesay]

I'm sharing a lot of code between my client and my server via NodeServices, and some of this client side code is heavy. If we receive several requests at the same time, nodejs can only process them one at a time. Because of this, I was wondering if would be possible to start up more than one HttpNodeInstance and spread the requests over multiple processes.

It should be possible in theory, but most of the code in this library is internal so I'm not sure if there is an extension point somewhere (I can't even instantiate a HttpNodeInstance which appears to be the only supported hosting model)

The prompt for this query is because of something i've observed - if I hit F5 a bunch of times in rapid succession, it processes each request one after the other, taking 1-2 seconds each. No other requests are processed by node until each of these has finished. This means that 20 connections at the same time will take 20-40 seconds to render.

[SteveSandersonMS]

I'm sharing a lot of code between my client and my server via NodeServices, and some of this client side code is heavy. If we receive several requests at the same time, nodejs can only process them one at a time.

It's normal in Node development to have only one instance. Node's event loop model means that it can process a large number of requests simultaneously as long as most of the processing cost is async I/O rather than CPU-blocking. Generally people don't use Node code for expensive CPU operations. As such it's very rare to need more than one Node instance even for a high throughput site.

It should be possible in theory, but most of the code in this library is internal so I'm not sure if there is an extension point somewhere (I can't even instantiate a HttpNodeInstance which appears to be the only supported hosting model)

You can call Microsoft.AspNetCore.NodeServices.NodeServicesFactory.CreateNodeServices to create as many INodeServices instances as you want. Each of them will represent a separate Node process.

if I hit F5 a bunch of times in rapid succession, it processes each request one after the other, taking 1-2 seconds each. No other requests are processed by node until each of these has finished

That suggests that your Node calls are CPU heavy or does blocking (non-async) I/O, which is not a recommended use case for Node in general (not just within NodeServices). If this is fundamental to your application then you're probably right to look at parallelising over multiple Node instances, but if you didn't think your calls were CPU blocking, then it would be worth investigating to see if there are unanticipated perf bottlenecks.

[hartmannr76]

I think some apps that may want to do server-side rendering may end up being heavier on CPU (though I could be wrong). I glanced through the setup of the extension methods and wrapped the node services in a simple object pool. I ran some samples against the React/Redux sample app that I configured to do SSR.

(I spent 5 min on this and I've occasionally seen it not dispose as intended so it would need to be cleaned up, but here is the gist)

    public class NodeServicesPool : INodeServices
    {
        private readonly ConcurrentBag<INodeServices> _serviceBag;
        private readonly SemaphoreSlim _semaphore;

        public NodeServicesPool(int serviceCount, IServiceProvider sp)
        {
            _serviceBag = new ConcurrentBag<INodeServices>();
            var options = new NodeServicesOptions(sp);
            for (int i = 0; i < serviceCount; i++)
            {
                _serviceBag.Add(NodeServicesFactory.CreateNodeServices(options));
            }
            _semaphore = new SemaphoreSlim(serviceCount);
        }

        public void Dispose()
        {
            foreach (var service in _serviceBag)
            {
                service.Dispose();
            }
        }

        private async Task<T> InternalInvoke<T>(Func<INodeServices, Task<T>> callback)
        {
            INodeServices service = null;
            try
            {
                await _semaphore.WaitAsync();
                _serviceBag.TryTake(out service);

                return await callback(service);
            }
            finally
            {
                if (service != null)
                {
                    _serviceBag.Add(service);
                }

                _semaphore.Release();
            }
        }

        public async Task<T> InvokeAsync<T>(string moduleName, params object[] args) =>
            await InternalInvoke(service => service.InvokeAsync<T>(moduleName, args));

        public async Task<T> InvokeAsync<T>(CancellationToken cancellationToken, string moduleName, params object[] args) =>
            await InternalInvoke(service => service.InvokeAsync<T>(cancellationToken, moduleName, args));

        public async Task<T> InvokeExportAsync<T>(string moduleName, string exportedFunctionName, params object[] args) =>
            await InternalInvoke(service => service.InvokeExportAsync<T>(moduleName, exportedFunctionName, args));

        public async Task<T> InvokeExportAsync<T>(CancellationToken cancellationToken, string moduleName, string exportedFunctionName, params object[] args) =>
            await InternalInvoke(service => service.InvokeExportAsync<T>(cancellationToken, moduleName, exportedFunctionName, args));
    }

But the results spoke for themselves when I benchmarked the different INodeServices configurations

Control configuration
services.AddNodeServices();
Results

bombardier http://localhost:5000 -c 50 -l -d 60s -k
Bombarding http://localhost:5000 for 1m0s using 50 connections
[===================================================================================] 1m0s
Done!
Statistics        Avg      Stdev        Max
  Reqs/sec       157.51      93.12       2575
  Latency      316.12ms   117.76ms      1.27s
  Latency Distribution
     50%   290.81ms
     75%   319.98ms
     90%   379.93ms
     99%      1.05s
  HTTP codes:
    1xx - 0, 2xx - 9509, 3xx - 0, 4xx - 0, 5xx - 0
    others - 0
  Throughput:   686.91KB/s

Experimental configuration

services.AddSingleton<INodeServices>(sp =>
{
    return new NodeServicesPool(3, sp);
});

Results

bombardier http://localhost:5000 -c 50 -l -d 60s -k
Bombarding http://localhost:5000 for 1m0s using 50 connections
[===================================================================================] 1m0s
Done!
Statistics        Avg      Stdev        Max
  Reqs/sec       276.78      72.59        497
  Latency      180.72ms    48.62ms      0.91s
  Latency Distribution
     50%   171.67ms
     75%   192.16ms
     90%   212.83ms
     99%   285.02ms
  HTTP codes:
    1xx - 0, 2xx - 16620, 3xx - 0, 4xx - 0, 5xx - 0
    others - 0
  Throughput:     1.17MB/s

Latency was consistently lower with this higher concurrent connection count and throughput was generally ~50% more.

I glanced at my running process to make sure there were multiple node processes running and was able to confirm that

-> % ps -a | grep node
16355 ttys003    0:00.68 node /Users/me/src/dotnet-cra-ssr/ClientApp/node_modules/.bin/react-scripts start
16356 ttys003    0:16.10 node /Users/me/src/dotnet-cra-ssr/ClientApp/node_modules/react-scripts/scripts/start.js
16357 ttys003    0:02.04 node /var/folders/89/dsbgqdv14595ytn3pjw4mp5w0000gn/T/ujdytd2i.3kn --parentPid 16351 --port 0
16360 ttys003    0:02.10 node /var/folders/89/dsbgqdv14595ytn3pjw4mp5w0000gn/T/njwxgzkr.zxy --parentPid 16351 --port 0
16384 ttys003    0:01.46 node /var/folders/89/dsbgqdv14595ytn3pjw4mp5w0000gn/T/3xipvyqw.5a1 --parentPid 16351 --port 0
16386 ttys011    0:00.01 grep --color=auto --exclude-dir=.bzr --exclude-dir=CVS --exclude-dir=.git --exclude-dir=.hg --exclude-dir=.svn node

I think if this was added as a base offering in this library, the community would greatly benefit.

Would you be open to me submitting a PR to introduce this @SteveSanderson?

[timothywisdom]

@hartmannr76 - this is great. There is definitely a use case for this as RenderToString is a blocking operation so Node currently blocks when performing server side rendering (i.e. you can only render one request at a time).

Another solution, which I'm still investigating, is to create your pool of Node instances via the Node Clustering feature:
https://nodejs.org/api/cluster.html