Serverless offline does not work correctly with yarn pnp and (--useChildProcesses, --useWorkerThreads) flags

[Travellerme]

Bug Report

Hello

We use yarn pnp approach for managing our packages.
And serverless-offline works if we use one process for handling all functions
But in this case, as we have a lot of functions executed, after some time we get Javascript heap of memory error that is was reported by someone else here

As a solution, we decided to use "--useChildProcesses" flag (Run handlers in a child process)
And for our other project that uses regular npm with node_modules it works fine.
But for the project that uses yarn pnp we got another error.

packages/api/package.json:
... "serverless": "^3.18.2", "serverless-offline": "^8.8.0", ...

Current Behavior

sls offline --httpPort 3000 --useChildProcesses

`
┌─────────────────────────────────────────────────────────────────────────────┐
│ │
│ GET | http://0.0.0.0:3000/api/v1/openapi │
│ POST | http://0.0.0.0:3000/2015-03-31/functions/apiVersion1/invocations │
│ GET | http://0.0.0.0:3000/api/v1/client-config │
│ POST | http://0.0.0.0:3000/2015-03-31/functions/apiVersion1/invocations │
│ ANY | http://0.0.0.0:3000/api/v1/{any*} │
│ POST | http://0.0.0.0:3000/2015-03-31/functions/apiVersion1/invocations │
│ │
└─────────────────────────────────────────────────────────────────────────────┘

Server ready: http://0.0.0.0:3000 🚀

Enter "rp" to replay the last request
internal/modules/cjs/loader.js:905
throw err;
^

Error: Cannot find module '/home/username/dev/myProjectName/.yarn/virtual/serverless-offline-virtual-66d2b3ea74/0/cache/serverless-offline-npm-8.8.0-0905a4015f-cc9ebc2016.zip/node_modules/serverless-offline/dist/lambda/handler-runner/child-process-runner/childProcessHelper.js'
at Function.Module._resolveFilename (internal/modules/cjs/loader.js:902:15)
at Function.Module._load (internal/modules/cjs/loader.js:746:27)
at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:76:12)
at internal/main/run_main_module.js:17:47 {
code: 'MODULE_NOT_FOUND',
requireStack: []
}

`
Also, we tried using --useWorkerThreads but got a similar error

sls offline --httpPort 3000 --useWorkerThreads

`
┌─────────────────────────────────────────────────────────────────────────────┐
│ │
│ GET | http://0.0.0.0:3000/api/v1/openapi │
│ POST | http://0.0.0.0:3000/2015-03-31/functions/apiVersion1/invocations │
│ GET | http://0.0.0.0:3000/api/v1/client-config │
│ POST | http://0.0.0.0:3000/2015-03-31/functions/apiVersion1/invocations │
│ ANY | http://0.0.0.0:3000/api/v1/{any*} │
│ POST | http://0.0.0.0:3000/2015-03-31/functions/apiVersion1/invocations │
│ │
└─────────────────────────────────────────────────────────────────────────────┘

Server ready: http://0.0.0.0:3000 🚀

Enter "rp" to replay the last request

✖ Uncaught exception
Environment: linux, node 14.18.2, framework 3.18.2 (local), plugin 6.2.2, SDK 4.3.2
Docs: docs.serverless.com
Support: forum.serverless.com
Bugs: github.com/serverless/serverless/issues

Error:
Error: Cannot find module '/home/username/dev/myProjectName/.yarn/virtual/serverless-offline-virtual-66d2b3ea74/0/cache/serverless-offline-npm-8.8.0-0905a4015f-cc9ebc2016.zip/node_modules/serverless-offline/dist/lambda/handler-runner/worker-thread-runner/workerThreadHelper.js'
at Function.Module._resolveFilename (internal/modules/cjs/loader.js:902:15)
at Function.Module._load (internal/modules/cjs/loader.js:746:27)
at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:76:12)
at MessagePort. (internal/main/worker_thread.js:187:24)
at MessagePort.[nodejs.internal.kHybridDispatch] (internal/event_target.js:399:24)
at MessagePort.exports.emitMessage (internal/per_context/messageport.js:18:26)
Error: Command failed: yarn run start

`

• file: serverless.yml

service: my-services
configValidationMode: error

package:
  individually: true

plugins:
  - serverless-offline
  - serverless-webpack
  - serverless-prune-plugin
  - serverless-domain-manager

custom:
  webpack:
    packager: yarn
    webpackConfig: ./webpack.config.js
    excludeFiles: "**/*.spec.*"


  prune:
    automatic: true
    number: 2

  serverless-offline:
    host: 0.0.0.0
    ignoreJWTSignature: true
    noAuth: true

provider:
  name: aws
  runtime: nodejs14.x
  region: eu-central-1
  
functions:
  refreshFeatureFlagsCache:
    handler: src/feature-flags/lambdas/refresh-feature-flags-cache-handler.main
    events:
      - schedule:
          rate: rate(1 minute)
    timeout: 60
  apiVersion1:
    handler: src/main.handler
    events:
      - httpApi:
          path: /api/v1/openapi
          method: GET
      - httpApi:
          path: /api/v1/client-config
          method: GET
      - httpApi:
          path: /api/v1/{any+}
          method: "*"
          authorizer: cognitoJwtAuthorizer

• file: webpack.config.js

const path = require('path')
const webpack = require('webpack')
const slsw = require('serverless-webpack')

module.exports = {
  context: __dirname,
  mode: slsw.lib.webpack.isLocal ? 'development' : 'production',
  stats: slsw.lib.webpack.isLocal ? undefined : 'minimal',
  entry: slsw.lib.entries, // required for serverless-offline
  target: 'node',
  optimization: {
    mangleExports: false, // required for swagger openapi generation
    minimize: false, // minization causes compilation to take too long
  },
  devtool: slsw.lib.webpack.isLocal ? 'eval-source-map' : 'source-map',
  resolve: {
    extensions: ['.mjs', '.json', '.ts', '...'],
  },
  externalsPresets: { node: true },
  module: {
    rules: [
      {
        test: /\.svg/,
        type: 'asset/resource',
      },
      {
        test: /\.(tsx?)$/,
        loader: 'ts-loader',
        exclude: [
          path.resolve(__dirname, 'node_modules'),
          path.resolve(__dirname, '.serverless'),
          path.resolve(__dirname, '.webpack'),
        ],
        options: {
          transpileOnly: true,
          experimentalWatchApi: true,
        },
      },
    ],
  },
  plugins: [
    new webpack.IgnorePlugin({
      checkResource(resource) {
        // see: https://github.com/nestjs/nest/issues/1706
        const lazyImports = [
          '@grpc/grpc-js',
          '@grpc/proto-loader',
          '@nestjs/microservices',
          '@nestjs/microservices/microservices-module',
          '@nestjs/websockets',
          '@nestjs/websockets/socket-module',
          '@nestjs/platform-socket.io',
          'aws-crt',
          'amqp-connection-manager',
          'amqplib',
          'cache-manager',
          'class-transformer/storage',
          'fastify-swagger',
          'kafkajs',
          'mqtt',
          'nats',
          'redis',
        ]
        if (!lazyImports.includes(resource)) {
          return false
        }
        try {
          require.resolve(resource)
        } catch (err) {
          return true
        }
        return false
      },
    }),
  ],
}

Our project structure:
myProjectName/
----/packages
--------/api
------------/src
------------/package.json
------------/serverless.yml
------------/webpack.config.js
----/.pnp.cjs
----/.pnp.loader.mjs
----/package.json
Expected behavior/code

Environment

• serverless version: v3.18.2
• serverless-offline version: v8.8.0
• node.js version: v14.18.2
• OS: Debian 11

[dnalborczyk]

thank you for filing the issue @Travellerme

it seems the childProcessHelper.js or the workerThreadsHelper.js can't be loaded which are located in the same folder as the parent caller.

could you try if the issue still exists with the latest serverless v9? also, is serverless installed globally or locally next to serverless-offline ?

if it's not yet fixed, could you also try to create a repro repository, as I'm not familiar with yarn pnp.

labeling as bug for now, but I'm fairly certain the problem lies elsewhere (e.g. yarn)

[pjain11]

I ran into similar issue. downgrading to serverless offline v 8.5.0 works for me w Serverless 3.22.

[dnalborczyk]

could you create a small repro repository @pjain11 (or anyone reading this with the same problem) and open a new issue ?

closing this in the meanwhile.