nodemon with babel-node in docker address already in use

[theprobugmaker]

I configured a simple app in express to run in a docker container but whenever I make a change nodemon gives me an error saying that the port is already in use.

Error: listen EADDRINUSE: address already in use :::4000

That is my start script: nodemon -L --exec babel-node -- src/index

"nodemon": "^1.18.7"

and my index.js:

import express from 'express'

const app = express();

app.get("/", (req, res) => {
  res.send("hello world");
});

app.listen(4000, () => console.log("The server is listening on port 4000"));

It's like it's not closing the process and it's trying to execute again

[remy]

Can you provide a dockerfile? I don't use docker so I don't know how to run these things

[theprobugmaker]

It's simple @remy, I setup this for development only, I mounted the current directory with the container so it's mirrored and for that reason I don't need to copy the package.json or anything, just run yarn to install the packages and it will work just fine:

Dockerfile:

FROM node:11.2.0-alpine
WORKDIR /usr/src/app
RUN yarn

docker-compose.yml:

version: "3.7"
services:
  server:
    build: .
    command: yarn start
    ports:
      - "4000:4000"
    volumes:
      - .:/usr/src/app

[theprobugmaker]

I noticed that it only works without babel-node, if I change:

import express from 'express'

to:

const express = require('express')

and use this as script:

"start": "nodemon -L src/index"

It works, it doesn't however when using --exec and babel-node

[richardkall]

I'm having the same issue when running with docker-compose. However, adding --signal SIGINT to the nodemon command seems to fix it.

[theprobugmaker]

@richardkall I haven't tested the --signal yet, I will try it out. It would be nice to have this fixed by default, specially the legacy watch one, I'm having a lot of problems with it. Earlier today I was having a problem that nodemon was restarting over and over again the same file (I wasn't even changing anything in the file) and it was opening a lot of processes and it was crashing (freezing) my docker, it was really weird.

[remy]

@zefexdeveloper Ah…you say "simple" like I'm supposed to know how docker works. What do I need to run those files? I've assumed docker for Mac, but how do I start it and test it?

[theprobugmaker]

@remy Oh, sorry, I thought you knew, my bad, that is why I said it's simple, because indeed the file is pretty simple.
You just have to have docker installed on your machine, create a folder with a Dockerfile (code above), create the docker-compose.yml (code above), a index.js file with the express and your package.json with nodemon installed.
You can run like this: docker-compose up

[theprobugmaker]

@richardkall I just tried using --signal and for some reason nodemon got stuck at:

[nodemon] restarting due to changes...

[richardkall]

Weird, for me it's working with the --signal argument. Here's my setup, hopefully it can help in some way.

❯ docker-compose --version
docker-compose version 1.23.2, build 1110ad01

❯ docker --version
Docker version 18.09.0, build 4d60db4

./api/Dockerfile:

FROM node:10.13.0-alpine as base
WORKDIR /app
COPY package.json .
COPY yarn.lock .
RUN yarn --frozen-lockfile
COPY . .

docker-compose.yml:

version: '3.7'
services:
  api:
    build:
      context: ./api
      target: base
    command: yarn dev
    container_name: api
    ports:
      - 4000:4000
    tty: true
    volumes:
      - /app/node_modules
      - ./api:/app

  web:
    ...

./api/package.json:

{
  "engines": {
    "node": "^10.13.0",
    "yarn": "^1.10.1"
  },
  "scripts": {
    "dev": "nodemon --signal SIGINT -e ts -x 'babel-node --extensions .ts' src"
  },
  "devDependencies": {
    "@babel/node": "^7.0.0",
    "nodemon": "^1.18.7",
     ...
  }
}

[theprobugmaker]

@richardkall It's still not working for me, I noticed that you are not using the legacy watch, for me if I don't use nodemon doesn't work at all, specially now setting --signal even with the legacy watch it's not working either.

[nodemon] restarting due to changes...
[nodemon] restarting due to changes...
[nodemon] restarting due to changes...

btw: my settings are not that different from yours.

[tibotiber]

I'm facing the same issue. This has started after I upgraded from 1.18.4 to 1.18.7. I think it could be related to #1476.

Also I'm controlling the shutdown of my server with the approach described at https://github.com/remy/nodemon#controlling-shutdown-of-your-script but I don't seem to ever receive SIGUSR2.

One more thing, if I start and immediately change a file I can clearly see the logs of 2 servers starting, the first one with success and the second one failing due to EADDRINUSE. Quite clearly the process is not shutdown.

I've tried to debug but I can't seem to find much, not familiar with the nodemon codebase. @remy is there a build process if I change the source or I can just push the code to my fork and npm install from there?

[20k-ultra]

To add some more details on this issue, I've been using nodemon to hot reload my API inside docker for the past 2 months. It was worked extremely well with no issues. After I upgraded I began to experience OPs issue.

docker-compose version 1.23.2, build 1110ad01
Docker version 18.09.0, build 4d60db4
Alpine 3.8
Node v8.11.4
Nodemon version 1.18.7

Modifying my command to use --signal SIGINT as suggested above didn't solve the issue.

However, I tested my API outside of docker with and without --signal SIGINT and it reloads correctly in both scenarios.

I can't debug much further right now but hopefully this bit of info can help.

[OlivierCuyp]

I face the same issue and I believe somehow it is a Docker + nodemon --exec flag issue

In the following package.json, if you replace:

  ...
  "start:dev": "nodemon --watch './src' --exec 'npm start'"
  ...

by:

  ...
  "start:dev": "nodemon --watch './src'"
  ...

it works perfectly under docker.
Note that both work fine under OSX.

Here is the test case:

src/index.js

const express = require('express');
const app = express();
app.listen(17600, () => {
  console.log('Server MediaStreamTrackAudioSourceNode, Olé !');
});

package.json

{
  "name": "nodemon-test",
  "version": "1.0.0",
  "description": "",
  "main": "./src/index.js",
  "scripts": {
    "start": "node .",
    "start:dev": "nodemon --watch './src' --exec 'npm start'"
  },
  "author": "",
  "license": "ISC",
  "dependencies": {
    "express": "^4.16.4"
  },
  "devDependencies": {
    "nodemon": "^1.18.7"
  }
}

Dockerfile

FROM node:10-alpine

WORKDIR /app

COPY ./src /app/src
COPY ./package.json /app/package.json
COPY ./package-lock.json /app/package-lock.json

RUN npm install

CMD [ "npm", "start" ]

docker-compose.yml

version: '3.7'

services:
  node:
    build: .
    command: npm run start:dev
    ports:
      - 17600:17600
    volumes:
      - ./src:/app/src
      - ./package.json:/app/package.json

Note: I need --exec because I pipe the output in the bunyan parser like this --exec 'npm start' | bunyan in real use case.

[20k-ultra]

@OlivierCuyp see Nodemon inside a docker container requires the use of the legacy watch flag -L

However, even after I use legacy watch like OP my nodemon script does not correctly restart. I am also using --exec to pipe my nodemon watch to trigger a node-babel build.

I'm using nodemon the following way:

    "dev": "nodemon --legacy-watch -w src --exec \"babel-node src --presets env,stage-0\"",

[theprobugmaker]

Let's wait for @remy to see what he has to say about that, the next steps.

[OlivierCuyp]

@20k-ultra To my understanding under OSX docker is not using virtualbox with vboxsf since a long time so the -L to watch the filesystem is useless in this case.
I got no idea if it is still required on Windows though...
But this is not relevant to the issue since the -L is use to make sure nodemon catches the file changes and tries to restart.
Here it tries to restart but fail trying.

[spalberg]

Same error using linux, docker, yarn and ts-node.

Dockerfile:

FROM node:8.11.3-alpine
...
CMD ["yarn", "run", "dev"]

package.json:

"scripts": {
    "dev": "NODE_ENV=development nodemon --exec ts-node src/server.ts",
},
"devDependencies": {
    ....
    "nodemon": "^1.18.7",
    "ts-node": "^7.0.1",
    ...

Versions:

Alpine 3.8
Node v8.11.4
Nodemon version 1.18.7

Error in terminal:

...
server: [nodemon] restarting due to changes...
server: [nodemon] starting `ts-node src/server.ts`
server: Starting server...
server: Error: listen EADDRINUSE :::4000
...

But if I exec into the running container I can't find the blocking process:

docker-composer exec server sh
fuser 4000/tcp
# No Output

[OlivierCuyp]

@tibotiber I tried to reinstall the version 1.18.4 but it didn't solve the issue.
But then I rollbacked to the version 1.12.1 and voilà ! I worked like a charm.
So as a temporary fix you can do:

npm rm nodemon
npm i nodem@1.12.1 -D

Note: I didn't checked all versions between 1.12.1 and 1.18.4

[OlivierCuyp]

@remy if you need help for testing let me know if I can do something.

[theprobugmaker]

@OlivierCuyp @spalberg @20k-ultra @tibotiber Try sharing your Dockerfile for development as well so we can test or help @remy or anyone else interesting in fixing this.

[20k-ultra]

This issue is affecting my workflow so I'll try my best to be as helpful as possible!

Here's a repo I've made to make it very easy to reproduce the issue. Should be easy for new docker users to use as well.

[tibotiber]

@OlivierCuyp I'm surprised 1.18.4 even successfully installed, it had event-stream as deep dependency at a version which was compromised (see https://arstechnica.com/information-technology/2018/11/hacker-backdoors-widely-used-open-source-software-to-steal-bitcoin/) and has been pulled off npm. Good that rolling back helped, but do check your deep deps to be sure you don't have the malicious package (flatmap-stream) in there.

@zefexdeveloper here is my setup:
Dockerfile

FROM node:8-alpine
WORKDIR /usr/src/api
COPY package.json .
COPY yarn.lock .
RUN yarn install
COPY . .
EXPOSE 4000
CMD ["yarn", "start"]

docker-compose (parts)

api:
    build:
      context: ./api
    restart: always
    environment:
      - [...]
    ports:
      - "${API_PORT}:3000"
    volumes:
      - "./api:/usr/src/api"
    depends_on:
      - mongo

yarn script

"start": "nodemon src/server.js --exec babel-node"

Note: I've moved to babel-watch due to this issue, that might help anybody stuck in their workflow (@20k-ultra)

[theprobugmaker]

To be honest with you guys I'm really considering not using Docker for development mostly because Docker volumes doesn't make a lot of sense for it anyway. It doesn't matter if your Dockerfile copy your package.json, install the dependencies, when you mount the volume the host files will override the container files anyway, and it might be even worse if your host is Windows and the container is using Linux, if a module needs compilation 100% sure you will have problems. Perhaps keep the development locally and use the containers for things like Postgres, NGINX, Redis and so on, at the end you might create a Dockerfile for your API and/or client in prod as you won't have the volume issues.
I'm open to suggestions, I might be wrong. Share your thoughts.

---

Nevermind, in resume you can use Docker for development, you just have to be sure you are mounting the right directory as volume, don't .:/usr/src/app (prefer ./src:/usr/src/app/src instead) because if you have packages that needs compilation it can be a mess. Your container might not be the same OS as your host. Instead mount the source directory and use the container node_modules.

[OlivierCuyp]

@zefexdeveloper I posted a full test case up in the issue: here.
I made it as small as possible for fast test iteration.
The issue is really the nodemon --exec in Docker context.

For the dev Docker chat, we should find an other place to discuss it. I'll be glad to share my experience with it.

@tibotiber I check the npm auditresult after installing nodemon@1.12.1 only 2 low severity vulnerabilities and no sign of flatmap-stream

[GryaznovGI]

Hi guys.
I've tried to subscribe to all signals from this file "nodemon/lib/monitor/signals.js" inside my nodejs app running in docker. Then I ran nodemon in "verbose" mode and saw that "SIGHUP" signal was sent, but it wasn't delivered to my app inside docker. Looks like there is the root of evil.

PS: I don't have --exec parameter in my startup script. But I faced with the same issue.
PPS: 1.12.1 really works fine. Thank you @OlivierCuyp!

[remy]

I have a feeling (though I'm on a mobile phone so it's not straight forward to confirm), that nodemon@1.12.x was pre-chokidar which meant it used the find command to poll and watch (it's quite a long time ago, so I'm not 100% sure…

Any more digging you're able to do is definitely a help!

[OlivierCuyp]

@remy don't think chokidar is the issue here. Since file changes are catched by nodemon.
The issue is after, the process not being killed for some reason (in Docker context).
Going the github releases I couldn't find the 1.12.1 version but I checked the package.json of the 1.12.3.
Chokidar was already there.

[OlivierCuyp]

@GryaznovGI could you provide your startup script ?

[remy]

Basically: dupe of #1484 and it's in progress now - feel free to follow along there if you want to see how I fixed it.

[remy]

Live fix in nodemon@1.18.8 (core change in pstree.remy…again).

[theprobugmaker]

@remy Thank you very much for the effort. I will be testing the changes today.

[nanxiaobei]

Just add --delay 300ms into the command solved my problem.

https://github.com/remy/nodemon#delaying-restarting

[chadlittle]

Anyone else from the future visiting this, I had to make sure my volume was mounted in docker-compose.yml to get nodemon to find the files.

[krisdover]

I encountered this issue trying to get nodemon to restart a development server in a docker container and after debugging it I discovered the issue was the pstree.remy library wasn't able to generate a list of the child process subtree for termination so it only terminates the top level child (the shell which launches node) and orphans the actual node server. This happens because:

1. the ps util wasn't installed by default in my node:lts-slim docker image.
2. the user the docker process runs as did not have permission to stats /proc (which is the fallback for when ps is unavailable).

The solution for me was to install ps in my docker image:

# install ps util needed by nodemon
RUN apt-get update && apt-get install -y procps && rm -rf /var/lib/apt/lists/*

[ctooley21]

I encountered this issue trying to get nodemon to restart a development server in a docker container and after debugging it I discovered the issue was the pstree.remy library wasn't able to generate a list of the child process subtree for termination so it only terminates the top level child (the shell which launches node) and orphans the actual node server. This happens because:

1. the `ps` util wasn't installed by default in my `node:lts-slim` docker image.

2. the user the docker process runs as did not have permission to stats `/proc` (which is the fallback for when `ps` is unavailable).

The solution for me was to install ps in my docker image:

# install ps util needed by nodemon
RUN apt-get update && apt-get install -y procps && rm -rf /var/lib/apt/lists/*

Thank you! This solved the issue for me as well.