OSError: [Errno 9] Bad file descriptor on containerized Gunicorn + Fastapi + NGINX

[alessandromoscato]

Hi,

I have a (Docker) containerized application running with Gunicorn + Fastapi + NGINX. I use in particular:

fastapi==0.78.0
uvicorn==0.17.6
starlette==0.19.1
gunicorn==20.1.0

and my Gunicorn config file is as follows:

from multiprocessing import cpu_count

bind = '127.0.0.1:8000'

workers = cpu_count() + 1
worker_class = 'uvicorn.workers.UvicornWorker'

loglevel = 'error'
accesslog = './access_log'
errorlog =  './error_log'

I use NGINX as reverse proxy, so requests are redirected from port 80 to port 8000 from NGINX. My Dockerfile is as follows:

# STAGE 1: BUILD DEPENDENCIES
FROM python:3.9-slim AS compiler
ENV PYTHONUNBUFFERED 1
ENV DEBIAN_FRONTEND=noninteractive
ENV VIRTUAL_ENV=/opt/venv

WORKDIR /app/

# Activate venv
RUN python -m venv $VIRTUAL_ENV
ENV PATH="$VIRTUAL_ENV/bin:$PATH"

# Install project dependencies
ARG INDEX_URL
RUN pip install <PACKAGE_FROM_PRIVATE_REGISTRY>==0.0.53 --index-url $INDEX_URL

COPY ./requirements.txt ./requirements.txt
RUN pip install -r requirements.txt

COPY . .

# Run unit tests
RUN pip install pytest trio
RUN python -m pytest tests/
RUN pip uninstall -y pytest trio

# STAGE 2: RUN
FROM python:3.9-slim AS runner
ENV PYTHONUNBUFFERED 1
ENV DEBIAN_FRONTEND=noninteractive
ENV VIRTUAL_ENV=/opt/venv

WORKDIR /app/

# Install nginx
RUN apt-get update -y
RUN apt-get install -y nginx

COPY --from=compiler $VIRTUAL_ENV $VIRTUAL_ENV

# Activate venv
ENV PATH="$VIRTUAL_ENV/bin:$PATH"

COPY . .

# Set up NGINX configuration
RUN cp ./nginx.conf /etc/nginx/sites-available/nginx.conf 
RUN ln -s /etc/nginx/sites-available/nginx.conf /etc/nginx/sites-enabled/
RUN rm /etc/nginx/sites-enabled/default
RUN nginx -t
RUN service nginx restart

CMD ["gunicorn", "-c", "gunicorn_config.py", "main:app"]

However, Gunicorn does not seem to be working. I get the following error messages:

Fatal error on SSL transport
protocol: <asyncio.sslproto.SSLProtocol object at 0x7f64047a4910>
transport: <_SelectorSocketTransport closing fd=13>
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/asyncio/selector_events.py", line 916, in write
    n = self._sock.send(data)
OSError: [Errno 9] Bad file descriptor

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/asyncio/sslproto.py", line 690, in _process_write_backlog
    self._transport.write(chunk)
  File "/usr/local/lib/python3.9/asyncio/selector_events.py", line 922, in write
    self._fatal_error(exc, 'Fatal write error on socket transport')
  File "/usr/local/lib/python3.9/asyncio/selector_events.py", line 717, in _fatal_error
    self._force_close(exc)
  File "/usr/local/lib/python3.9/asyncio/selector_events.py", line 729, in _force_close
    self._loop.call_soon(self._call_connection_lost, exc)
  File "/usr/local/lib/python3.9/asyncio/base_events.py", line 751, in call_soon
    self._check_closed()
  File "/usr/local/lib/python3.9/asyncio/base_events.py", line 515, in _check_closed
    raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed
Fatal error on SSL transport
protocol: <asyncio.sslproto.SSLProtocol object at 0x7f640479c670>
transport: <_SelectorSocketTransport closing fd=13>
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/asyncio/selector_events.py", line 916, in write
    n = self._sock.send(data)
OSError: [Errno 9] Bad file descriptor

and a little down below the call stack

Traceback (most recent call last):
  File "/opt/venv/lib/python3.9/site-packages/gunicorn/arbiter.py", line 211, in run
  File "/opt/venv/lib/python3.9/site-packages/gunicorn/arbiter.py", line 551, in manage_workers
  File "/opt/venv/lib/python3.9/site-packages/gunicorn/arbiter.py", line 623, in spawn_workers
    time.sleep(0.1 * random.random())

which in turn causes the workers to fail to boot:

Traceback (most recent call last):
  File "/opt/venv/bin/gunicorn", line 8, in <module>
    sys.exit(run())
  File "/opt/venv/lib/python3.9/site-packages/gunicorn/app/wsgiapp.py", line 67, in run
    WSGIApplication("%(prog)s [OPTIONS] [APP_MODULE]").run()
  File "/opt/venv/lib/python3.9/site-packages/gunicorn/app/base.py", line 231, in run
    super().run()
  File "/opt/venv/lib/python3.9/site-packages/gunicorn/app/base.py", line 72, in run
    Arbiter(self).run()
  File "/opt/venv/lib/python3.9/site-packages/gunicorn/arbiter.py", line 229, in run
    self.halt(reason=inst.reason, exit_status=inst.exit_status)
  File "/opt/venv/lib/python3.9/site-packages/gunicorn/arbiter.py", line 342, in halt
    self.stop()
  File "/opt/venv/lib/python3.9/site-packages/gunicorn/arbiter.py", line 393, in stop
    time.sleep(0.1)
  File "/opt/venv/lib/python3.9/site-packages/gunicorn/arbiter.py", line 242, in handle_chld
    self.reap_workers()
  File "/opt/venv/lib/python3.9/site-packages/gunicorn/arbiter.py", line 525, in reap_workers
    raise HaltServer(reason, self.WORKER_BOOT_ERROR)
gunicorn.errors.HaltServer: <HaltServer 'Worker failed to boot.' 3>

A couple of notes:

• This error does occur locally when I launch Gunicorn with same exact set up.
• I don't understand why the call stack mentions "/usr/local/lib/python3.9", since, as it should be clear from my Dockerfile, I do everything inside a virtual env at "/opt/venv". Not sure if this is related to the issue but I just wanted to mention it.

Could someone please help?

[lucasguiss]

I`m currently having the same issue, but my application works fine, it just throw this error whenever I add more workers, did you solved this?

[tilgovi]

See #3127 and the linked issues there.