util-http breaks auto-instrumentation since v0.25b0

[adamantike]

Describe your environment

Dockerfile provided for reproducibility, using Python 3.9, and installing the following libraries:

flask==2.0.2
opentelemetry-sdk==1.6.0
opentelemetry-instrumentation-flask==0.25b0

Steps to reproduce

The following Dockerfile provides a simple Flask server, instrumented using opentelemetry-instrument:

FROM python:3.9-slim

WORKDIR /app

RUN pip install --no-cache-dir \
    opentelemetry-sdk==1.6.0 \
    opentelemetry-instrumentation-flask==0.25b0 \
    flask==2.0.2

RUN echo "\n\
from flask import Flask\n\
\n\
app = Flask(__name__)\n\
\n\
@app.route('/')\n\
def server_request():\n\
    return 'ok'\n\
\n\
app.run(host='0.0.0.0', port=5000)\n\
" > server.py

EXPOSE 5000

CMD ["opentelemetry-instrument", "--traces_exporter", "console", "python", "server.py"]

Use the following command to build and run the Docker container:

docker build -t test-otel-http .
docker run --rm -it -p5000:5000 test-otel-http

What is the expected behavior?

Correctly instrumented Flask server.

What is the actual behavior?

Running the Docker container raises an exception in OpenTelemetry. The Flask server starts nevertheless, without being correctly instrumented:

Instrumenting of httplib failed
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/pkg_resources/__init__.py", line 2739, in requires
    deps.extend(dm[safe_extra(ext)])
KeyError: 'instruments'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/opentelemetry/instrumentation/auto_instrumentation/sitecustomize.py", line 74, in _load_instrumentors
    conflict = get_dist_dependency_conflicts(entry_point.dist)
  File "/usr/local/lib/python3.9/site-packages/opentelemetry/instrumentation/dependencies.py", line 32, in get_dist_dependency_conflicts
    for dep in dist.requires(("instruments",)):
  File "/usr/local/lib/python3.9/site-packages/pkg_resources/__init__.py", line 2741, in requires
    raise UnknownExtra(
pkg_resources.UnknownExtra: opentelemetry-util-http 0.25b0 has no such extra feature 'instruments'
Failed to auto initialize opentelemetry
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/pkg_resources/__init__.py", line 2739, in requires
    deps.extend(dm[safe_extra(ext)])
KeyError: 'instruments'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/opentelemetry/instrumentation/auto_instrumentation/sitecustomize.py", line 117, in initialize
    _load_instrumentors(distro)
  File "/usr/local/lib/python3.9/site-packages/opentelemetry/instrumentation/auto_instrumentation/sitecustomize.py", line 88, in _load_instrumentors
    raise exc
  File "/usr/local/lib/python3.9/site-packages/opentelemetry/instrumentation/auto_instrumentation/sitecustomize.py", line 74, in _load_instrumentors
    conflict = get_dist_dependency_conflicts(entry_point.dist)
  File "/usr/local/lib/python3.9/site-packages/opentelemetry/instrumentation/dependencies.py", line 32, in get_dist_dependency_conflicts
    for dep in dist.requires(("instruments",)):
  File "/usr/local/lib/python3.9/site-packages/pkg_resources/__init__.py", line 2741, in requires
    raise UnknownExtra(
pkg_resources.UnknownExtra: opentelemetry-util-http 0.25b0 has no such extra feature 'instruments'
 * Serving Flask app 'server' (lazy loading)
 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.
   Use a production WSGI server instead.
 * Debug mode: off
 * Running on all addresses.
   WARNING: This is a development server. Do not use it in a production deployment.
 * Running on http://172.17.0.4:5000/ (Press CTRL+C to quit)

Additional context

As far as I could investigate, this is what I understand is happening:

1. In Add net.peer.ip in requests & urllib3 instrumentations. #661, opentelemetry-util-http got a provider (HttpClientInstrumentor) as an entry point.
2. Since opentelemetry-instrument iterates over all OpenTelemetry providers, it now finds HttpClientInstrumentor.
3. Auto-instrumentation looks for the instruments extra in opentelemetry-util-http, which doesn't exist, and fails.

As opentelemetry-util-http doesn't depend on non-stdlib Python packages, it seems that the only missing pieces to fix this issue are:

1. Add _instruments = tuple() to package.py (e.g.

   opentelemetry-python-contrib/instrumentation/opentelemetry-instrumentation-urllib/src/opentelemetry/instrumentation/urllib/package.py

   Line 16 in f7256f5

   _instruments = tuple()

   )
2. Add extras_require configuration to setup.py (e.g. https://github.com/open-telemetry/opentelemetry-python-contrib/blob/354bdc42d9e3d071a60737c47c7b1e17f77e8425/instrumentation/opentelemetry-instrumentation-urllib/setup.py)

[adamantike]

cc relevant authors:

• @owais (Instrumentation runtime checks #475)
• @Oberon00 and @lzchen (Add net.peer.ip in requests & urllib3 instrumentations. #661)

[owais]

We will try to release a fix for this next week. Until then please downgrade to the previous release if you are affected by this.