AWS Lambda gives OSError: [Errno 38] Function not implemented with ddtrace 2.1.0

[rolsen]

Summary of problem

AWS Lambda gives OSError: [Errno 38] Function not implemented with ddtrace 2.1.0

See Parallel Processing in Python with AWS Lambda

Which version of dd-trace-py are you using?

2.1.0 causes the issue
2.0.2 does not cause the issue

Which version of pip are you using?

pip-23.2.1

Which libraries and their versions are you using?

aioboto3-11.3.0
aiobotocore-2.6.0
aiohttp-3.8.6
aioitertools-0.11.0
aiosignal-1.3.1
annotated-types-0.6.0
astroid-3.0.0
async-timeout-4.0.3
attrs-23.1.0
aws-lambda-powertools-2.25.1
aws-sam-translator-1.78.0
behave-1.2.6
black-23.9.1
boto3-1.28.17
botocore-1.31.17
bytecode-0.15.1
cattrs-23.1.2
certifi-2023.7.22
cfn-lint-0.81.0
charset-normalizer-3.3.0
click-8.1.7
dacite-1.8.1
ddsketch-2.0.4
ddtrace-2.1.0
deprecated-1.2.14
dill-0.3.7
envier-0.4.0
exceptiongroup-1.1.3
frozenlist-1.4.0
idna-3.4
importlib-metadata-6.8.0
iniconfig-2.0.0
isort-5.12.0
jmespath-1.0.1
jschema-to-python-1.2.3
jsonpatch-1.33
jsonpickle-3.0.2
jsonpointer-2.4
jsonschema-4.19.1
jsonschema-specifications-2023.7.1
junit-xml-1.9
mccabe-0.7.0
mpmath-1.3.0
multidict-6.0.4
mypy-1.6.0
mypy-extensions-1.0.0
networkx-3.1
opentelemetry-api-1.20.0
packaging-23.2
parse-1.19.1
parse-type-0.6.2
pathspec-0.11.2
pbr-5.11.1
platformdirs-3.11.0
pluggy-1.3.0
protobuf-4.24.4
psutil-5.9.5
pydantic-2.4.2
pydantic-core-2.10.1
pylint-3.0.1
pytest-7.4.2
pytest-mock-3.11.1
python-dateutil-2.8.2
python-graphql-client-0.4.3
pyyaml-6.0.1
referencing-0.30.2
regex-2023.10.3
requests-2.31.0
rpds-py-0.10.6
s3transfer-0.6.2
sarif-om-1.0.4
six-1.16.0
structlog-20.2.0
sympy-1.12
tomli-2.0.1
tomlkit-0.12.1
typing-extensions-4.8.0
urllib3-1.26.17
websockets-11.0.3
wheel-0.41.2
wrapt-1.15.0
xmltodict-0.13.0
yarl-1.9.2
zipp-3.17.0

How can we reproduce your problem?

Deploy to AWS Lambda.

What is the result that you get?

Traceback (most recent call last):
  File "/var/lang/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 850, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/opt/python/lib/python3.9/site-packages/datadog_lambda/handler.py", line 10, in <module>
    from datadog_lambda.wrapper import datadog_lambda_wrapper
  File "/opt/python/lib/python3.9/site-packages/datadog_lambda/wrapper.py", line 23, in <module>
    from datadog_lambda.patch import patch_all
  File "/opt/python/lib/python3.9/site-packages/datadog_lambda/patch.py", line 14, in <module>
    from ddtrace import patch_all as patch_all_dd
  File "/var/task/ddtrace/__init__.py", line 24, in <module>
    from ddtrace.internal import telemetry
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "/var/task/ddtrace/internal/module.py", line 220, in _exec_module
    self.loader.exec_module(module)
  File "/var/task/ddtrace/internal/telemetry/__init__.py", line 12, in <module>
    from .writer import TelemetryWriter
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "/var/task/ddtrace/internal/module.py", line 220, in _exec_module
    self.loader.exec_module(module)
  File "/var/task/ddtrace/internal/telemetry/writer.py", line 19, in <module>
    from ...settings import _config as config
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "/var/task/ddtrace/internal/module.py", line 220, in _exec_module
    self.loader.exec_module(module)
  File "/var/task/ddtrace/settings/__init__.py", line 2, in <module>
    from .config import Config
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "/var/task/ddtrace/internal/module.py", line 220, in _exec_module
    self.loader.exec_module(module)
  File "/var/task/ddtrace/settings/config.py", line 188, in <module>
    class Config(object):
  File "/var/task/ddtrace/settings/config.py", line 195, in Config
    _extra_services_queue = multiprocessing.get_context("fork" if sys.platform != "win32" else "spawn").Queue(
  File "/var/lang/lib/python3.9/multiprocessing/context.py", line 103, in Queue
    return Queue(maxsize, ctx=self.get_context())
  File "/var/lang/lib/python3.9/multiprocessing/queues.py", line 43, in __init__
    self._rlock = ctx.Lock()
  File "/var/lang/lib/python3.9/multiprocessing/context.py", line 68, in Lock
    return Lock(ctx=self.get_context())
  File "/var/lang/lib/python3.9/multiprocessing/synchronize.py", line 162, in __init__
    SemLock.__init__(self, SEMAPHORE, 1, 1, ctx=ctx)
  File "/var/lang/lib/python3.9/multiprocessing/synchronize.py", line 57, in __init__
    sl = self._semlock = _multiprocessing.SemLock(

What is the result that you expected?

No error because DataDog is great :)

[hfern]

We are experiencing this as well. This library is not usable in Lambda. Cause seems to be #6929 and last updated by #7126.

[lautjy]

We just hit this issue after accepting our bot-created automated version updates - oof.
This error also exists in 2.0 versions. 1.20.10 was the highest I could go, but did not test every 2.0 version.

The above linked AWS documentation reveals what I'd wager is the root cause:

Due to the Lambda execution environment not having /dev/shm (shared memory for processes) support, you can’t use multiprocessing.Queue or multiprocessing.Pool.

They further claim that

.. you can use multiprocessing.Pipe instead of multiprocessing.Queue to accomplish what you need ...

[astuyve]

This should be fixed in v5.83/layer v83.

Note it's a major change as we move from ddtrace 1.x to 2.x, but presumably you're posting here because you have already attempted this.