Add support for generic OTEL_PYTHON_EXCLUDED_URLS variable (#790)

* Add support for generic OTEL_PYTHON_EXCLUDED_URLS variable

Use `OTEL_PYTHON_EXCLUDED_URLS` environment variable as a fallback of
`OTEL_PYTHON_{instrumentation}_EXCLUDED_URLS`.

* Only use generic variable when instrumentation-specific is not set

* Change wording on docs

Co-authored-by: Leighton Chen <lechen@microsoft.com>

CHANGELOG.md

@@ -18,6 +18,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
   ([#793](https://github.com/open-telemetry/opentelemetry-python-contrib/pull/793))
 - `opentelemetry-instrumentation-pymysql` Add support for PyMySQL 1.x series
   ([#792](https://github.com/open-telemetry/opentelemetry-python-contrib/pull/792))
+- Add support for generic OTEL_PYTHON_EXCLUDED_URLS variable
+  ([#790](https://github.com/open-telemetry/opentelemetry-python-contrib/pull/790))
 
 ### Fixed
 

instrumentation/opentelemetry-instrumentation-django/src/opentelemetry/instrumentation/django/__init__.py

@@ -32,7 +32,8 @@
 
 Exclude lists
 *************
-To exclude certain URLs from being tracked, set the environment variable ``OTEL_PYTHON_DJANGO_EXCLUDED_URLS`` with comma delimited regexes representing which URLs to exclude.
+To exclude certain URLs from being tracked, set the environment variable ``OTEL_PYTHON_DJANGO_EXCLUDED_URLS``
+(or ``OTEL_PYTHON_EXCLUDED_URLS`` as fallback) with comma delimited regexes representing which URLs to exclude.
 
 For example,
 

instrumentation/opentelemetry-instrumentation-falcon/README.rst

@@ -21,7 +21,8 @@ Configuration
 
 Exclude lists
 *************
-To exclude certain URLs from being tracked, set the environment variable ``OTEL_PYTHON_FALCON_EXCLUDED_URLS`` with comma delimited regexes representing which URLs to exclude.
+To exclude certain URLs from being tracked, set the environment variable ``OTEL_PYTHON_FALCON_EXCLUDED_URLS``
+(or ``OTEL_PYTHON_EXCLUDED_URLS`` as fallback) with comma delimited regexes representing which URLs to exclude.
 
 For example,
 
@@ -34,7 +35,7 @@ will exclude requests such as ``https://site/client/123/info`` and ``https://sit
 Request attributes
 ********************
 To extract certain attributes from Falcon's request object and use them as span attributes, set the environment variable ``OTEL_PYTHON_FALCON_TRACED_REQUEST_ATTRS`` to a comma
-delimited list of request attribute names. 
+delimited list of request attribute names.
 
 For example,
 

instrumentation/opentelemetry-instrumentation-falcon/src/opentelemetry/instrumentation/falcon/__init__.py

@@ -26,7 +26,8 @@
 
 Exclude lists
 *************
-To exclude certain URLs from being tracked, set the environment variable ``OTEL_PYTHON_FALCON_EXCLUDED_URLS`` with comma delimited regexes representing which URLs to exclude.
+To exclude certain URLs from being tracked, set the environment variable ``OTEL_PYTHON_FALCON_EXCLUDED_URLS``
+(or ``OTEL_PYTHON_EXCLUDED_URLS`` as fallback) with comma delimited regexes representing which URLs to exclude.
 
 For example,
 

instrumentation/opentelemetry-instrumentation-fastapi/src/opentelemetry/instrumentation/fastapi/__init__.py

@@ -34,7 +34,8 @@ async def foobar():
 
 Exclude lists
 *************
-To exclude certain URLs from being tracked, set the environment variable ``OTEL_PYTHON_FASTAPI_EXCLUDED_URLS`` with comma delimited regexes representing which URLs to exclude.
+To exclude certain URLs from being tracked, set the environment variable ``OTEL_PYTHON_FASTAPI_EXCLUDED_URLS``
+(or ``OTEL_PYTHON_EXCLUDED_URLS`` as fallback) with comma delimited regexes representing which URLs to exclude.
 
 For example,
 

instrumentation/opentelemetry-instrumentation-flask/README.rst

@@ -21,7 +21,8 @@ Configuration
 
 Exclude lists
 *************
-To exclude certain URLs from being tracked, set the environment variable ``OTEL_PYTHON_FLASK_EXCLUDED_URLS`` with comma delimited regexes representing which URLs to exclude.
+To exclude certain URLs from being tracked, set the environment variable ``OTEL_PYTHON_FLASK_EXCLUDED_URLS``
+(or ``OTEL_PYTHON_EXCLUDED_URLS`` as fallback) with comma delimited regexes representing which URLs to exclude.
 
 For example,
 

instrumentation/opentelemetry-instrumentation-flask/src/opentelemetry/instrumentation/flask/__init__.py

@@ -48,7 +48,8 @@ def hello():
 
 Exclude lists
 *************
-To exclude certain URLs from being tracked, set the environment variable ``OTEL_PYTHON_FLASK_EXCLUDED_URLS`` with comma delimited regexes representing which URLs to exclude.
+To exclude certain URLs from being tracked, set the environment variable ``OTEL_PYTHON_FLASK_EXCLUDED_URLS``
+(or ``OTEL_PYTHON_EXCLUDED_URLS`` as fallback) with comma delimited regexes representing which URLs to exclude.
 
 For example,
 

instrumentation/opentelemetry-instrumentation-pyramid/src/opentelemetry/instrumentation/pyramid/__init__.py

@@ -79,7 +79,8 @@
 
 Exclude lists
 *************
-To exclude certain URLs from being tracked, set the environment variable ``OTEL_PYTHON_PYRAMID_EXCLUDED_URLS`` with comma delimited regexes representing which URLs to exclude.
+To exclude certain URLs from being tracked, set the environment variable ``OTEL_PYTHON_PYRAMID_EXCLUDED_URLS``
+(or ``OTEL_PYTHON_EXCLUDED_URLS`` as fallback) with comma delimited regexes representing which URLs to exclude.
 
 For example,
 

instrumentation/opentelemetry-instrumentation-requests/README.rst

@@ -15,13 +15,14 @@ Installation
 ::
 
      pip install opentelemetry-instrumentation-requests
-     
+
 Configuration
 -------------
 
 Exclude lists
 *************
-To exclude certain URLs from being tracked, set the environment variable ``OTEL_PYTHON_REQUESTS_EXCLUDED_URLS`` with comma delimited regexes representing which URLs to exclude.
+To exclude certain URLs from being tracked, set the environment variable ``OTEL_PYTHON_REQUESTS_EXCLUDED_URLS``
+(or ``OTEL_PYTHON_EXCLUDED_URLS`` as fallback) with comma delimited regexes representing which URLs to exclude.
 
 For example,
 

instrumentation/opentelemetry-instrumentation-requests/src/opentelemetry/instrumentation/requests/__init__.py

@@ -33,7 +33,8 @@
 
 Exclude lists
 *************
-To exclude certain URLs from being tracked, set the environment variable ``OTEL_PYTHON_REQUESTS_EXCLUDED_URLS`` with comma delimited regexes representing which URLs to exclude.
+To exclude certain URLs from being tracked, set the environment variable ``OTEL_PYTHON_REQUESTS_EXCLUDED_URLS``
+(or ``OTEL_PYTHON_EXCLUDED_URLS`` as fallback) with comma delimited regexes representing which URLs to exclude.
 
 For example,
 

instrumentation/opentelemetry-instrumentation-starlette/src/opentelemetry/instrumentation/starlette/__init__.py

@@ -36,7 +36,8 @@ def home(request):
 
 Exclude lists
 *************
-To exclude certain URLs from being tracked, set the environment variable ``OTEL_PYTHON_STARLETTE_EXCLUDED_URLS`` with comma delimited regexes representing which URLs to exclude.
+To exclude certain URLs from being tracked, set the environment variable ``OTEL_PYTHON_STARLETTE_EXCLUDED_URLS``
+(or ``OTEL_PYTHON_EXCLUDED_URLS`` as fallback) with comma delimited regexes representing which URLs to exclude.
 
 For example,
 

util/opentelemetry-util-http/src/opentelemetry/util/http/__init__.py

@@ -15,13 +15,14 @@
 from os import environ
 from re import compile as re_compile
 from re import search
+from typing import Iterable
 from urllib.parse import urlparse, urlunparse
 
 
 class ExcludeList:
     """Class to exclude certain paths (given as a list of regexes) from tracing requests"""
 
-    def __init__(self, excluded_urls):
+    def __init__(self, excluded_urls: Iterable[str]):
         self._excluded_urls = excluded_urls
         if self._excluded_urls:
             self._regex = re_compile("|".join(excluded_urls))
@@ -47,24 +48,29 @@ def get_traced_request_attrs(instrumentation):
     return traced_request_attrs
 
 
-def get_excluded_urls(instrumentation):
+def get_excluded_urls(instrumentation: str) -> ExcludeList:
+    # Get instrumentation-specific excluded URLs. If not set, retrieve them
+    # from generic variable.
     excluded_urls = environ.get(
-        _root.format(f"{instrumentation}_EXCLUDED_URLS"), []
+        _root.format(f"{instrumentation}_EXCLUDED_URLS"),
+        environ.get(_root.format("EXCLUDED_URLS"), ""),
     )
 
     return parse_excluded_urls(excluded_urls)
 
 
-def parse_excluded_urls(excluded_urls):
+def parse_excluded_urls(excluded_urls: str) -> ExcludeList:
     """
     Small helper to put an arbitrary url list inside of ExcludeList
     """
     if excluded_urls:
-        excluded_urls = [
+        excluded_url_list = [
             excluded_url.strip() for excluded_url in excluded_urls.split(",")
         ]
+    else:
+        excluded_url_list = []
 
-    return ExcludeList(excluded_urls)
+    return ExcludeList(excluded_url_list)
 
 
 def remove_url_credentials(url: str) -> str:

util/opentelemetry-util-http/tests/test_http_excluded_urls.py

@@ -0,0 +1,72 @@
+# Copyright The OpenTelemetry Authors
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from unittest.mock import patch
+
+from opentelemetry.test.test_base import TestBase
+from opentelemetry.util.http import get_excluded_urls
+
+
+class TestGetExcludedUrls(TestBase):
+    @patch.dict(
+        "os.environ",
+        {
+            "OTEL_PYTHON_DJANGO_EXCLUDED_URLS": "excluded_arg/123,excluded_noarg"
+        },
+    )
+    def test_config_from_instrumentation_env(self):
+        exclude_list = get_excluded_urls("DJANGO")
+
+        self.assertTrue(exclude_list.url_disabled("/excluded_arg/123"))
+        self.assertTrue(exclude_list.url_disabled("/excluded_noarg"))
+        self.assertFalse(exclude_list.url_disabled("/excluded_arg/125"))
+
+    @patch.dict(
+        "os.environ",
+        {"OTEL_PYTHON_EXCLUDED_URLS": "excluded_arg/123,excluded_noarg"},
+    )
+    def test_config_from_generic_env(self):
+        exclude_list = get_excluded_urls("DJANGO")
+
+        self.assertTrue(exclude_list.url_disabled("/excluded_arg/123"))
+        self.assertTrue(exclude_list.url_disabled("/excluded_noarg"))
+        self.assertFalse(exclude_list.url_disabled("/excluded_arg/125"))
+
+    @patch.dict(
+        "os.environ",
+        {
+            "OTEL_PYTHON_DJANGO_EXCLUDED_URLS": "excluded_arg/123,excluded_noarg",
+            "OTEL_PYTHON_EXCLUDED_URLS": "excluded_arg/125",
+        },
+    )
+    def test_config_from_instrumentation_env_takes_precedence(self):
+        exclude_list = get_excluded_urls("DJANGO")
+
+        self.assertTrue(exclude_list.url_disabled("/excluded_arg/123"))
+        self.assertTrue(exclude_list.url_disabled("/excluded_noarg"))
+        self.assertFalse(exclude_list.url_disabled("/excluded_arg/125"))
+
+    @patch.dict(
+        "os.environ",
+        {
+            "OTEL_PYTHON_DJANGO_EXCLUDED_URLS": "",
+            "OTEL_PYTHON_EXCLUDED_URLS": "excluded_arg/125",
+        },
+    )
+    def test_config_from_instrumentation_env_empty(self):
+        exclude_list = get_excluded_urls("DJANGO")
+
+        self.assertFalse(exclude_list.url_disabled("/excluded_arg/123"))
+        self.assertFalse(exclude_list.url_disabled("/excluded_noarg"))
+        self.assertFalse(exclude_list.url_disabled("/excluded_arg/125"))