Merge pull request #31 from br3ndonland/conf-testing

Refactor configuration files and tests

README.md

@@ -260,10 +260,10 @@ ENV APP_MODULE="package.custom.module:api" WORKERS_PER_CORE="2"
   - Custom:
     - `GUNICORN_CONF="/app/package/custom_gunicorn_conf.py"`
 - [Gunicorn worker processes](https://docs.gunicorn.org/en/latest/settings.html#worker-processes): The number of Gunicorn worker processes to run is determined based on the `MAX_WORKERS`, `WEB_CONCURRENCY`, and `WORKERS_PER_CORE` environment variables, with a default of 1 worker per CPU core and a default minimum of 2. This is the "performance auto-tuning" feature described in [tiangolo/uvicorn-gunicorn-docker](https://github.com/tiangolo/uvicorn-gunicorn-docker).
-  - `MAX_WORKERS`: Maximum number of workers to use, independent of number of CPU cores.
+  - `MAX_WORKERS`: Maximum number of workers, independent of number of CPU cores.
     - Default: not set (unlimited)
     - Custom: `MAX_WORKERS="24"`
-  - `WEB_CONCURRENCY`: Set number of workers independently of number of CPU cores.
+  - `WEB_CONCURRENCY`: Total number of workers, independent of number of CPU cores.
     - Default: not set
     - Custom: `WEB_CONCURRENCY="4"`
   - `WORKERS_PER_CORE`: Number of Gunicorn workers per CPU core. Overridden if `WEB_CONCURRENCY` is set.
@@ -272,7 +272,7 @@ ENV APP_MODULE="package.custom.module:api" WORKERS_PER_CORE="2"
       - `WORKERS_PER_CORE="2"`: Run 2 worker processes per core (8 worker processes on a server with 4 cores).
       - `WORKERS_PER_CORE="0.5"` (floating point values permitted): Run 1 worker process for every 2 cores (2 worker processes on a server with 4 cores).
   - Notes:
-    - The default number of workers is the number of CPU cores multiplied by the environment variable `WORKERS_PER_CORE="1"`. On a machine with only 1 CPU core, the default minimum number of workers is 2 to avoid poor performance and blocking, as explained in the release notes for [tiangolo/uvicorn-gunicorn-docker 0.3.0](https://github.com/tiangolo/uvicorn-gunicorn-docker/releases/tag/0.3.0).
+    - The default number of workers is the number of CPU cores multiplied by the value of the environment variable `WORKERS_PER_CORE` (which defaults to 1). On a machine with only 1 CPU core, the default minimum number of workers is 2 to avoid poor performance and blocking, as explained in the release notes for [tiangolo/uvicorn-gunicorn-docker 0.3.0](https://github.com/tiangolo/uvicorn-gunicorn-docker/releases/tag/0.3.0).
     - If both `MAX_WORKERS` and `WEB_CONCURRENCY` are set, the least of the two will be used as the total number of workers.
     - If either `MAX_WORKERS` or `WEB_CONCURRENCY` are set to 1, the total number of workers will be 1, overriding the default minimum of 2.
 - `PROCESS_MANAGER`: Manager for Uvicorn worker processes. As described in the [Uvicorn docs](https://www.uvicorn.org), "Uvicorn includes a Gunicorn worker class allowing you to run ASGI applications, with all of Uvicorn's performance benefits, while also giving you Gunicorn's fully-featured process management."
@@ -452,17 +452,18 @@ See _[CONTRIBUTING.md](./.github/CONTRIBUTING.md)_ for general information on ho
 ### Testing with pytest
 
 - Tests are in the _tests/_ directory.
-- Run tests by [invoking `pytest` from the command-line](https://docs.pytest.org/en/stable/usage.html) within the Poetry environment in the root directory of the repo.
+- Run tests by [invoking `pytest` from the command-line](https://docs.pytest.org/en/latest/how-to/usage.html) within the Poetry environment in the root directory of the repo.
 - [pytest](https://docs.pytest.org/en/latest/) features used include:
-  - [fixtures](https://docs.pytest.org/en/latest/fixture.html)
-  - [monkeypatch](https://docs.pytest.org/en/latest/monkeypatch.html)
-  - [parametrize](https://docs.pytest.org/en/latest/parametrize.html)
-  - [`tmp_path`](https://docs.pytest.org/en/latest/tmpdir.html)
-- [pytest plugins](https://docs.pytest.org/en/stable/plugins.html) include:
+  - [capturing `stdout` with `capfd`](https://docs.pytest.org/en/latest/how-to/capture-stdout-stderr.html)
+  - [fixtures](https://docs.pytest.org/en/latest/how-to/fixtures.html)
+  - [monkeypatch](https://docs.pytest.org/en/latest/how-to/monkeypatch.html)
+  - [parametrize](https://docs.pytest.org/en/latest/how-to/parametrize.html)
+  - [temporary directories and files (`tmp_path` and `tmp_dir`)](https://docs.pytest.org/en/latest/how-to/tmpdir.html)
+- [pytest plugins](https://docs.pytest.org/en/latest/how-to/plugins.html) include:
   - [pytest-cov](https://github.com/pytest-dev/pytest-cov)
   - [pytest-mock](https://github.com/pytest-dev/pytest-mock)
-- [pytest configuration](https://docs.pytest.org/en/stable/customize.html) is in _[pyproject.toml](https://github.com/br3ndonland/inboard/blob/develop/pyproject.toml)_.
-- [FastAPI testing](https://fastapi.tiangolo.com/tutorial/testing/) and [Starlette testing](https://www.starlette.io/testclient/) rely on the [Starlette `TestClient`](https://www.starlette.io/testclient/), which uses [Requests](https://requests.readthedocs.io/en/master/) under the hood.
+- [pytest configuration](https://docs.pytest.org/en/latest/reference/customize.html) is in _[pyproject.toml](https://github.com/br3ndonland/inboard/blob/develop/pyproject.toml)_.
+- [FastAPI testing](https://fastapi.tiangolo.com/tutorial/testing/) and [Starlette testing](https://www.starlette.io/testclient/) rely on the [Starlette `TestClient`](https://www.starlette.io/testclient/).
 - Test coverage results are reported when invoking `pytest` from the command-line. To see interactive HTML coverage reports, invoke pytest with `pytest --cov-report=html`.
 - Test coverage reports are generated within GitHub Actions workflows by [pytest-cov](https://github.com/pytest-dev/pytest-cov) with [coverage.py](https://github.com/nedbat/coveragepy), and uploaded to [Codecov](https://docs.codecov.io/docs) using [codecov/codecov-action](https://github.com/marketplace/actions/codecov). Codecov is then integrated into pull requests with the [Codecov GitHub app](https://github.com/marketplace/codecov).
 

inboard/gunicorn_conf.py

@@ -2,57 +2,35 @@
 import os
 from typing import Optional
 
-from inboard.start import configure_logging
+from inboard.logging_conf import configure_logging
 
 
 def calculate_workers(
-    max_workers_str: Optional[str],
-    web_concurrency_str: Optional[str],
-    workers_per_core_str: str,
-    cores: int = multiprocessing.cpu_count(),
+    max_workers: Optional[str] = None,
+    total_workers: Optional[str] = None,
+    workers_per_core: str = "1",
 ) -> int:
     """Calculate the number of Gunicorn worker processes."""
-    use_default_workers = max(int(float(workers_per_core_str) * cores), 2)
-    if max_workers_str and int(max_workers_str) > 0:
-        use_max_workers = int(max_workers_str)
-    if web_concurrency_str and int(web_concurrency_str) > 0:
-        use_web_concurrency = int(web_concurrency_str)
-    return (
-        min(use_max_workers, use_web_concurrency)
-        if max_workers_str and web_concurrency_str
-        else use_web_concurrency
-        if web_concurrency_str
-        else use_default_workers
-    )
+    cores = multiprocessing.cpu_count()
+    use_default = max(int(float(workers_per_core) * cores), 2)
+    use_max = m if max_workers and (m := int(max_workers)) > 0 else False
+    use_total = t if total_workers and (t := int(total_workers)) > 0 else False
+    use_least = min(use_max, use_total) if use_max and use_total else False
+    return use_least or use_max or use_total or use_default
 
 
-# Gunicorn setup
-max_workers_str = os.getenv("MAX_WORKERS")
-web_concurrency_str = os.getenv("WEB_CONCURRENCY")
-workers_per_core_str = os.getenv("WORKERS_PER_CORE", "1")
-workers = calculate_workers(max_workers_str, web_concurrency_str, workers_per_core_str)
+# Gunicorn settings
+bind = os.getenv("BIND") or f'{os.getenv("HOST", "0.0.0.0")}:{os.getenv("PORT", "80")}'
+accesslog = os.getenv("ACCESS_LOG", "-")
+errorlog = os.getenv("ERROR_LOG", "-")
+graceful_timeout = int(os.getenv("GRACEFUL_TIMEOUT", "120"))
+keepalive = int(os.getenv("KEEP_ALIVE", "5"))
+logconfig_dict = configure_logging()
+loglevel = os.getenv("LOG_LEVEL", "info")
+timeout = int(os.getenv("TIMEOUT", "120"))
 worker_tmp_dir = "/dev/shm"
-host = os.getenv("HOST", "0.0.0.0")
-port = os.getenv("PORT", "80")
-bind_env = os.getenv("BIND")
-use_bind = bind_env or f"{host}:{port}"
-use_loglevel = os.getenv("LOG_LEVEL", "info")
-accesslog_var = os.getenv("ACCESS_LOG", "-")
-use_accesslog = accesslog_var or None
-errorlog_var = os.getenv("ERROR_LOG", "-")
-use_errorlog = errorlog_var or None
-graceful_timeout_str = os.getenv("GRACEFUL_TIMEOUT", "120")
-timeout_str = os.getenv("TIMEOUT", "120")
-keepalive_str = os.getenv("KEEP_ALIVE", "5")
-
-# Gunicorn config variables
-logconfig_dict = configure_logging(
-    logging_conf=os.getenv("LOGGING_CONF", "inboard.logging_conf")
+workers = calculate_workers(
+    os.getenv("MAX_WORKERS"),
+    os.getenv("WEB_CONCURRENCY"),
+    workers_per_core=os.getenv("WORKERS_PER_CORE", "1"),
 )
-loglevel = use_loglevel
-bind = use_bind
-errorlog = use_errorlog
-accesslog = use_accesslog
-graceful_timeout = int(graceful_timeout_str)
-timeout = int(timeout_str)
-keepalive = int(keepalive_str)

inboard/logging_conf.py

@@ -1,5 +1,52 @@
+import importlib.util
+import logging
+import logging.config
 import os
 import sys
+from pathlib import Path
+from typing import Optional
+
+
+def find_and_load_logging_conf(logging_conf: str) -> dict:
+    """Find and load a logging configuration module or file."""
+    logging_conf_path = Path(logging_conf)
+    spec = (
+        importlib.util.spec_from_file_location("confspec", logging_conf_path)
+        if logging_conf_path.is_file() and logging_conf_path.suffix == ".py"
+        else importlib.util.find_spec(logging_conf)
+    )
+    if not spec:
+        raise ImportError(f"Unable to import {logging_conf_path}")
+    logging_conf_module = importlib.util.module_from_spec(spec)
+    exec_module = getattr(spec.loader, "exec_module")
+    exec_module(logging_conf_module)
+    if not hasattr(logging_conf_module, "LOGGING_CONFIG"):
+        raise AttributeError(f"No LOGGING_CONFIG in {logging_conf_module.__name__}")
+    logging_conf_dict = getattr(logging_conf_module, "LOGGING_CONFIG")
+    if not isinstance(logging_conf_dict, dict):
+        raise TypeError("LOGGING_CONFIG is not a dictionary instance")
+    return logging_conf_dict
+
+
+def configure_logging(
+    logger: logging.Logger = logging.getLogger(),
+    logging_conf: Optional[str] = os.getenv("LOGGING_CONF"),
+) -> dict:
+    """Configure Python logging given the name of a logging module or file."""
+    try:
+        if not logging_conf:
+            logging_conf_path = __name__
+            logging_conf_dict = LOGGING_CONFIG
+        else:
+            logging_conf_path = logging_conf
+            logging_conf_dict = find_and_load_logging_conf(logging_conf_path)
+        logging.config.dictConfig(logging_conf_dict)
+        logger.debug(f"Logging dict config loaded from {logging_conf_path}.")
+        return logging_conf_dict
+    except Exception as e:
+        logger.error(f"Error when setting logging module: {e.__class__.__name__} {e}.")
+        raise
+
 
 LOG_COLORS = (
     True

inboard/start.py

@@ -1,42 +1,14 @@
 #!/usr/bin/env python3
 import importlib.util
 import logging
-import logging.config
 import os
 import subprocess
 from pathlib import Path
 from typing import Optional
 
 import uvicorn  # type: ignore
 
-
-def configure_logging(
-    logger: logging.Logger = logging.getLogger(),
-    logging_conf: str = os.getenv("LOGGING_CONF", "inboard.logging_conf"),
-) -> dict:
-    """Configure Python logging based on a path to a logging module or file."""
-    try:
-        logging_conf_path = Path(logging_conf)
-        spec = (
-            importlib.util.spec_from_file_location("confspec", logging_conf_path)
-            if logging_conf_path.is_file() and logging_conf_path.suffix == ".py"
-            else importlib.util.find_spec(logging_conf)
-        )
-        if not spec:
-            raise ImportError(f"Unable to import {logging_conf}")
-        logging_conf_module = importlib.util.module_from_spec(spec)
-        spec.loader.exec_module(logging_conf_module)  # type: ignore[union-attr]
-        if not hasattr(logging_conf_module, "LOGGING_CONFIG"):
-            raise AttributeError(f"No LOGGING_CONFIG in {logging_conf_module.__name__}")
-        logging_conf_dict = getattr(logging_conf_module, "LOGGING_CONFIG")
-        if not isinstance(logging_conf_dict, dict):
-            raise TypeError("LOGGING_CONFIG is not a dictionary instance")
-        logging.config.dictConfig(logging_conf_dict)
-        logger.debug(f"Logging dict config loaded from {logging_conf_path}.")
-        return logging_conf_dict
-    except Exception as e:
-        logger.error(f"Error when setting logging module: {e.__class__.__name__} {e}.")
-        raise
+from inboard.logging_conf import configure_logging
 
 
 def run_pre_start_script(logger: logging.Logger = logging.getLogger()) -> str:
@@ -68,34 +40,37 @@ def set_app_module(logger: logging.Logger = logging.getLogger()) -> str:
         raise
 
 
-def set_gunicorn_options() -> list:
+def set_gunicorn_options(app_module: str) -> list:
     """Set options for running the Gunicorn server."""
     gunicorn_conf_path = os.getenv("GUNICORN_CONF", "/app/inboard/gunicorn_conf.py")
     worker_class = os.getenv("WORKER_CLASS", "uvicorn.workers.UvicornWorker")
     if not Path(gunicorn_conf_path).is_file():
         raise FileNotFoundError(f"Unable to find {gunicorn_conf_path}")
-    return ["gunicorn", "-k", worker_class, "-c", gunicorn_conf_path]
+    return ["gunicorn", "-k", worker_class, "-c", gunicorn_conf_path, app_module]
 
 
 def set_uvicorn_options(log_config: Optional[dict] = None) -> dict:
     """Set options for running the Uvicorn server."""
-    with_reload = (
-        True
-        if (value := os.getenv("WITH_RELOAD")) and value.lower() == "true"
-        else False
-    )
+    host = os.getenv("HOST", "0.0.0.0")
+    port = int(os.getenv("PORT", "80"))
+    log_level = os.getenv("LOG_LEVEL", "info")
     reload_dirs = (
         [d.lstrip() for d in str(os.getenv("RELOAD_DIRS")).split(sep=",")]
         if os.getenv("RELOAD_DIRS")
         else None
     )
+    use_reload = (
+        True
+        if (value := os.getenv("WITH_RELOAD")) and value.lower() == "true"
+        else False
+    )
     return dict(
-        host=os.getenv("HOST", "0.0.0.0"),
-        port=int(os.getenv("PORT", "80")),
+        host=host,
+        port=port,
         log_config=log_config,
-        log_level=os.getenv("LOG_LEVEL", "info"),
-        reload=with_reload,
+        log_level=log_level,
         reload_dirs=reload_dirs,
+        reload=use_reload,
     )
 
 
@@ -109,8 +84,7 @@ def start_server(
     try:
         if process_manager == "gunicorn":
             logger.debug("Running Uvicorn with Gunicorn.")
-            gunicorn_options: list = set_gunicorn_options()
-            gunicorn_options.append(app_module)
+            gunicorn_options: list = set_gunicorn_options(app_module)
             subprocess.run(gunicorn_options)
         elif process_manager == "uvicorn":
             logger.debug("Running Uvicorn without Gunicorn.")

tests/app/test_main.py

@@ -110,7 +110,7 @@ class TestEndpoints:
     ---
     See the [FastAPI testing docs](https://fastapi.tiangolo.com/tutorial/testing/),
     [Starlette TestClient docs](https://www.starlette.io/testclient/), and the
-    [pytest docs on parametrize](https://docs.pytest.org/en/latest/parametrize.html).
+    [pytest docs](https://docs.pytest.org/en/latest/how-to/parametrize.html).
     """
 
     def test_get_asgi_uvicorn(