Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add PyprojectTomlConfigSettingsSource #255

Merged
merged 8 commits into from
Mar 28, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
124 changes: 123 additions & 1 deletion docs/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -539,9 +539,10 @@ docker service create --name pydantic-with-secrets --secret my_secret_data pydan

Other settings sources are available for common configuration files:

- `JsonConfigSettingsSource` using `json_file` and `json_file_encoding` arguments
- `PyprojectTomlConfigSettingsSource` using *(optional)* `pyproject_toml_depth` and *(optional)* `pyproject_toml_table_header` arguments
- `TomlConfigSettingsSource` using `toml_file` argument
- `YamlConfigSettingsSource` using `yaml_file` and yaml_file_encoding arguments
- `JsonConfigSettingsSource` using `json_file` and `json_file_encoding` arguments

You can also provide multiple files by providing a list of path:
```py
Expand Down Expand Up @@ -592,6 +593,127 @@ foobar = "Hello"
nested_field = "world!"
```

### pyproject.toml

"pyproject.toml" is a standardized file for providing configuration values in Python projects.
[PEP 518](https://peps.python.org/pep-0518/#tool-table) defines a `[tool]` table that can be used to provide arbitrary tool configuration.
While encouraged to use the `[tool]` table, `PyprojectTomlConfigSettingsSource` can be used to load variables from any location with in "pyproject.toml" file.

This is controlled by providing `SettingsConfigDict(pyproject_toml_table_header=tuple[str, ...])` where the value is a tuple of header parts.
By default, `pyproject_toml_table_header=('tool', 'pydantic-settings')` which will load variables from the `[tool.pydantic-settings]` table.

```python
from typing import Tuple, Type

from pydantic_settings import (
BaseSettings,
PydanticBaseSettingsSource,
PyprojectTomlConfigSettingsSource,
SettingsConfigDict,
)


class Settings(BaseSettings):
"""Example loading values from the table used by default."""

field: str

@classmethod
def settings_customise_sources(
cls,
settings_cls: Type[BaseSettings],
init_settings: PydanticBaseSettingsSource,
env_settings: PydanticBaseSettingsSource,
dotenv_settings: PydanticBaseSettingsSource,
file_secret_settings: PydanticBaseSettingsSource,
) -> Tuple[PydanticBaseSettingsSource, ...]:
return (PyprojectTomlConfigSettingsSource(settings_cls),)


class SomeTableSettings(Settings):
"""Example loading values from a user defined table."""

model_config = SettingsConfigDict(
pyproject_toml_table_header=('tool', 'some-table')
)


class RootSettings(Settings):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why did you define the RootSettings?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like I forgot to update the example code after adding a default. This is to show how to load from different locations. Updating this in an incoming commit plus adding a third example to show another use case.

  • Settings for using the default table
  • SomeTableSettings replaces what Settings was doing before, providing a pyproject_toml_table_header config value with a different table header
  • RootSettings updated to work as I had it before setting a default header, loading from the root of pyproject.toml

"""Example loading values from the root of a pyproject.toml file."""

model_config = SettingsConfigDict(extra='ignore', pyproject_toml_table_header=())
```

This will be able to read the following "pyproject.toml" file, located in your working directory, resulting in `Settings(field='default-table')`, `SomeTableSettings(field='some-table')`, & `RootSettings(field='root')`:

```toml
field = "root"

[tool.pydantic-settings]
field = "default-table"

[tool.some-table]
field = "some-table"
```

By default, `PyprojectTomlConfigSettingsSource` will only look for a "pyproject.toml" in the your current working directory.
However, there are two options to change this behavior.

* `SettingsConfigDict(pyproject_toml_depth=<int>)` can be provided to check `<int>` number of directories **up** in the directory tree for a "pyproject.toml" if one is not found in the current working directory.
By default, no parent directories are checked.
* An explicit file path can be provided to the source when it is instantiated (e.g. `PyprojectTomlConfigSettingsSource(settings_cls, Path('~/.config').resolve() / 'pyproject.toml')`).
If a file path is provided this way, it will be treated as absolute (no other locations are checked).

```python
from pathlib import Path
from typing import Tuple, Type

from pydantic_settings import (
BaseSettings,
PydanticBaseSettingsSource,
PyprojectTomlConfigSettingsSource,
SettingsConfigDict,
)


class DiscoverSettings(BaseSettings):
"""Example of discovering a pyproject.toml in parent directories in not in `Path.cwd()`."""

model_config = SettingsConfigDict(pyproject_toml_depth=2)

@classmethod
def settings_customise_sources(
cls,
settings_cls: Type[BaseSettings],
init_settings: PydanticBaseSettingsSource,
env_settings: PydanticBaseSettingsSource,
dotenv_settings: PydanticBaseSettingsSource,
file_secret_settings: PydanticBaseSettingsSource,
) -> Tuple[PydanticBaseSettingsSource, ...]:
return (PyprojectTomlConfigSettingsSource(settings_cls),)


class ExplicitFilePathSettings(BaseSettings):
"""Example of explicitly providing the path to the file to load."""

field: str

@classmethod
def settings_customise_sources(
cls,
settings_cls: Type[BaseSettings],
init_settings: PydanticBaseSettingsSource,
env_settings: PydanticBaseSettingsSource,
dotenv_settings: PydanticBaseSettingsSource,
file_secret_settings: PydanticBaseSettingsSource,
) -> Tuple[PydanticBaseSettingsSource, ...]:
return (
PyprojectTomlConfigSettingsSource(
settings_cls, Path('~/.config').resolve() / 'pyproject.toml'
),
)
```

## Field value priority

In the case where a value is specified for the same `Settings` field in multiple ways,
Expand Down
2 changes: 2 additions & 0 deletions pydantic_settings/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@
InitSettingsSource,
JsonConfigSettingsSource,
PydanticBaseSettingsSource,
PyprojectTomlConfigSettingsSource,
SecretsSettingsSource,
TomlConfigSettingsSource,
YamlConfigSettingsSource,
Expand All @@ -17,6 +18,7 @@
'EnvSettingsSource',
'InitSettingsSource',
'JsonConfigSettingsSource',
'PyprojectTomlConfigSettingsSource',
'PydanticBaseSettingsSource',
'SecretsSettingsSource',
'SettingsConfigDict',
Expand Down
20 changes: 20 additions & 0 deletions pydantic_settings/main.py
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,26 @@ class SettingsConfigDict(ConfigDict, total=False):
json_file_encoding: str | None
yaml_file: PathType | None
yaml_file_encoding: str | None
pyproject_toml_depth: int
"""
Number of levels **up** from the current working directory to attempt to find a pyproject.toml
file.

This is only used when a pyproject.toml file is not found in the current working directory.
"""

pyproject_toml_table_header: tuple[str, ...]
"""
Header of the TOML table within a pyproject.toml file to use when filling variables.
This is supplied as a `tuple[str, ...]` instead of a `str` to accommodate for headers
containing a `.`.

For example, `toml_table_header = ("tool", "my.tool", "foo")` can be used to fill variable
values from a table with header `[tool."my.tool".foo]`.

To use the root table, exclude this config setting or provide an empty tuple.
"""

toml_file: PathType | None


Expand Down
48 changes: 47 additions & 1 deletion pydantic_settings/sources.py
Original file line number Diff line number Diff line change
Expand Up @@ -783,7 +783,7 @@ def _read_file(self, file_path: Path) -> dict[str, Any]:

class TomlConfigSettingsSource(InitSettingsSource, ConfigFileSourceMixin):
"""
A source class that loads variables from a JSON file
A source class that loads variables from a TOML file
"""

def __init__(
Expand All @@ -803,6 +803,52 @@ def _read_file(self, file_path: Path) -> dict[str, Any]:
return tomllib.load(toml_file)


class PyprojectTomlConfigSettingsSource(TomlConfigSettingsSource):
"""
A source class that loads variables from a `pyproject.toml` file.
"""

def __init__(
self,
settings_cls: type[BaseSettings],
toml_file: Path | None = None,
) -> None:
self.toml_file_path = self._pick_pyproject_toml_file(
toml_file, settings_cls.model_config.get('pyproject_toml_depth', 0)
)
self.toml_table_header: tuple[str, ...] = settings_cls.model_config.get(
'pyproject_toml_table_header', ('tool', 'pydantic-settings')
)
self.toml_data = self._read_files(self.toml_file_path)
for key in self.toml_table_header:
self.toml_data = self.toml_data.get(key, {})
super(TomlConfigSettingsSource, self).__init__(settings_cls, self.toml_data)

@staticmethod
def _pick_pyproject_toml_file(provided: Path | None, depth: int) -> Path:
"""Pick a `pyproject.toml` file path to use.

Args:
provided: Explicit path provided when instantiating this class.
depth: Number of directories up the tree to check of a pyproject.toml.

"""
if provided:
return provided.resolve()
rv = Path.cwd() / 'pyproject.toml'
count = 0
if not rv.is_file():
child = rv.parent.parent / 'pyproject.toml'
while count < depth:
if child.is_file():
return child
if str(child.parent) == rv.root:
break # end discovery after checking system root once
child = child.parent.parent / 'pyproject.toml'
count += 1
return rv


class YamlConfigSettingsSource(InitSettingsSource, ConfigFileSourceMixin):
"""
A source class that loads variables from a yaml file
Expand Down
35 changes: 35 additions & 0 deletions tests/conftest.py
Original file line number Diff line number Diff line change
@@ -1,7 +1,14 @@
from __future__ import annotations

import os
from pathlib import Path
from typing import TYPE_CHECKING

import pytest

if TYPE_CHECKING:
from collections.abc import Iterator


class SetEnv:
def __init__(self):
Expand All @@ -20,6 +27,34 @@ def clear(self):
os.environ.pop(n)


@pytest.fixture
def cd_tmp_path(tmp_path: Path) -> Iterator[Path]:
"""Change directory into the value of the ``tmp_path`` fixture.

.. rubric:: Example
.. code-block:: python

from typing import TYPE_CHECKING

if TYPE_CHECKING:
from pathlib import Path


def test_something(cd_tmp_path: Path) -> None:
...

Returns:
Value of the :fixture:`tmp_path` fixture (a :class:`~pathlib.Path` object).

"""
prev_dir = Path.cwd()
os.chdir(tmp_path)
try:
yield tmp_path
finally:
os.chdir(prev_dir)


@pytest.fixture
def env():
setenv = SetEnv()
Expand Down
Loading
Loading