Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Split core functionality and support orjson and msgspec #9

Merged
merged 27 commits into from
May 5, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
27 commits
Select commit Hold shift + click to select a range
60893a5
Refactor into core and json module, add orjson support
nhairs Apr 27, 2024
7f7ea2a
Format tests
nhairs Apr 27, 2024
e9aff5b
Allow tests to complete on all platforms
nhairs Apr 27, 2024
eaf56e2
Fix broken GHA spec
nhairs Apr 27, 2024
c36d6da
Don't support defaults kwargs (py310+ only)
nhairs Apr 27, 2024
4b19678
Drop py37 support, begin testing again py313
nhairs Apr 27, 2024
0d8b8a1
Fix py313 in GHA
nhairs Apr 27, 2024
3897a97
Don't instlal python on pypy
nhairs Apr 27, 2024
2eb8820
Run py313 in py313...
nhairs Apr 27, 2024
bc2b96e
Avoid orjson on python 3.13 while its not supported
nhairs Apr 27, 2024
1441721
Migrate tests to pytest, test OrjsonFormatter where possible
nhairs Apr 28, 2024
c017bef
Update docstrings to use mkdocstrings compatible
nhairs Apr 28, 2024
457a74c
Fix formatting, linting, typing
nhairs Apr 28, 2024
0d2bc65
Remove py37 from tox config
nhairs Apr 28, 2024
0ee6b9a
Maintain backwards compatibility
nhairs Apr 29, 2024
a0b595b
Add more tests
nhairs Apr 30, 2024
77dcdae
Add support for deprecated json.RESERVED_ATTRS
nhairs Apr 30, 2024
b61fc98
fix assert in test
nhairs May 1, 2024
89f0820
Update test names
nhairs May 3, 2024
a2df91b
Add support for msgspec
nhairs May 3, 2024
44c610d
Fix msgspec specifiers
nhairs May 3, 2024
32f763f
simplify ORJSON_AVAILABLE check
nhairs May 3, 2024
487388c
Update README and CHANGELOG
nhairs May 3, 2024
5cc6e3d
Fix formatting
nhairs May 3, 2024
ce76b66
Add other encoders to README
nhairs May 5, 2024
44406d3
Update freezegun issue references
nhairs May 5, 2024
5a6b959
Remove optional dependencies for specific encoders
nhairs May 5, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .github/workflows/test-suite.yml
Original file line number Diff line number Diff line change
Expand Up @@ -33,18 +33,18 @@ jobs:
needs: [lint]
runs-on: "${{ matrix.os }}"
strategy:
fail-fast: false # allow tests to run on all platforms
matrix:
python-version:
- "pypy-3.7"
- "pypy-3.8"
- "pypy-3.9"
- "pypy-3.10"
- "3.7"
- "3.8"
- "3.9"
- "3.10"
- "3.11"
- "3.12"
- "3.13-dev"
os:
- ubuntu-latest
- windows-latest
Expand Down
27 changes: 27 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,33 @@ All notable changes to this project will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## [3.1.0.rc1](https://github.com/nhairs/python-json-logger/compare/v3.0.1...v3.1.0.rc1) - 2023-05-03

This splits common funcitonality out to allow supporting other JSON encoders. Although this is a large refactor, backwards compatibility has been maintained.

### Added
- `.core` - more details below.
- Orjson encoder support via `.orjson.OrjsonFormatter`.
- MsgSpec encoder support via `.msgspec.MsgspecFormatter`.

### Changed
- `.jsonlogger` has been moved to `.json` with core functionality moved to `.core`.
- `.core.BaseJsonFormatter` properly supports all `logging.Formatter` arguments:
- `fmt` is unchanged.
- `datefmt` is unchanged.
- `style` can now support non-standard arguments by setting `validate` to `False`
- `validate` allows non-standard `style` arguments or prevents calling `validate` on standard `style` arguments.
- `default` is ignored.

### Deprecated
- `.jsonlogger` is now `.json`
- `.jsonlogger.RESERVED_ATTRS` is now `.core.RESERVED_ATTRS`.
- `.jsonlogger.merge_record_extra` is now `.core.merge_record_extra`.

### Removed
- Python 3.7 support dropped
- `.jsonlogger.JsonFormatter._str_to_fn` replaced with `.core.str_to_object`.

## [3.0.1](https://github.com/nhairs/python-json-logger/compare/v3.0.0...v3.0.1) - 2023-04-01

### Fixes
Expand Down
80 changes: 51 additions & 29 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ Until the PEP 541 request is complete you will need to install directly from git
To install from releases:

```shell
# 3.0.0 wheel
# e.g. 3.0.0 wheel
pip install 'python-json-logger@https://github.com/nhairs/python-json-logger/releases/download/v3.0.0/python_json_logger-3.0.0-py3-none-any.whl'
```

Expand All @@ -53,38 +53,30 @@ pip install -e .

## Usage

Python JSON Logger provides `logging.Formatter`s that encode the logged message into JSON. Although a variety of JSON encoders are supported, in the following examples we will use the `pythonjsonlogger.json.JsonFormatter` which uses the the `json` module from the standard library.

### Integrating with Python's logging framework

Json outputs are provided by the JsonFormatter logging formatter. You can add the custom formatter like below:
To produce JSON output, attach the formatter to a logging handler:

```python
import logging
from pythonjsonlogger import jsonlogger
from pythonjsonlogger.json import JsonFormatter

logger = logging.getLogger()

logHandler = logging.StreamHandler()
formatter = jsonlogger.JsonFormatter()
formatter = JsonFormatter()
logHandler.setFormatter(formatter)
logger.addHandler(logHandler)
```

### Customizing fields

The fmt parser can also be overidden if you want to have required fields that differ from the default of just `message`.
### Output fields

These two invocations are equivalent:
You can control the logged fields by setting the `fmt` argument when creating the formatter. By default formatters will follow the same `style` of `fmt` as the `logging` module: `%`, `$`, and `{`. All [`LogRecord` attributes](https://docs.python.org/3/library/logging.html#logrecord-attributes) can be output using their name.

```python
class CustomJsonFormatter(jsonlogger.JsonFormatter):
def parse(self):
return self._fmt.split(';')

formatter = CustomJsonFormatter('one;two')

# is equivalent to:

formatter = jsonlogger.JsonFormatter('%(one)s %(two)s')
formatter = JsonFormatter("{message}{asctime}{exc_info}", style="{")
```

You can also add extra fields to your json output by specifying a dict in place of message, as well as by specifying an `extra={}` argument.
Expand All @@ -94,9 +86,9 @@ Contents of these dictionaries will be added at the root level of the entry and
You can also use the `add_fields` method to add to or generally normalize the set of default set of fields, it is called for every log event. For example, to unify default fields with those provided by [structlog](http://www.structlog.org/) you could do something like this:

```python
class CustomJsonFormatter(jsonlogger.JsonFormatter):
class CustomJsonFormatter(JsonFormatter):
def add_fields(self, log_record, record, message_dict):
super(CustomJsonFormatter, self).add_fields(log_record, record, message_dict)
super().add_fields(log_record, record, message_dict)
if not log_record.get('timestamp'):
# this doesn't use record.created, so it is slightly off
now = datetime.utcnow().strftime('%Y-%m-%dT%H:%M:%S.%fZ')
Expand All @@ -105,32 +97,55 @@ class CustomJsonFormatter(jsonlogger.JsonFormatter):
log_record['level'] = log_record['level'].upper()
else:
log_record['level'] = record.levelname
return

formatter = CustomJsonFormatter('%(timestamp)s %(level)s %(name)s %(message)s')
```

Items added to the log record will be included in *every* log message, no matter what the format requires.

### Adding custom object serialization
You can also override the `process_log_record` method to modify fields before they are serialized to JSON.

```python
class SillyFormatter(JsonFormatter):
def process_log_record(log_record):
new_record = {k[::-1]: v for k, v in log_record.items()}
return new_record
```

#### Supporting custom styles

It is possible to support custom `style`s by setting `validate=False` and overriding the `parse` method.

For example:

```python
class CommaSupport(JsonFormatter):
def parse(self) -> list[str]:
if isinstance(self._style, str) and self._style == ",":
return self._fmt.split(",")
return super().parse()

formatter = CommaSupport("message,asctime", style=",", validate=False)
```

### Custom object serialization

Most formatters support `json_default` which is used to control how objects are serialized.

For custom handling of object serialization you can specify default json object translator or provide a custom encoder

```python
def json_translate(obj):
def my_default(obj):
if isinstance(obj, MyClass):
return {"special": obj.special}

formatter = jsonlogger.JsonFormatter(json_default=json_translate,
json_encoder=json.JSONEncoder)
logHandler.setFormatter(formatter)

logger.info({"special": "value", "run": 12})
logger.info("classic message", extra={"special": "value", "run": 12})
formatter = JsonFormatter(json_default=my_default)
```

### Using a Config File

To use the module with a config file using the [`fileConfig` function](https://docs.python.org/3/library/logging.config.html#logging.config.fileConfig), use the class `pythonjsonlogger.jsonlogger.JsonFormatter`. Here is a sample config file.
To use the module with a config file using the [`fileConfig` function](https://docs.python.org/3/library/logging.config.html#logging.config.fileConfig), use the class `pythonjsonlogger.json.JsonFormatter`. Here is a sample config file.

```ini
[loggers]
Expand Down Expand Up @@ -161,6 +176,13 @@ format = %(message)s
class = pythonjsonlogger.jsonlogger.JsonFormatter
```

### Alternate JSON Encoders

The following JSON encoders are also supported:

- [orjson](https://github.com/ijl/orjson) - `pythonjsonlogger.orjon.OrjsonFormatter`
- [msgspec](https://github.com/jcrist/msgspec) - `pythonjsonlogger.msgspec.MsgspecFormatter`

## Example Output

Sample JSON with a full formatter (basically the log message from the unit test). Every log message will appear on 1 line like a typical logger.
Expand All @@ -180,7 +202,7 @@ Sample JSON with a full formatter (basically the log message from the unit test)
"msecs": 506.24799728393555,
"pathname": "tests/tests.py",
"lineno": 60,
"asctime": ["12-05-05 22:11:08,506248"],
"asctime": "12-05-05 22:11:08,506248",
"message": "testing logging format",
"filename": "tests.py",
"levelname": "INFO",
Expand Down
2 changes: 1 addition & 1 deletion pylintrc
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
# A comma-separated list of package or module names from where C extensions may
# be loaded. Extensions are loading into the active Python interpreter and may
# run arbitrary code.
extension-pkg-whitelist=
extension-pkg-whitelist=orjson

# Add files or directories to the blacklist. They should be base names, not
# paths.
Expand Down
21 changes: 13 additions & 8 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -4,18 +4,21 @@ build-backend = "setuptools.build_meta"

[project]
name = "python-json-logger"
version = "3.0.1"
version = "3.1.0.rc1"
description = "JSON Log Formatter for the Python Logging Package"
authors = [
{name = "Zakaria Zajac", email = "zak@madzak.com"},
{name = "Nicholas Hairs", email = "info+python-json-logger@nicholashairs.com"},
]
maintainers = [
{name = "Nicholas Hairs", email = "info+python-json-logger@nicholashairs.com"},
]

# Dependency Information
requires-python = ">=3.7"
# dependencies = []
requires-python = ">=3.8"
dependencies = [
"typing_extensions",
Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do I need to specify when to install this?

]

# Extra information
readme = "README.md"
Expand All @@ -26,7 +29,6 @@ classifiers = [
"License :: OSI Approved :: BSD License",
"Operating System :: OS Independent",
"Programming Language :: Python :: 3 :: Only",
"Programming Language :: Python :: 3.7",
"Programming Language :: Python :: 3.8",
"Programming Language :: Python :: 3.9",
"Programming Language :: Python :: 3.10",
Expand All @@ -41,15 +43,18 @@ classifiers = [
GitHub = "https://github.com/nhairs/python-json-logger"

[project.optional-dependencies]
lint = [
dev = [
## Optional but required for dev
"orjson;implementation_name!='pypy' and python_version<'3.13'",
"msgspec;implementation_name!='pypy' and python_version<'3.13'",
## Lint
"validate-pyproject[all]",
"black",
"pylint",
"mypy",
]

test = [
## Test
"pytest",
"freezegun",
]

[tool.setuptools.packages.find]
Expand Down
40 changes: 40 additions & 0 deletions src/pythonjsonlogger/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
### IMPORTS
### ============================================================================
## Future

## Standard Library
import warnings

## Installed

## Application
import pythonjsonlogger.json

### CONSTANTS
### ============================================================================
try:
import orjson

ORJSON_AVAILABLE = True
except ImportError:
ORJSON_AVAILABLE = False


try:
import msgspec

MSGSPEC_AVAILABLE = True
except ImportError:
MSGSPEC_AVAILABLE = False


### DEPRECATED COMPATIBILITY
### ============================================================================
def __getattr__(name: str):
if name == "jsonlogger":
warnings.warn(
"pythonjsonlogger.jsonlogger has been moved to pythonjsonlogger.json",
DeprecationWarning,
)
return pythonjsonlogger.json
raise AttributeError(f"module {__name__} has no attribute {name}")
Loading
Loading