0.16.0

What's Changed

Dependency upgrade

• Add timeout to the test and ruff workflows by @kxtran in #348
• Bump openai from 1.52.1 to 1.54.4 by @wenzhe-log10 in #352
• upgrade anyio version to 4.4.0 by @wenzhe-log10 in #353
• upgrade cryptograph to 43.0.1 by @wenzhe-log10 in #354
• Bump anthropic from 0.37.1 to 0.39.0 by @wenzhe-log10 in #355
• upgrade langchain and langchain-community to 0.2 by @wenzhe-log10 in #357
• upgrade requests to 2.32.2 for snyk by @wenzhe-log10 in #358
• upgrade together 1.0.1 for snyk by @wenzhe-log10 in #359
• add workflow test option to run litellm test only by @wenzhe-log10 in #361
• upgrade litellm 1.52.8 for Snyk by @wenzhe-log10 in #360
• update minor deps for snyk by @wenzhe-log10 in #370
• validate url before post by @wenzhe-log10 in #372
• try requests.post for snyk SSRF by @wenzhe-log10 in #373
• update is_safe_url func with more checks by @wenzhe-log10 in #374
• bump version 0.16.0 by @wenzhe-log10 in #376

Full Changelog: 0.15.0...0.16.0

0.15.0

What's Changed

Feature

• add Completions.mock_chat_completions to insert logs into Log10 by @wenzhe-log10 in #328

Maintenance

• add default value for LOG10_URL if not set in env by @wenzhe-log10 in #332
• update test large image (10MB) url by @wenzhe-log10 in #333
• replace pydantic copy with model_copy. copy is deprecated by @wenzhe-log10 in #327
• replace deprecated models in the examples by @wenzhe-log10 in #338

Upgrade and test dependencies

• Bump anthropic from 0.34.1 to 0.37.1 by @wenzhe-log10 in #336
• Bump openai from 1.51.0 to 1.52.1 by @wenzhe-log10 in #337

Full Changelog: 0.14.1...0.15.0

0.14.1

What's Changed

Pytest Log10 Evaluations

• pytest log10 updates to include url to evaluation page by @wenzhe-log10 in #322

Example output and note the Log10 Evaluation URL: at the end:

========================================= test session starts ==========================================
platform darwin -- Python 3.12.2, pytest-8.3.3, pluggy-1.5.0
rootdir: test_log10_eval
plugins: metadata-3.1.1, anyio-4.6.0, log10-io-0.14.0
collected 1 item

test_example.py F                                                                                [100%]

=============================================== FAILURES ===============================================
_____________________________________________ test_example _____________________________________________

    def test_example():
>       assert 0 == 1
E       assert 0 == 1

test_example.py:4: AssertionError
========================================== Log10 Eval Report ===========================================
Log10 Eval is enabled.
Test run: tests-06aedd36-69af-45bb-918a-b1cab4e79298
Log10 Evaluation URL: log10.io/app/test-org/evaluations?id=06aedd36-69af-45bb-918a-b1cab4e79298
Report saved to: test_log10_eval/.pytest_log10_eval_reports/tests-06aedd36-69af-45bb-918a-b1cab4e79298.report.json
Report successfully uploaded to Log10
======================================= short test summary info ========================================
FAILED test_example.py::test_example - assert 0 == 1
========================================== 1 failed in 27.13s ==========================================

Full Changelog: 0.14.0...0.14.1

0.14.0

What's Changed

Pytest plugin Log10 Managed Evaluation

A pytest plugin for managing evaluation in Log10 platform.

Installation

After configuring the Log10 environment variables,

pip install log10-io

Usage

To enable the Log10 managed evaluation reporting, you need to use the --log10 option or set log10 = true in your pytest configuration file. Once enabled, execute your tests as you normally would:

pytest tests --log10

This will run your tests and upload the results to the Log10 platform for managed evaluation.

Please refer to the doc for detailed options, such as running it locally.

PRs

@wenzhe-log10 in #316 and #318

Maintenance

• handle openai legacy response by @wenzhe-log10 in #315

Full Changelog: 0.13.0...0.14.0

0.14.0-rc2

What's Changed

• Integrate Log10 plugin: Add opt-in reporting and restructure by @wenzhe-log10 in #318
• release - bump version 0.14.0-rc2 by @wenzhe-log10 in #320

Full Changelog: 0.14.0-rc1...0.14.0-rc2

0.14.0-rc1

What's Changed

• Add pytest plugin Log10 managed evaluation by @wenzhe-log10 in #316
• handle openai legacy response by @wenzhe-log10 in #315

Full Changelog: 0.13.0...0.14.0rc1

0.13.0

What's Changed

API change

Add log10_tags context manager and with_log10_tags decorator by @wenzhe-log10 in #299
This PR introduces:

1. log10_session Enhancement - Improved handling of tags in nested log10_session calls. Tags will now be appended instead of overwritten.

Example:

with log10_session(tags=["A"]):
  with log10_session(tags=["B"]):
    # New behavior: tags will be ["A", "B"]
    # Old behavior: tags were just ["B"]

Note: this only affects nested log10_session usage. Single log10_session calls are unchanged.

2. Introducing log10_tags context manager and with_log10_tags function decorator to add tags
   • A new context manager for adding tags to logs without creating a new session ID.
   • Can be used independently or with log10_session.
   • Tags will be appended in nested case.

Example link

Misc

• Bump minimum python version to 3.10 by @kxtran in #292
• Bump openai from 1.42.0 to 1.46.1 by @wenzhe-log10 in #302
• replace logging.warn with logger.warning, logger is defined and warn … by @wenzhe-log10 in #306
• bump openai to 1.51.0 by @wenzhe-log10 in #313

Full Changelog: 0.12.0...0.13.0

0.12.0

What's Changed

CLI

• add cli auto-prompt analyze - prompt analyzer in cli by @wenzhe-log10 in #288
  • Please check the CLI doc for detailed usage for input prompt/messages.
• cli benchmark_models - support OpenAI fine-tuned models by @wenzhe-log10 in #287

OpenAI Compatibility

• Generalize perplexity logic for streaming end detection across OpenAI compatible models by @kxtran in #286
  • Tested with Cerebras, Mistral, and Perplexity.

Maintenance

• update dependencies - OpenAI, Anthropic, and Magentic by @wenzhe-log10 in #289

Full Changelog: 0.11.0...0.12.0

0.11.0

What's Changed

New

• Add async streaming support for OpenAI compatible models (e.g Mistral and Perplexity) by @kxtran in #281

Full Changelog: 0.10.11...0.11.0

0.10.11

What's Changed

CLI

• fix - benchmark for responses does not have usage field by @wenzhe-log10 in #275

Full Changelog: 0.10.10...0.10.11