Added generic approach to strict type checking for constrained types #799

DerRidda · 2019-09-09T12:19:48Z

Change Summary

Use the arbitrary validator to build strict validators for ConstrainedInt, ConstrainedFloat, ConstrainedStr
Make StrictStr a derived class of ConstrainedStr
Add tests for new strict cases for ConstrainedInt and ConstrainedFloat

This PR adds configurable strictness validation to ConstrainedStr, ConstrainedInt and ConstrainedFloat.

This behaviour is desired for our use case as we generally never want coercion for anything,
especially those that loses information such as float -> int coercion.

Related issue number

This issue seems related but was not regarded when creating this PR: #780

Checklist

Unit tests for the changes exist
Tests pass on CI and coverage remains at 100%
Documentation reflects the changes where applicable
changes/<pull request or issue id>-<github username>.rst file added describing change
(see changes/README.md for details)

- Use the arbitrary validator to build strict validators for ConstrainedInt, ConstrainedFloat, ConstrainedStr - Make StrictStr a derived class of ConstrainedStr - Add tests for new strict cases for ConstrainedInt and ConstrainedFloat

codecov · 2019-09-09T12:25:40Z

Codecov Report

Merging #799 into master will not change coverage.
The diff coverage is 100%.

@@          Coverage Diff          @@
##           master   #799   +/-   ##
=====================================
  Coverage     100%   100%           
=====================================
  Files          17     17           
  Lines        2972   2984   +12     
  Branches      578    580    +2     
=====================================
+ Hits         2972   2984   +12

Impacted Files	Coverage Δ
pydantic/types.py	`100% <100%> (ø)`	⬆️
pydantic/validators.py	`100% <100%> (ø)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0c57937...359b468. Read the comment docs.

samuelcolvin · 2019-09-09T12:26:53Z

I think this looks good.

Please can you

also export StrictFloat and StrictInt (same way you do StrictStr)
add something to the docs about this, perhaps by extending and changing the StrictBool section
add a change description.

- Changed strictness tests to use these new subclasses

DerRidda · 2019-09-09T13:31:28Z

I have added all requested changes.

Docs have their own section now.

docs/index.rst

Co-Authored-By: Zaar Hai <haizaar@users.noreply.github.com>

samuelcolvin · 2019-09-11T09:21:47Z

docs/index.rst

@@ -785,6 +785,14 @@ The SecretStr and SecretBytes will be formatted as either `'**********'` or `''`

 (This script is complete, it should run "as is")

+Strict Types


can you move the "StrictBool" section into here.

You still need to remove the "StrictBool" section: https://pydantic-docs.helpmanual.io/#strictbool

docs/index.rst

samuelcolvin · 2019-09-11T09:24:06Z

docs/index.rst

+These types will only pass validation when the validated value is of the respective type or is a subtype of that type.
+This behavior is also exposed via the ``strict`` field of the ``ConstrainedStr``, ``ConstrainedFloat`` and
+``ConstrainedInt`` classes and can be combined with a multitude of complex validation rules (please note that there is no ``ConstrainedBool`` class).
+


Would be good to add a code example demonstrating this behaviour.

pydantic/types.py

tests/test_types.py

docs/index.rst

- Make ConstrainedInt and ConstrainedFloat use those validators instead of abusing arbitrary type validator for strictness - Prevent double validaton of same conditions by only yielding either the strict or non-strict type validator for for those classes

…e in invalid cases

- Added example for strict type usage - Added note about caveats for StrictInt and StrictFloat

pydantic/validators.py

samuelcolvin · 2019-09-12T09:52:21Z

docs/index.rst

@@ -785,6 +785,14 @@ The SecretStr and SecretBytes will be formatted as either `'**********'` or `''`

 (This script is complete, it should run "as is")

+Strict Types


You still need to remove the "StrictBool" section: https://pydantic-docs.helpmanual.io/#strictbool

pydantic/validators.py

faster method to check if a value is boolean for strict int validator Co-Authored-By: Samuel Colvin <samcolvin@gmail.com>

- Moved the Strctbool code example into strict_types.py example file

…other constrained types

samuelcolvin · 2019-09-16T12:05:31Z

docs/index.rst

@@ -617,12 +617,9 @@ A standard ``bool`` field will raise a ``ValidationError`` if the value is not o
  ``'off', 'f', 'false', 'n', 'no', '1', 'on', 't', 'true', 'y', 'yes'``
 * a ``bytes`` which is valid (per the previous rule) when decoded to ``str``

-For stricter behavior, ``StrictBool`` can be used to require specifically ``True`` or ``False``;


Can we keep this line and add a link to the "Strict Types" section.

Sorry to mess you around.

pydantic/validators.py

samuelcolvin · 2019-09-16T12:08:45Z

pydantic/types.py


    @classmethod
    def __get_validators__(cls) -> 'CallableGenerator':
        yield not_none_validator
-        yield str_validator
+        yield make_arbitrary_type_validator(str) if cls.strict else str_validator


does this work if you pass it an enum that inherits from str? Perhaps it should, but it will likely confuse many people since the __str__ method is different from normal I think.

We should at least add a note to the docs.

Naturally this should pass with all subclasses as we only make the exception for integers with boolean as that is a weird quirk of Python where Guido probably looked at C too much.

I can add the note to the docs.

samuelcolvin · 2019-09-17T11:38:59Z

great, thank you so much.

DerRidda · 2019-09-17T11:43:44Z

Thank you for merging, I was actually still touching on a few things you had commented on but if it is fine like that I don't mind.

samuelcolvin · 2019-09-17T11:44:30Z

sorry about that, I'm crashing about trying to get to v1. Feel free to submit another PR if you have any tweaks.

skewty · 2019-11-18T01:51:29Z

I deleted my previous posts because the solution I came up with in the end seems trivial.

I guess responses were created due to emails generated.. So here is a response to a question that disappeared :)

@dmontagu

Why can't you do IdentStr = typing.NewType('IdentStr', ConstrainedStr) or similar?

import pydantic, typing

IdentStr = typing.NewType('IdentStr', pydantic.ConstrainedStr)
IdentStr.min_length

Traceback (most recent call last):
  File "<input>", line 1, in <module>
AttributeError: 'function' object has no attribute 'min_length'

typing.NewType only fakes making a real class.

Since in real life, we control the code for IdentStr, I am playing around with:

if PYDANTIC_VERSION >= "1.0.0":
    from pydantic import StrictStr
else:  # min version is 0.32.2 in setup.py
    from pydantic import ConstrainedStr
    from pydantic.errors import StrError

    class StrictStr(ConstrainedStr):
        @classmethod
        def strict_str_validator(cls, v: str) -> str:
            if not isinstance(v, str):
                raise StrError()
            return v

        @classmethod
        def __get_validators__(cls):
            yield cls.strict_str_validator
            yield from super().__get_validators__()

class IdentStr(StrictStr):
    min_length = 1

skewty · 2019-12-01T18:51:57Z

I am sorta lost as to why this isn't working as expected for me in v1.2:

from typing import Type, TypeVar
import pydantic


class ConnectionName(pydantic.StrictStr):
    min_length = 1


class EndpointStr(pydantic.StrictStr):
    min_length = 1


DEFAULT_ENDPOINT = EndpointStr("default")
T = TypeVar("T", bound="UpRequestEvent")


class UpRequestEvent(pydantic.BaseModel):
    connection_name: ConnectionName  # connection name
    connection_endpoint: EndpointStr  # connection name's endpoint (for server type connections)

    @classmethod
    def build_from(
        cls: Type[T],
        connection_name: ConnectionName,
        connection_endpoint: EndpointStr = DEFAULT_ENDPOINT,
        **kwargs
    ) -> T:
        return cls(
            connection_name=connection_name,
            connection_endpoint=connection_endpoint,
        )

    class Config:
        use_enum_values = True


# This works
UpRequestEvent.build_from(
    connection_name="test", connection_endpoint="default",
)

# This fails
UpRequestEvent.build_from(
    connection_name=ConnectionName("test"), connection_endpoint=EndpointStr("default"),
)

Traceback (most recent call last):
  File "/opt/pycharm-eap/plugins/python/helpers/pydev/pydevd.py", line 1415, in _exec
    pydev_imports.execfile(file, globals, locals)  # execute the script
  File "/opt/pycharm-eap/plugins/python/helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "/home/spalmer1009/.PyCharm2019.3/config/scratches/scratch.py", line 44, in <module>
    connection_name=ConnectionName("test"), connection_endpoint=EndpointStr("default"),
  File "/home/spalmer1009/.PyCharm2019.3/config/scratches/scratch.py", line 30, in build_from
    connection_endpoint=connection_endpoint,
  File "pydantic/main.py", line 274, in pydantic.main.BaseModel.__init__
pydantic.error_wrappers.ValidationError: 2 validation errors for UpRequestEvent
connection_name
  Expected unicode, got ConnectionName (type=type_error)
connection_endpoint
  Expected unicode, got EndpointStr (type=type_error)

My special types ConnectionName and EndpointStr are a subclass of str (through StrictStr -> ConstrainedStr)

Am I not able to subclass these two types?

dmontagu · 2019-12-01T22:15:14Z

This appears to be an issue with the compiled vs. uncompiled versions of the package, as it works for me when not compiled (and fails when compiled). I'm trying to figure out if there is a workaround/solution now.

dmontagu · 2019-12-01T22:33:07Z

@skewty I think I found the issue.

If you change the signature of ConstrainedStr.validate to

def validate(cls, value: Union[str]) -> Union[str]: ...

and of strict_str_validator to:

def strict_str_validator(v: Any) -> Union[str]:

your code works.

I think what is happening is something weird with how cython handles str; using Union[str] prevents cython from applying some compilation magic.

@skewty It would be great if you could submit a new (bug) issue about this. It would also be great if you could make a PR with these changes and adding an appropriate test, but we can get to it eventually if you create an issue.

dmontagu · 2019-12-01T22:39:31Z

Strangely, looking at the differences in the generated cython code, you can see cython is actually injecting this error:

+060:         return v

    __Pyx_XDECREF(__pyx_r);
    if (!(likely(PyString_CheckExact(__pyx_v_v))||((__pyx_v_v) == Py_None)||(PyErr_Format(PyExc_TypeError, "Expected %.16s, got %.200s", "str", Py_TYPE(__pyx_v_v)->tp_name), 0))) __PYX_ERR(0, 60, __pyx_L1_error)
    __Pyx_INCREF(__pyx_v_v);
    __pyx_r = ((PyObject*)__pyx_v_v);

(In the -> Union[str] version no type error is injected like this.)

It also makes another minor change in the code earlier on, but I can't tell if that is affecting anything meaningful at first glance (looks like it is just looking up the Union type).

Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

DerRidda mentioned this pull request Sep 9, 2019

Generic method to create Strict<type> fields #780

Closed

DerRidda added 3 commits September 9, 2019 15:05

- Created StrictInt and StrictFloat subclasses and exported them

8ae5ec9

- Changed strictness tests to use these new subclasses

- Added section for Strict Types to docs

fcb9a19

- Added changes file

ff89243

DerRidda changed the title ~~Added generic approach to strict type checking for constraint types~~ Added generic approach to strict type checking for constrained types Sep 9, 2019

haizaar reviewed Sep 11, 2019

View reviewed changes

docs/index.rst Show resolved Hide resolved

haizaar reviewed Sep 11, 2019

View reviewed changes

docs/index.rst Outdated Show resolved Hide resolved

haizaar reviewed Sep 11, 2019

View reviewed changes

docs/index.rst Show resolved Hide resolved

DerRidda and others added 2 commits September 11, 2019 10:08

Update docs/index.rst

be04dcb

Co-Authored-By: Zaar Hai <haizaar@users.noreply.github.com>

Update docs/index.rst

b5da961

Co-Authored-By: Zaar Hai <haizaar@users.noreply.github.com>

samuelcolvin reviewed Sep 11, 2019

View reviewed changes

haizaar reviewed Sep 11, 2019

View reviewed changes

docs/index.rst Outdated Show resolved Hide resolved

DerRidda added 3 commits September 11, 2019 13:52

Changed strict int and float tests to alos match for exception messag…

5c9481b

…e in invalid cases

- Removed obvious note about lack of ConstrainedBool

858c5d7

- Added example for strict type usage - Added note about caveats for StrictInt and StrictFloat

samuelcolvin reviewed Sep 12, 2019

View reviewed changes

DerRidda and others added 4 commits September 16, 2019 09:55

Update pydantic/validators.py

1e88a10

faster method to check if a value is boolean for strict int validator Co-Authored-By: Samuel Colvin <samcolvin@gmail.com>

- Removed StrictBool part from Boolean section of docs

29a53b0

- Moved the Strctbool code example into strict_types.py example file

- Changed behavior of strictness for COnstrainedStr to match that fo …

c2451b3

…other constrained types

- Actually make ConstrainedStr use the correct validator for strictness

408759d

samuelcolvin reviewed Sep 16, 2019

View reviewed changes

Merge branch 'master' into strict-constrained-types

359b468

samuelcolvin merged commit 1b467da into pydantic:master Sep 17, 2019

DerRidda deleted the strict-constrained-types branch November 18, 2019 09:09

dmontagu mentioned this pull request Dec 1, 2019

Question: Why is PyString_CheckExact used (preventing str subclass returns)? cython/cython#3256

Closed

skewty mentioned this pull request Dec 1, 2019

Subclassing StrictStr / ConstrainedStr causes ValidationError #1060

Closed

samuelcolvin mentioned this pull request Dec 3, 2019

Trick Cython into allowing str subclassing #1061

Merged

4 tasks

samuelcolvin mentioned this pull request Jan 6, 2020

orjson encoding issue #1150

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Added generic approach to strict type checking for constrained types #799

Added generic approach to strict type checking for constrained types #799

DerRidda commented Sep 9, 2019 •

edited

Loading

codecov bot commented Sep 9, 2019 •

edited

Loading

samuelcolvin commented Sep 9, 2019

DerRidda commented Sep 9, 2019

samuelcolvin Sep 11, 2019

samuelcolvin Sep 12, 2019

DerRidda Sep 16, 2019

samuelcolvin Sep 11, 2019

DerRidda Sep 16, 2019

samuelcolvin Sep 12, 2019

samuelcolvin Sep 16, 2019

samuelcolvin Sep 16, 2019

DerRidda Sep 16, 2019 •

edited

Loading

samuelcolvin commented Sep 17, 2019

DerRidda commented Sep 17, 2019

samuelcolvin commented Sep 17, 2019

skewty commented Nov 18, 2019

skewty commented Dec 1, 2019

dmontagu commented Dec 1, 2019 •

edited

Loading

dmontagu commented Dec 1, 2019

dmontagu commented Dec 1, 2019 •

edited

Loading

		@@ -785,6 +785,14 @@ The SecretStr and SecretBytes will be formatted as either `'**********'` or `''`

		(This script is complete, it should run "as is")

		Strict Types

Added generic approach to strict type checking for constrained types #799

Added generic approach to strict type checking for constrained types #799

Conversation

DerRidda commented Sep 9, 2019 • edited Loading

Related issue number

Checklist

codecov bot commented Sep 9, 2019 • edited Loading

Codecov Report

samuelcolvin commented Sep 9, 2019

DerRidda commented Sep 9, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

DerRidda Sep 16, 2019 • edited Loading

Choose a reason for hiding this comment

samuelcolvin commented Sep 17, 2019

DerRidda commented Sep 17, 2019

samuelcolvin commented Sep 17, 2019

skewty commented Nov 18, 2019

skewty commented Dec 1, 2019

dmontagu commented Dec 1, 2019 • edited Loading

dmontagu commented Dec 1, 2019

dmontagu commented Dec 1, 2019 • edited Loading

DerRidda commented Sep 9, 2019 •

edited

Loading

codecov bot commented Sep 9, 2019 •

edited

Loading

DerRidda Sep 16, 2019 •

edited

Loading

dmontagu commented Dec 1, 2019 •

edited

Loading

dmontagu commented Dec 1, 2019 •

edited

Loading