Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Custom Codec doesn't work with DOMAIN types #457

Closed
kjmph opened this issue Jun 13, 2019 · 7 comments · Fixed by #663
Closed

Custom Codec doesn't work with DOMAIN types #457

kjmph opened this issue Jun 13, 2019 · 7 comments · Fixed by #663

Comments

@kjmph
Copy link

kjmph commented Jun 13, 2019

  • asyncpg version: 0.18.3
  • PostgreSQL version: 11.3
  • Do you use a PostgreSQL SaaS? If so, which? Can you reproduce
    the issue with a local PostgreSQL install?
    : Local
  • Python version: 3.7.3
  • Platform: MacOS
  • Do you use pgbouncer?: No
  • Did you install asyncpg with pip?: Yes
  • If you built asyncpg locally, which version of Cython did you use?: PyPI
  • Can the issue be reproduced under both asyncio and
    uvloop?
    : Don't Know

When creating a DOMAIN, and using a custom codec, it calls the native type's codec, as opposed to the custom code. Here is the test code that is supposed to test for this:

https://github.com/MagicStack/asyncpg/blob/master/tests/test_codecs.py#L1224-L1239

However, the lambda functions aren't called. It appears the data type is recognized as an int.

@elprans
Copy link
Member

elprans commented Jul 8, 2019

Postgres seems to always report the base type OID in the RowDescription message. Interestingly, ParameterDescription contains the correct domain OID, so we can set up a custom encoder, but not a custom decoder. Given this, I think we should declare custom codecs on domains to be unsupported altogether.

@kjmph
Copy link
Author

kjmph commented Jul 8, 2019

That’s unfortunate. I’m sure you’ve gotten this before, however, with some rules on the decoder function couldn’t we support composite types?

If we removed this check:

if not introspection.is_scalar_type(typeinfo):
raise ValueError(
'cannot use custom codec on non-scalar type {}.{}'.format(
schema, typename))

And required the decoder function to do something like x.strip('()').split(','), as well as making the decoder handle Python type conversion, couldn’t asyncpg support composites?

@elprans
Copy link
Member

elprans commented Oct 3, 2019

Text I/O for composites is gnarly and wouldn't work very well, since asyncpg would refuse to do text I/O for the component types. Theoretically, this might be possible using the 'tuple' format protocol, but there's quite a few changes to be made to make this work, and I don't see a compelling use case for it.

@kjmph
Copy link
Author

kjmph commented Oct 29, 2019

I hear your point around the text I/O processing. The use case was around having a composite type so that one can input the result into a dataclass. row_to_json works for now, as the json type can be registered and decoded to a dictionary. However, it is beneficial to have errors if what is returned doesn't fit into a specific structure. I can close this if you don't think there is value here.

@elprans
Copy link
Member

elprans commented Oct 29, 2019

This issue is originally about domains, and that still needs to be resolved properly, so let's keep this open.

Regarding custom codecs for composite types -- feel free to open a separate issue (with the description of your use case). It's not impossible to implement and there's some value in that, so maybe someone will chime in as well.

@kjmph
Copy link
Author

kjmph commented Oct 29, 2019

Right! Sorry, my slow responses made me lose sight of the originating issue. I'll create a new ticket.

elprans added a commit that referenced this issue Nov 26, 2020
Postgres always includes the base type OID in the RowDescription message
even if the query is technically returning domain values.  This makes
custom codecs on domains ineffective, and so prohibit them to avoid
confusion and bug reports.

See postgres/postgres@d9b679c and
https://postgr.es/m/27307.1047485980%40sss.pgh.pa.us for context.

Fixes: #457.
elprans added a commit that referenced this issue Dec 2, 2020
Postgres always includes the base type OID in the RowDescription message
even if the query is technically returning domain values.  This makes
custom codecs on domains ineffective, and so prohibit them to avoid
confusion and bug reports.

See postgres/postgres@d9b679c and
https://postgr.es/m/27307.1047485980%40sss.pgh.pa.us for context.

Fixes: #457.
elprans added a commit that referenced this issue Dec 21, 2020
A new asyncpg release is here, just in time for Christmas.  Notable
additions include Python 3.9 support, support for recently added
PostgreSQL types like `jsonpath`, and last but not least, vastly
improved `executemany()` performance.  Importantly, `executemany()` is
also now _atomic_, which means that either all iterations succeed, or
none at all, whereas previously partial results would have remained in
place, unless `executemany()` was called in a transaction.

There is also the usual assortment of improvements and bugfixes, see the
details below.

This is the last release of asyncpg that supports Python 3.5, which is
has reached EOL in September.

Improvements
------------

* Vastly speedup executemany by batching protocol messages (#295)
  (by @fantix in 690048d for #295)

* Allow using custom Record class
  (by @elprans in db4f1a6 for #577)

* Add Python 3.9 support (#610)
  (by @elprans in c05d726 for #610)

* Prefer SSL connections by default (#660)
  (by @elprans in 16183aa for #660)

* Add codecs for a bunch of new builtin types (#665)
  (by @elprans in b53f038 for #665)

* Expose Pool as asyncpg.Pool (#669)
  (by @rugleb in 0e0eb8d for #669)

Fixes
-----

* Add a workaround for bpo-37658
  (by @elprans in 2bac166 for #21894)

* Fix wrong default transaction isolation level (#622)
  (by @fantix in 4a627d5 for #622)

* Fix set_type_codec() to accept standard SQL type names (#619)
  (by @elprans in 68b40cb for #619)

* Ignore custom data codec for internal introspection (#618)
  (by @fantix in e064f59 for #618)

* Fix null/NULL quoting in array text encoder (#627)
  (by @fantix in 92aa806 for #627)

* Fix link in connect docstring (#653)
  (by @samuelcolvin in 8b313bd for #653)

* Make asyncpg work with pyinstaller (#651)
  (by @Atem18 in 5ddabb1 for #651)

* Fix possible AttributeError exception in `ConnectionSettings` (#632)
  (by @petriborg in 0d23182 for #632)

* Prohibit custom codecs on domains
  (by @elprans in 50f964f for #457)

* Raise proper error on anonymous composite input (tuple arguments) (#664)
  (by @elprans in 7252dbe for #664)

* Fix incorrect application of custom codecs in some cases (#662)
  (by @elprans in 50f65fb for #662)
elprans added a commit that referenced this issue Dec 21, 2020
A new asyncpg release is here, just in time for Christmas.  Notable
additions include Python 3.9 support, support for recently added
PostgreSQL types like `jsonpath`, and last but not least, vastly
improved `executemany()` performance.  Importantly, `executemany()` is
also now _atomic_, which means that either all iterations succeed, or
none at all, whereas previously partial results would have remained in
place, unless `executemany()` was called in a transaction.

There is also the usual assortment of improvements and bugfixes, see the
details below.

This is the last release of asyncpg that supports Python 3.5, which is
has reached EOL in September.

Improvements
------------

* Vastly speedup executemany by batching protocol messages (#295)
  (by @fantix in 690048d for #295)

* Allow using custom `Record` class
  (by @elprans in db4f1a6 for #577)

* Add Python 3.9 support (#610)
  (by @elprans in c05d726 for #610)

* Prefer SSL connections by default (#660)
  (by @elprans in 16183aa for #660)

* Add codecs for a bunch of new builtin types (#665)
  (by @elprans in b53f038 for #665)

* Expose Pool as `asyncpg.Pool` (#669)
  (by @rugleb in 0e0eb8d for #669)

Fixes
-----

* Add a workaround for bpo-37658
  (by @elprans in 2bac166 for #21894)

* Fix wrong default transaction isolation level (#622)
  (by @fantix in 4a627d5 for #622)

* Fix `set_type_codec()` to accept standard SQL type names (#619)
  (by @elprans in 68b40cb for #619)

* Ignore custom data codec for internal introspection (#618)
  (by @fantix in e064f59 for #618)

* Fix null/NULL quoting in array text encoder (#627)
  (by @fantix in 92aa806 for #627)

* Fix link in connect docstring (#653)
  (by @samuelcolvin in 8b313bd for #653)

* Make asyncpg work with pyinstaller (#651)
  (by @Atem18 in 5ddabb1 for #651)

* Fix possible `AttributeError` exception in `ConnectionSettings` (#632)
  (by @petriborg in 0d23182 for #632)

* Prohibit custom codecs on domains
  (by @elprans in 50f964f for #457)

* Raise proper error on anonymous composite input (tuple arguments) (#664)
  (by @elprans in 7252dbe for #664)

* Fix incorrect application of custom codecs in some cases (#662)
  (by @elprans in 50f65fb for #662)
elprans added a commit that referenced this issue Dec 21, 2020
A new asyncpg release is here, just in time for Christmas.  Notable
additions include Python 3.9 support, support for recently added
PostgreSQL types like `jsonpath`, and last but not least, vastly
improved `executemany()` performance.  Importantly, `executemany()` is
also now _atomic_, which means that either all iterations succeed, or
none at all, whereas previously partial results would have remained in
place, unless `executemany()` was called in a transaction.

There is also the usual assortment of improvements and bugfixes, see the
details below.

This is the last release of asyncpg that supports Python 3.5, which has
reached EOL in September.

Improvements
------------

* Vastly speedup executemany by batching protocol messages (#295)
  (by @fantix in 690048d for #295)

* Allow using custom `Record` class
  (by @elprans in db4f1a6 for #577)

* Add Python 3.9 support (#610)
  (by @elprans in c05d726 for #610)

* Prefer SSL connections by default (#660)
  (by @elprans in 16183aa for #660)

* Add codecs for a bunch of new builtin types (#665)
  (by @elprans in b53f038 for #665)

* Expose Pool as `asyncpg.Pool` (#669)
  (by @rugleb in 0e0eb8d for #669)

Fixes
-----

* Add a workaround for bpo-37658
  (by @elprans in 2bac166 for #21894)

* Fix wrong default transaction isolation level (#622)
  (by @fantix in 4a627d5 for #622)

* Fix `set_type_codec()` to accept standard SQL type names (#619)
  (by @elprans in 68b40cb for #619)

* Ignore custom data codec for internal introspection (#618)
  (by @fantix in e064f59 for #618)

* Fix null/NULL quoting in array text encoder (#627)
  (by @fantix in 92aa806 for #627)

* Fix link in connect docstring (#653)
  (by @samuelcolvin in 8b313bd for #653)

* Make asyncpg work with pyinstaller (#651)
  (by @Atem18 in 5ddabb1 for #651)

* Fix possible `AttributeError` exception in `ConnectionSettings` (#632)
  (by @petriborg in 0d23182 for #632)

* Prohibit custom codecs on domains
  (by @elprans in 50f964f for #457)

* Raise proper error on anonymous composite input (tuple arguments) (#664)
  (by @elprans in 7252dbe for #664)

* Fix incorrect application of custom codecs in some cases (#662)
  (by @elprans in 50f65fb for #662)
elprans added a commit that referenced this issue Dec 21, 2020
A new asyncpg release is here, just in time for Christmas.  Notable
additions include Python 3.9 support, support for recently added
PostgreSQL types like `jsonpath`, and last but not least, vastly
improved `executemany()` performance.  Importantly, `executemany()` is
also now _atomic_, which means that either all iterations succeed, or
none at all, whereas previously partial results would have remained in
place, unless `executemany()` was called in a transaction.

There is also the usual assortment of improvements and bugfixes, see the
details below.

This is the last release of asyncpg that supports Python 3.5, which has
reached EOL in September.

Improvements
------------

* Vastly speedup executemany by batching protocol messages (#295)
  (by @fantix in 690048d for #295)

* Allow using custom `Record` class
  (by @elprans in db4f1a6 for #577)

* Add Python 3.9 support (#610)
  (by @elprans in c05d726 for #610)

* Prefer SSL connections by default (#660)
  (by @elprans in 16183aa for #660)

* Add codecs for a bunch of new builtin types (#665)
  (by @elprans in b53f038 for #665)

* Expose Pool as `asyncpg.Pool` (#669)
  (by @rugleb in 0e0eb8d for #669)

Fixes
-----

* Add a workaround for bpo-37658
  (by @elprans in 2bac166 for #21894)

* Fix wrong default transaction isolation level (#622)
  (by @fantix in 4a627d5 for #622)

* Fix `set_type_codec()` to accept standard SQL type names (#619)
  (by @elprans in 68b40cb for #619)

* Ignore custom data codec for internal introspection (#618)
  (by @fantix in e064f59 for #618)

* Fix null/NULL quoting in array text encoder (#627)
  (by @fantix in 92aa806 for #627)

* Fix link in connect docstring (#653)
  (by @samuelcolvin in 8b313bd for #653)

* Make asyncpg work with pyinstaller (#651)
  (by @Atem18 in 5ddabb1 for #651)

* Fix possible `AttributeError` exception in `ConnectionSettings` (#632)
  (by @petriborg in 0d23182 for #632)

* Prohibit custom codecs on domains
  (by @elprans in 50f964f for #457)

* Raise proper error on anonymous composite input (tuple arguments) (#664)
  (by @elprans in 7252dbe for #664)

* Fix incorrect application of custom codecs in some cases (#662)
  (by @elprans in 50f65fb for #662)
elprans added a commit that referenced this issue Feb 10, 2021
A new asyncpg release is here.

Notable additions include Python 3.9 support, support for recently added
PostgreSQL types like `jsonpath`, and last but not least, vastly
improved `executemany()` performance.  Importantly, `executemany()` is
also now _atomic_, which means that either all iterations succeed, or
none at all, whereas previously partial results would have remained in
place, unless `executemany()` was called in a transaction.

There is also the usual assortment of improvements and bugfixes, see the
details below.

This is the last release of asyncpg that supports Python 3.5, which has
reached EOL last September.

Improvements
------------

* Vastly speedup executemany by batching protocol messages (#295)
  (by @fantix in 690048d for #295)

* Allow using custom `Record` class
  (by @elprans in db4f1a6 for #577)

* Add Python 3.9 support (#610)
  (by @elprans in c05d726 for #610)

* Prefer SSL connections by default (#660)
  (by @elprans in 16183aa for #660)

* Add codecs for a bunch of new builtin types (#665)
  (by @elprans in b53f038 for #665)

* Expose Pool as `asyncpg.Pool` (#669)
  (by @rugleb in 0e0eb8d for #669)

* Avoid unnecessary overhead during connection reset (#648)
  (by @kitogo in ff5da5f for #648)

Fixes
-----

* Add a workaround for bpo-37658
  (by @elprans in 2bac166 for #21894)

* Fix wrong default transaction isolation level (#622)
  (by @fantix in 4a627d5 for #622)

* Fix `set_type_codec()` to accept standard SQL type names (#619)
  (by @elprans in 68b40cb for #619)

* Ignore custom data codec for internal introspection (#618)
  (by @fantix in e064f59 for #618)

* Fix null/NULL quoting in array text encoder (#627)
  (by @fantix in 92aa806 for #627)

* Fix link in connect docstring (#653)
  (by @samuelcolvin in 8b313bd for #653)

* Make asyncpg work with pyinstaller (#651)
  (by @Atem18 in 5ddabb1 for #651)

* Fix possible `AttributeError` exception in `ConnectionSettings` (#632)
  (by @petriborg in 0d23182 for #632)

* Prohibit custom codecs on domains
  (by @elprans in 50f964f for #457)

* Raise proper error on anonymous composite input (tuple arguments) (#664)
  (by @elprans in 7252dbe for #664)

* Fix incorrect application of custom codecs in some cases (#662)
  (by @elprans in 50f65fb for #662)
dmig pushed a commit to dmig/asyncpg that referenced this issue Feb 22, 2021
A new asyncpg release is here.

Notable additions include Python 3.9 support, support for recently added
PostgreSQL types like `jsonpath`, and last but not least, vastly
improved `executemany()` performance.  Importantly, `executemany()` is
also now _atomic_, which means that either all iterations succeed, or
none at all, whereas previously partial results would have remained in
place, unless `executemany()` was called in a transaction.

There is also the usual assortment of improvements and bugfixes, see the
details below.

This is the last release of asyncpg that supports Python 3.5, which has
reached EOL last September.

Improvements
------------

* Vastly speedup executemany by batching protocol messages (MagicStack#295)
  (by @fantix in 690048d for MagicStack#295)

* Allow using custom `Record` class
  (by @elprans in db4f1a6 for MagicStack#577)

* Add Python 3.9 support (MagicStack#610)
  (by @elprans in c05d726 for MagicStack#610)

* Prefer SSL connections by default (MagicStack#660)
  (by @elprans in 16183aa for MagicStack#660)

* Add codecs for a bunch of new builtin types (MagicStack#665)
  (by @elprans in b53f038 for MagicStack#665)

* Expose Pool as `asyncpg.Pool` (MagicStack#669)
  (by @rugleb in 0e0eb8d for MagicStack#669)

* Avoid unnecessary overhead during connection reset (MagicStack#648)
  (by @kitogo in ff5da5f for MagicStack#648)

Fixes
-----

* Add a workaround for bpo-37658
  (by @elprans in 2bac166 for #21894)

* Fix wrong default transaction isolation level (MagicStack#622)
  (by @fantix in 4a627d5 for MagicStack#622)

* Fix `set_type_codec()` to accept standard SQL type names (MagicStack#619)
  (by @elprans in 68b40cb for MagicStack#619)

* Ignore custom data codec for internal introspection (MagicStack#618)
  (by @fantix in e064f59 for MagicStack#618)

* Fix null/NULL quoting in array text encoder (MagicStack#627)
  (by @fantix in 92aa806 for MagicStack#627)

* Fix link in connect docstring (MagicStack#653)
  (by @samuelcolvin in 8b313bd for MagicStack#653)

* Make asyncpg work with pyinstaller (MagicStack#651)
  (by @Atem18 in 5ddabb1 for MagicStack#651)

* Fix possible `AttributeError` exception in `ConnectionSettings` (MagicStack#632)
  (by @petriborg in 0d23182 for MagicStack#632)

* Prohibit custom codecs on domains
  (by @elprans in 50f964f for MagicStack#457)

* Raise proper error on anonymous composite input (tuple arguments) (MagicStack#664)
  (by @elprans in 7252dbe for MagicStack#664)

* Fix incorrect application of custom codecs in some cases (MagicStack#662)
  (by @elprans in 50f65fb for MagicStack#662)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants