-
Notifications
You must be signed in to change notification settings - Fork 3.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
sql: FK options (deferrable, etc) #31632
Comments
Zendesk ticket #3474 has been linked to this issue. |
The spreadsheet we discussed is unwieldy - hard to edit and impossible to keep up to date. If we write down blacklists in code, then we can use an approach like this to always have an up to date aggregation. So far it seems like there's just a lot of unknowns to categorize still. The output today: ``` === RUN TestBlacklists 648: unknown (unknown) 493: cockroachdb#5807 (sql: Add support for TEMP tables) 151: cockroachdb#17511 (sql: support stored procedures) 86: cockroachdb#26097 (sql: make TIMETZ more pg-compatible) 56: cockroachdb#10735 (sql: support SQL savepoints) 55: cockroachdb#32552 (multi-dim arrays) 55: cockroachdb#26508 (sql: restricted DDL / DML inside transactions) 52: cockroachdb#32565 (sql: support optional TIME precision) 39: cockroachdb#243 (roadmap: Blob storage) 33: cockroachdb#26725 (sql: support postgres' API to handle blob storage (incl lo_creat, lo_from_bytea)) 31: cockroachdb#27793 (sql: support custom/user-defined base scalar (primitive) types) 24: cockroachdb#12123 (sql: Can't drop and replace a table within a transaction) 24: cockroachdb#26443 (sql: support user-defined schemas between database and table) 20: cockroachdb#21286 (sql: Add support for geometric types) 18: cockroachdb#6583 (sql: explicit lock syntax (SELECT FOR {SHARE,UPDATE} {skip locked,nowait})) 17: cockroachdb#22329 (Support XA distributed transactions in CockroachDB) 16: cockroachdb#24062 (sql: 32 bit SERIAL type) 16: cockroachdb#30352 (roadmap:when CockroachDB will support cursor?) 12: cockroachdb#27791 (sql: support RANGE types) 8: cockroachdb#40195 (pgwire: multiple active result sets (portals) not supported) 8: cockroachdb#6130 (sql: add support for key watches with notifications of changes) 5: Expected Failure (unknown) 5: cockroachdb#23468 (sql: support sql arrays of JSONB) 5: cockroachdb#40854 (sql: set application_name from connection string) 4: cockroachdb#35879 (sql: `default_transaction_read_only` should also accept 'on' and 'off') 4: cockroachdb#32610 (sql: can't insert self reference) 4: cockroachdb#40205 (sql: add non-trivial implementations of FOR UPDATE, FOR NO KEY UPDATE, FOR SHARE, FOR NO KEY SHARE) 4: cockroachdb#35897 (sql: unknown function: pg_terminate_backend()) 4: cockroachdb#4035 (sql/pgwire: missing support for row count limits in pgwire) 3: cockroachdb#27796 (sql: support user-defined DOMAIN types) 3: cockroachdb#3781 (sql: Add Data Type Formatting Functions) 3: cockroachdb#40476 (sql: support `FOR {UPDATE,SHARE} {SKIP LOCKED,NOWAIT}`) 3: cockroachdb#35882 (sql: support other character sets) 2: cockroachdb#10028 (sql: Support view queries with star expansions) 2: cockroachdb#35807 (sql: INTERVAL output doesn't match PG) 2: cockroachdb#35902 (sql: large object support) 2: cockroachdb#40474 (sql: support `SELECT ... FOR UPDATE OF` syntax) 1: cockroachdb#18846 (sql: Support CIDR column type) 1: cockroachdb#9682 (sql: implement computed indexes) 1: cockroachdb#31632 (sql: FK options (deferrable, etc)) 1: cockroachdb#24897 (sql: CREATE OR REPLACE VIEW) 1: pass? (unknown) 1: cockroachdb#36215 (sql: enable setting standard_conforming_strings to off) 1: cockroachdb#32562 (sql: support SET LOCAL and txn-scoped session variable changes) 1: cockroachdb#36116 (sql: psychopg: investigate how `'infinity'::timestamp` is presented) 1: cockroachdb#26732 (sql: support the binary operator: <int> / <float>) 1: cockroachdb#23299 (sql: support coercing string literals to arrays) 1: cockroachdb#36115 (sql: psychopg: investigate if datetimetz is being returned instead of datetime) 1: cockroachdb#26925 (sql: make the CockroachDB integer types more compatible with postgres) 1: cockroachdb#21085 (sql: WITH RECURSIVE (recursive common table expressions)) 1: cockroachdb#36179 (sql: implicity convert date to timestamp) 1: cockroachdb#36118 (sql: Cannot parse '24:00' as type time) 1: cockroachdb#31708 (sql: support current_time) ``` Release justification: non-production change Release note: None
The spreadsheet we discussed is unwieldy - hard to edit and impossible to keep up to date. If we write down blacklists in code, then we can use an approach like this to always have an up to date aggregation. So far it seems like there's just a lot of unknowns to categorize still. The output today: ``` === RUN TestBlacklists 648: unknown (unknown) 493: cockroachdb#5807 (sql: Add support for TEMP tables) 151: cockroachdb#17511 (sql: support stored procedures) 86: cockroachdb#26097 (sql: make TIMETZ more pg-compatible) 56: cockroachdb#10735 (sql: support SQL savepoints) 55: cockroachdb#32552 (multi-dim arrays) 55: cockroachdb#26508 (sql: restricted DDL / DML inside transactions) 52: cockroachdb#32565 (sql: support optional TIME precision) 39: cockroachdb#243 (roadmap: Blob storage) 33: cockroachdb#26725 (sql: support postgres' API to handle blob storage (incl lo_creat, lo_from_bytea)) 31: cockroachdb#27793 (sql: support custom/user-defined base scalar (primitive) types) 24: cockroachdb#12123 (sql: Can't drop and replace a table within a transaction) 24: cockroachdb#26443 (sql: support user-defined schemas between database and table) 20: cockroachdb#21286 (sql: Add support for geometric types) 18: cockroachdb#6583 (sql: explicit lock syntax (SELECT FOR {SHARE,UPDATE} {skip locked,nowait})) 17: cockroachdb#22329 (Support XA distributed transactions in CockroachDB) 16: cockroachdb#24062 (sql: 32 bit SERIAL type) 16: cockroachdb#30352 (roadmap:when CockroachDB will support cursor?) 12: cockroachdb#27791 (sql: support RANGE types) 8: cockroachdb#40195 (pgwire: multiple active result sets (portals) not supported) 8: cockroachdb#6130 (sql: add support for key watches with notifications of changes) 5: Expected Failure (unknown) 5: cockroachdb#23468 (sql: support sql arrays of JSONB) 5: cockroachdb#40854 (sql: set application_name from connection string) 4: cockroachdb#35879 (sql: `default_transaction_read_only` should also accept 'on' and 'off') 4: cockroachdb#32610 (sql: can't insert self reference) 4: cockroachdb#40205 (sql: add non-trivial implementations of FOR UPDATE, FOR NO KEY UPDATE, FOR SHARE, FOR NO KEY SHARE) 4: cockroachdb#35897 (sql: unknown function: pg_terminate_backend()) 4: cockroachdb#4035 (sql/pgwire: missing support for row count limits in pgwire) 3: cockroachdb#27796 (sql: support user-defined DOMAIN types) 3: cockroachdb#3781 (sql: Add Data Type Formatting Functions) 3: cockroachdb#40476 (sql: support `FOR {UPDATE,SHARE} {SKIP LOCKED,NOWAIT}`) 3: cockroachdb#35882 (sql: support other character sets) 2: cockroachdb#10028 (sql: Support view queries with star expansions) 2: cockroachdb#35807 (sql: INTERVAL output doesn't match PG) 2: cockroachdb#35902 (sql: large object support) 2: cockroachdb#40474 (sql: support `SELECT ... FOR UPDATE OF` syntax) 1: cockroachdb#18846 (sql: Support CIDR column type) 1: cockroachdb#9682 (sql: implement computed indexes) 1: cockroachdb#31632 (sql: FK options (deferrable, etc)) 1: cockroachdb#24897 (sql: CREATE OR REPLACE VIEW) 1: pass? (unknown) 1: cockroachdb#36215 (sql: enable setting standard_conforming_strings to off) 1: cockroachdb#32562 (sql: support SET LOCAL and txn-scoped session variable changes) 1: cockroachdb#36116 (sql: psychopg: investigate how `'infinity'::timestamp` is presented) 1: cockroachdb#26732 (sql: support the binary operator: <int> / <float>) 1: cockroachdb#23299 (sql: support coercing string literals to arrays) 1: cockroachdb#36115 (sql: psychopg: investigate if datetimetz is being returned instead of datetime) 1: cockroachdb#26925 (sql: make the CockroachDB integer types more compatible with postgres) 1: cockroachdb#21085 (sql: WITH RECURSIVE (recursive common table expressions)) 1: cockroachdb#36179 (sql: implicity convert date to timestamp) 1: cockroachdb#36118 (sql: Cannot parse '24:00' as type time) 1: cockroachdb#31708 (sql: support current_time) ``` Release justification: non-production change Release note: None
This is used by the Hive metadata store |
41252: roachtest: add test that aggregates orm blacklist failures r=jordanlewis a=jordanlewis The spreadsheet we discussed is unwieldy - hard to edit and impossible to keep up to date. If we write down blacklists in code, then we can use an approach like this to always have an up to date aggregation. So far it seems like there's just a lot of unknowns to categorize still. The output today: ``` === RUN TestBlacklists 648: unknown (unknown) 493: #5807 (sql: Add support for TEMP tables) 151: #17511 (sql: support stored procedures) 86: #26097 (sql: make TIMETZ more pg-compatible) 56: #10735 (sql: support SQL savepoints) 55: #32552 (multi-dim arrays) 55: #26508 (sql: restricted DDL / DML inside transactions) 52: #32565 (sql: support optional TIME precision) 39: #243 (roadmap: Blob storage) 33: #26725 (sql: support postgres' API to handle blob storage (incl lo_creat, lo_from_bytea)) 31: #27793 (sql: support custom/user-defined base scalar (primitive) types) 24: #12123 (sql: Can't drop and replace a table within a transaction) 24: #26443 (sql: support user-defined schemas between database and table) 20: #21286 (sql: Add support for geometric types) 18: #6583 (sql: explicit lock syntax (SELECT FOR {SHARE,UPDATE} {skip locked,nowait})) 17: #22329 (Support XA distributed transactions in CockroachDB) 16: #24062 (sql: 32 bit SERIAL type) 16: #30352 (roadmap:when CockroachDB will support cursor?) 12: #27791 (sql: support RANGE types) 8: #40195 (pgwire: multiple active result sets (portals) not supported) 8: #6130 (sql: add support for key watches with notifications of changes) 5: Expected Failure (unknown) 5: #23468 (sql: support sql arrays of JSONB) 5: #40854 (sql: set application_name from connection string) 4: #35879 (sql: `default_transaction_read_only` should also accept 'on' and 'off') 4: #32610 (sql: can't insert self reference) 4: #40205 (sql: add non-trivial implementations of FOR UPDATE, FOR NO KEY UPDATE, FOR SHARE, FOR NO KEY SHARE) 4: #35897 (sql: unknown function: pg_terminate_backend()) 4: #4035 (sql/pgwire: missing support for row count limits in pgwire) 3: #27796 (sql: support user-defined DOMAIN types) 3: #3781 (sql: Add Data Type Formatting Functions) 3: #40476 (sql: support `FOR {UPDATE,SHARE} {SKIP LOCKED,NOWAIT}`) 3: #35882 (sql: support other character sets) 2: #10028 (sql: Support view queries with star expansions) 2: #35807 (sql: INTERVAL output doesn't match PG) 2: #35902 (sql: large object support) 2: #40474 (sql: support `SELECT ... FOR UPDATE OF` syntax) 1: #18846 (sql: Support CIDR column type) 1: #9682 (sql: implement computed indexes) 1: #31632 (sql: FK options (deferrable, etc)) 1: #24897 (sql: CREATE OR REPLACE VIEW) 1: pass? (unknown) 1: #36215 (sql: enable setting standard_conforming_strings to off) 1: #32562 (sql: support SET LOCAL and txn-scoped session variable changes) 1: #36116 (sql: psychopg: investigate how `'infinity'::timestamp` is presented) 1: #26732 (sql: support the binary operator: <int> / <float>) 1: #23299 (sql: support coercing string literals to arrays) 1: #36115 (sql: psychopg: investigate if datetimetz is being returned instead of datetime) 1: #26925 (sql: make the CockroachDB integer types more compatible with postgres) 1: #21085 (sql: WITH RECURSIVE (recursive common table expressions)) 1: #36179 (sql: implicity convert date to timestamp) 1: #36118 (sql: Cannot parse '24:00' as type time) 1: #31708 (sql: support current_time) ``` Release justification: non-production change Release note: None Co-authored-by: Jordan Lewis <jordanthelewis@gmail.com>
Hi @RaduBerinde, I added this to the optimizer board, just letting you know. This feels like far future, but seems like we'd definitely need some optimizer support to defer the checks - or would it mostly be in execution? We could move the post queries to a special slot and execute only at the end of the txn. |
It'd definitely require more work. We can't just defer the same check query we do today - some later statement might "undo" a mutation, and the check would fail incorrectly. |
No difference, if the RDBMS does not support deferrable keys. So it's simply a matter of accepting the syntax. An FK can be deferrable and if it is, it is per default deferred (INITIALLY DEFERRED) or not (INITIALLY IMMEDIATE). Marking an FK as deferrable allows to use "SET CONSTRAINTS ALL DEFERRED" (if the RDBMS supports it). |
The situation is more complicated if the RDBMS may accept deferrable keys at a later stage. As explained above, in that case we cannot let future SET CONSTRAINTS statements affect tables created today without the DEFERRABLE clause. Similarly, we will need to let future SET CONSTRAINTS statement accept to affect tables created today with the DEFERRABLE clause. So we need to remember the clause even though SET CONSTRAINT is not supported yet. |
My remark assumed that you might want to simply support the syntax to start with (and ignore it, like you do with INITIALLY DEFERRED) |
"INITIALLY DEFERRED" is not being ignored today. It generates an error with a link to this issue. |
This blocks Django support as that wants to do (At least with Postgres, although I've now spotted https://github.com/cockroachdb/django-cockroachdb) |
Hi awsome guys, using django, as of today, still need this package?
|
Operations to perform:
Apply all migrations: account, admin, auth, authtoken, contenttypes, leasikApp, sessions, socialaccount
Running migrations:
Applying account.0001_initial...Traceback (most recent call last):
File "/home/abhishek/.work_on_home_virtualenvs/leasik-iCbC7nbs/lib/python3.8/site-packages/django/db/backends/utils.py", line 89, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.FeatureNotSupported: at or near "deferred": syntax error: unimplemented: this syntax
DETAIL: source SQL:
ALTER TABLE "account_emailaddress" ADD CONSTRAINT "account_emailaddress_user_id_2c513194_fk_auth_user_id" FOREIGN KEY ("user_id") REFERENCES "auth_user" ("id") DEFERRABLE INITIALLY DEFERRED
^
HINT: You have attempted to use a feature that is not yet implemented.
See: https://go.crdb.dev/issue-v/31632/v21.2
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "./manage.py", line 22, in <module>
main()
File "./manage.py", line 18, in main
execute_from_command_line(sys.argv)
File "/home/abhishek/.work_on_home_virtualenvs/leasik-iCbC7nbs/lib/python3.8/site-packages/django/core/management/__init__.py", line 446, in execute_from_command_line
utility.execute()
File "/home/abhishek/.work_on_home_virtualenvs/leasik-iCbC7nbs/lib/python3.8/site-packages/django/core/management/__init__.py", line 440, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/home/abhishek/.work_on_home_virtualenvs/leasik-iCbC7nbs/lib/python3.8/site-packages/django/core/management/base.py", line 414, in run_from_argv
self.execute(*args, **cmd_options)
File "/home/abhishek/.work_on_home_virtualenvs/leasik-iCbC7nbs/lib/python3.8/site-packages/django/core/management/base.py", line 460, in execute
output = self.handle(*args, **options)
File "/home/abhishek/.work_on_home_virtualenvs/leasik-iCbC7nbs/lib/python3.8/site-packages/django/core/management/base.py", line 98, in wrapped
res = handle_func(*args, **kwargs)
File "/home/abhishek/.work_on_home_virtualenvs/leasik-iCbC7nbs/lib/python3.8/site-packages/django/core/management/commands/migrate.py", line 290, in handle
post_migrate_state = executor.migrate(
File "/home/abhishek/.work_on_home_virtualenvs/leasik-iCbC7nbs/lib/python3.8/site-packages/django/db/migrations/executor.py", line 131, in migrate
state = self._migrate_all_forwards(
File "/home/abhishek/.work_on_home_virtualenvs/leasik-iCbC7nbs/lib/python3.8/site-packages/django/db/migrations/executor.py", line 163, in _migrate_all_forwards
state = self.apply_migration(
File "/home/abhishek/.work_on_home_virtualenvs/leasik-iCbC7nbs/lib/python3.8/site-packages/django/db/migrations/executor.py", line 251, in apply_migration
migration_recorded = True
File "/home/abhishek/.work_on_home_virtualenvs/leasik-iCbC7nbs/lib/python3.8/site-packages/django/db/backends/base/schema.py", line 157, in __exit__
self.execute(sql)
File "/home/abhishek/.work_on_home_virtualenvs/leasik-iCbC7nbs/lib/python3.8/site-packages/django/db/backends/base/schema.py", line 192, in execute
cursor.execute(sql, params)
File "/home/abhishek/.work_on_home_virtualenvs/leasik-iCbC7nbs/lib/python3.8/site-packages/django/db/backends/utils.py", line 103, in execute
return super().execute(sql, params)
File "/home/abhishek/.work_on_home_virtualenvs/leasik-iCbC7nbs/lib/python3.8/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(
File "/home/abhishek/.work_on_home_virtualenvs/leasik-iCbC7nbs/lib/python3.8/site-packages/django/db/backends/utils.py", line 80, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/home/abhishek/.work_on_home_virtualenvs/leasik-iCbC7nbs/lib/python3.8/site-packages/django/db/backends/utils.py", line 89, in _execute
return self.cursor.execute(sql, params)
File "/home/abhishek/.work_on_home_virtualenvs/leasik-iCbC7nbs/lib/python3.8/site-packages/django/db/utils.py", line 91, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/home/abhishek/.work_on_home_virtualenvs/leasik-iCbC7nbs/lib/python3.8/site-packages/django/db/backends/utils.py", line 89, in _execute
return self.cursor.execute(sql, params)
django.db.utils.NotSupportedError: at or near "deferred": syntax error: unimplemented: this syntax
DETAIL: source SQL:
ALTER TABLE "account_emailaddress" ADD CONSTRAINT "account_emailaddress_user_id_2c513194_fk_auth_user_id" FOREIGN KEY ("user_id") REFERENCES "auth_user" ("id") DEFERRABLE INITIALLY DEFERRED
^
HINT: You have attempted to use a feature that is not yet implemented.
See: https://go.crdb.dev/issue-v/31632/v21.2 |
Running migrations: The above exception was the direct cause of the following exception: Traceback (most recent call last): |
I am getting such an error |
I'm also getting the UPDATE: Stupid mistake, I set |
I've encountered this with Vaultwarden (Bitwarden server implementation), which attempts to |
I've ran into this when deploying dpaste
|
This reverts commit 1d34a35. Turns out: nautobot requires some of the edgecases that CockroachDB does not support. See [#31632](cockroachdb/cockroach#31632)
Getting the same w doctrine migrations, I really hope this will be supported at some point |
Same issue, running Flagsmith (which uses Django). |
Informs cockroachdb#22329. This commit adds the max_prepared_transactions session variable, returning a value of math.MaxInt32 to indicate that there is no limit on the number of prepared transactions that can be created. This change has the effect of unblocking the pgjdbc XADataSourceTest test suite, which was previously failing during initialization. Is also includes a small patch to work around cockroachdb#31632 (unsupported deferrable qualifier) so that each test in the XA test suite actually runs (and fails). Release note: None
Informs cockroachdb#22329. This commit adds the max_prepared_transactions session variable, returning a value of math.MaxInt32 to indicate that there is no limit on the number of prepared transactions that can be created. This change has the effect of unblocking the pgjdbc XADataSourceTest test suite, which was previously failing during initialization. Is also includes a small patch to work around cockroachdb#31632 (unsupported deferrable qualifier) so that each test in the XA test suite actually runs (and fails). Release note: None
Informs cockroachdb#22329. This commit fixes the pgjdbc blocklist to reflect the current state of support for XA transactions in CockroachDB. With the newly added support for two-phase commit transactions, all tests in the XADataSourceTest test suite now pass except for `mappingOfConstraintViolations`, which is blocked by cockroachdb#31632 (DEFERRABLE foreign keys). Release note: None
Is your feature request related to a problem? Please describe.
FK options are tuning knob that help developers decide when the FK relations become active. They are useful to enable fast import of known-consistent data.
Describe the solution you'd like
Support DEFERRABLE, NOT DEFERRABLE, INITIALLY DEFERRED, INITIALLY IMMEDIATE like postgresql.
Epic: CRDB-9559
Jira issue: CRDB-4783
The text was updated successfully, but these errors were encountered: