Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

opt: memo cache is not invalidated when a user-defined type's schema is renamed #96674

Closed
mgartner opened this issue Feb 6, 2023 · 4 comments · Fixed by #96045
Closed

opt: memo cache is not invalidated when a user-defined type's schema is renamed #96674

mgartner opened this issue Feb 6, 2023 · 4 comments · Fixed by #96045
Labels
C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. T-sql-queries SQL Queries Team

Comments

@mgartner
Copy link
Collaborator

mgartner commented Feb 6, 2023

We don't invalidate a cached memo referencing user-defined types when the schema owning the UDT is renamed.

defaultdb> create type sc1.t as enum ('foo', 'bar');
CREATE TYPE

defaultdb> select 'foo'::sc1.t;
t
-------
foo
(1 row)

defaultdb> alter schema sc1 rename to sc2;
ALTER SCHEMA

defaultdb> select 'foo'::sc1.t;
t
-------
foo
(1 row)

The last query should fail because sc1 no longer exists.

Jira issue: CRDB-24272

@mgartner mgartner added the C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. label Feb 6, 2023
@blathers-crl blathers-crl bot added the T-sql-queries SQL Queries Team label Feb 6, 2023
@ajwerner
Copy link
Contributor

ajwerner commented Feb 6, 2023

Won't this fall out of drew's work on checking whether the memo is stale? If it doesn't the reason it doesn't would be that we're not tracking a reference to the schema.

@mgartner
Copy link
Collaborator Author

mgartner commented Feb 6, 2023

This is caused by the fact that we stored UDT OID's in the memo and lookup the types by OID to determine if they have changed.

toCheck, err := catalog.ResolveTypeByOID(ctx, typ.Oid())

We need to either:

  • Store the original type reference in the memo and look up by that to determine if we have a stale memo.
  • Or, track schemas referenced in a memo and check their version to determine if the memo is stale.

@DrewKimball
Copy link
Collaborator

This is a problem for tables referenced by IDs, too... I'm going to try adding schemas to the list of objects tracked during memo validation so we can just track function OIDs. I think it should allow us to make a similar simplification for data sources as well. @chengxiong-ruan added a fix for the switching databases problem in functions here; I think we could do something similar for resolving types, schemas, and data sources.

@chengxiong-ruan
Copy link
Contributor

This is a problem for tables referenced by IDs, too... I'm going to try adding schemas to the list of objects tracked during memo validation so we can just track function OIDs. I think it should allow us to make a similar simplification for data sources as well. @chengxiong-ruan added a fix for the switching databases problem in functions here; I think we could do something similar for resolving types, schemas, and data sources.

Yes, we'd better have defense on those OID references as well. Though it's only fun when users know the numeric representation for some reason and spend effort on finding the IDs heh.

DrewKimball added a commit to DrewKimball/cockroach that referenced this issue Feb 22, 2023
The metadata stores tracks each UDT referenced in the query by its OID. It
validates the cached data by checking that the OID still resolves to the
expected type and that the type hasn't undergone schema changes since the query
was resolved. The problem with tracking only by OID instead of by the name used
to reference the type is that the query cache can incorrectly pass validation
when the UDT's schema or database has been altered in some way. This previously
led to behavior like the following:
```
defaultdb> create type sc1.t as enum ('foo', 'bar');
CREATE TYPE

defaultdb> select 'foo'::sc1.t;
t
-------
foo
(1 row)

defaultdb> alter schema sc1 rename to sc2;
ALTER SCHEMA

defaultdb> select 'foo'::sc1.t;
t
-------
foo
(1 row)
```

This patch adds schema tracking to the metadata dependency tracking. For now,
this is only done for UDT resolution, but following commits will adapt data
source and function resolution to also take advantage of this change. This
allows catalog objects to be tracked in the metadata by ID rather than by
name, which simplifies the implementation. This ensures a cached query will
be properly invalidated if the schema or database of a UDT in the query is
altered or dropped.

Fixes cockroachdb#96674

Release note (bug fix): Fixed a bug that could cause a query involving
user-defined types to ignore schema changes when the same query was invoked
before and after the change.
DrewKimball added a commit to DrewKimball/cockroach that referenced this issue Feb 27, 2023
The metadata stores tracks each UDT referenced in the query by its OID. It
validates the cached data by checking that the OID still resolves to the
expected type and that the type hasn't undergone schema changes since the query
was resolved. The problem with tracking only by OID instead of by the name used
to reference the type is that the query cache can incorrectly pass validation
when the UDT's schema or database has been altered in some way. This previously
led to behavior like the following:
```
defaultdb> create type sc1.t as enum ('foo', 'bar');
CREATE TYPE

defaultdb> select 'foo'::sc1.t;
t
-------
foo
(1 row)

defaultdb> alter schema sc1 rename to sc2;
ALTER SCHEMA

defaultdb> select 'foo'::sc1.t;
t
-------
foo
(1 row)
```

This patch adds schema tracking to the metadata dependency tracking. For now,
this is only done for UDT resolution, but following commits will adapt data
source and function resolution to also take advantage of this change. This
allows catalog objects to be tracked in the metadata by ID rather than by
name, which simplifies the implementation. This ensures a cached query will
be properly invalidated if the schema or database of a UDT in the query is
altered or dropped.

Fixes cockroachdb#96674

Release note (bug fix): Fixed a bug that could cause a query involving
user-defined types to ignore schema changes when the same query was invoked
before and after the change.
DrewKimball added a commit to DrewKimball/cockroach that referenced this issue Feb 28, 2023
The metadata stores tracks each UDT referenced in the query by its OID. It
validates the cached data by checking that the OID still resolves to the
expected type and that the type hasn't undergone schema changes since the query
was resolved. The problem with tracking only by OID instead of by the name used
to reference the type is that the query cache can incorrectly pass validation
when the UDT's schema or database has been altered in some way. This previously
led to behavior like the following:
```
defaultdb> create type sc1.t as enum ('foo', 'bar');
CREATE TYPE

defaultdb> select 'foo'::sc1.t;
t
-------
foo
(1 row)

defaultdb> alter schema sc1 rename to sc2;
ALTER SCHEMA

defaultdb> select 'foo'::sc1.t;
t
-------
foo
(1 row)
```

This patch adds schema tracking to the metadata dependency tracking. For now,
this is only done for UDT resolution, but following commits will adapt data
source and function resolution to also take advantage of this change. This
allows catalog objects to be tracked in the metadata by ID rather than by
name, which simplifies the implementation. This ensures a cached query will
be properly invalidated if the schema or database of a UDT in the query is
altered or dropped.

Fixes cockroachdb#96674

Release note (bug fix): Fixed a bug that could cause a query involving
user-defined types to ignore schema changes when the same query was invoked
before and after the change.
DrewKimball added a commit to DrewKimball/cockroach that referenced this issue Mar 7, 2023
Previously, it was possible for invalid queries to be kept in the cache
after a schema change, since user-defined types were tracked only by OID.
This missed cases where the search path changed which object a name in
the query resolved to (or whether it resolved at all).

This patch fixes this behavior by tracking UDT references by name, similar
to what was already done for data sources. This ensures that the query
staleness check doesn't miss changes to the search path.

Fixes cockroachdb#96674

Release note (bug fix): Fixed a bug that could prevent a cached query with
a user-defined type reference from being invalidated even after a schema
change that should prevent the type from being resolved.
DrewKimball added a commit to DrewKimball/cockroach that referenced this issue Mar 8, 2023
Previously, it was possible for invalid queries to be kept in the cache
after a schema change, since user-defined types were tracked only by OID.
This missed cases where the search path changed which object a name in
the query resolved to (or whether it resolved at all).

This patch fixes this behavior by tracking UDT references by name, similar
to what was already done for data sources. This ensures that the query
staleness check doesn't miss changes to the search path.

Fixes cockroachdb#96674

Release note (bug fix): Fixed a bug that could prevent a cached query with
a user-defined type reference from being invalidated even after a schema
change that should prevent the type from being resolved.
DrewKimball added a commit to DrewKimball/cockroach that referenced this issue Mar 10, 2023
Previously, it was possible for invalid queries to be kept in the cache
after a schema change, since user-defined types were tracked only by OID.
This missed cases where the search path changed which object a name in
the query resolved to (or whether it resolved at all).

This patch fixes this behavior by tracking UDT references by name, similar
to what was already done for data sources. This ensures that the query
staleness check doesn't miss changes to the search path.

Fixes cockroachdb#96674

Release note (bug fix): Fixed a bug that could prevent a cached query with
a user-defined type reference from being invalidated even after a schema
change that should prevent the type from being resolved.
craig bot pushed a commit that referenced this issue Mar 20, 2023
96045: opt: check UDFs and UDTs by name when checking metadata dependencies r=DrewKimball a=DrewKimball

#### tree: distinguish UDFs and builtins that use a SQL string body

This refactor changes the meaning of the `Overload.IsUDF` field to
be true only for user-defined functions - meaning those created using
`CREATE FUNCTION`. This is contrasted with builtin functions that are
defined with a SQL string set in `Overload.Body`. Logic that cares only
about user-defined functions can continue checking `Overload.IsUDF`,
while cases that deal with the SQL body should use `Overload.HasSQLBody()`.
Note that it is insufficient to check whether `Overload.Body` is empty
because it is possible to define a UDF with an empty body.

#### opt: refactor metadata dependency tracking

This commit performs some refactoring for the way data source objects
are tracked in `opt.Metadata` in order to make future changes to UDT
and UDF dependency tracking easier. More particularly, UDTs and UDFs
will be able to re-resolve any references by name. This is necessary
in order to handle cases where changes to the search-path cause names
in the query to resolve to different objects.

#### opt: track UDT references by name in the Metadata

Previously, it was possible for invalid queries to be kept in the cache
after a schema change, since user-defined types were tracked only by OID.
This missed cases where the search path changed which object a name in
the query resolved to (or whether it resolved at all).

This patch fixes this behavior by tracking UDT references by name, similar
to what was already done for data sources. This ensures that the query
staleness check doesn't miss changes to the search path.

#### opt: track UDFs in the Metadata

This patch adds tracking for user-defined functions in `opt.Metadata`.
This ensures that the query cache will be correctly invalidated after
a schema change or search-path change that could change the query's
semantics, or cause it to error.

Fixes #95214
Fixes #93082
Fixes #93321
Fixes #96674

Release note (bug fix): Fixed a bug that could prevent a cached query from being invalidated when a UDF or UDT referenced by that query was altered or dropped, or when the schema or database of a UDF or UDT was altered or dropped.

98319: pkg/server: fix `/demologin` to properly redirect to home page r=dhartunian a=abarganier

With the introduction of the server controller, we introduced a layer between the HTTP handler and the HTTP server. When this was introduced, the logic to attempt a login to all tenants forgot to handle a specific case for `/demologin`, where the status code is set to a 302 redirect, instead of a 200 status OK.

This broke the redirect piece of the `/demologin` endpoint.

This patch updates the `attemptLoginToAllTenants` HTTP handler to properly set the 302 response code in the case where the underlying login function does so on the sessionWriter.

Release note: none

Epic: CRDB-12100

Fixes: #98253

98696: sql: disallow using cluster_logical_timestamp as column default when backfilling r=Xiang-Gu a=Xiang-Gu

Previously, `ADD COLUMN ... DEFAULT cluster_logical_timestamp()` would crash the node and leave the table in a corrupt state. The root cause is a nil pointer dereference. This commit fixed it by returning an unimplemented error and hence disallow using this builtin function as default value when backfilling.

Fixes: #98269
Release note (bug fix): fixed a bug as detailed in #98269.

Co-authored-by: Drew Kimball <drewk@cockroachlabs.com>
Co-authored-by: Alex Barganier <abarganier@cockroachlabs.com>
Co-authored-by: Xiang Gu <xiang@cockroachlabs.com>
@craig craig bot closed this as completed in 5552037 Mar 20, 2023
DrewKimball added a commit to DrewKimball/cockroach that referenced this issue Mar 30, 2023
Previously, it was possible for invalid queries to be kept in the cache
after a schema change, since user-defined types were tracked only by OID.
This missed cases where the search path changed which object a name in
the query resolved to (or whether it resolved at all).

This patch fixes this behavior by tracking UDT references by name, similar
to what was already done for data sources. This ensures that the query
staleness check doesn't miss changes to the search path.

Fixes cockroachdb#96674

Release note (bug fix): Fixed a bug that could prevent a cached query with
a user-defined type reference from being invalidated even after a schema
change that should prevent the type from being resolved.
DrewKimball added a commit to DrewKimball/cockroach that referenced this issue Mar 31, 2023
Previously, it was possible for invalid queries to be kept in the cache
after a schema change, since user-defined types were tracked only by OID.
This missed cases where the search path changed which object a name in
the query resolved to (or whether it resolved at all).

This patch fixes this behavior by tracking UDT references by name, similar
to what was already done for data sources. This ensures that the query
staleness check doesn't miss changes to the search path.

Fixes cockroachdb#96674

Release note (bug fix): Fixed a bug that could prevent a cached query with
a user-defined type reference from being invalidated even after a schema
change that should prevent the type from being resolved.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. T-sql-queries SQL Queries Team
Projects
Archived in project
Development

Successfully merging a pull request may close this issue.

4 participants