Skip to content
This repository has been archived by the owner on Apr 26, 2024. It is now read-only.

Convert simple_select_list and simple_select_list_txn to return lists of tuples #16505

Merged
merged 7 commits into from
Oct 26, 2023

Conversation

clokep
Copy link
Member

@clokep clokep commented Oct 16, 2023

More of #16431, this is again fairly large since these are used everywhere.

@clokep
Copy link
Member Author

clokep commented Oct 16, 2023

(I probably need to double check the return types of these.)

@clokep clokep force-pushed the clokep/axe-cursor-to-dict-5 branch from 965e465 to a91e34d Compare October 17, 2023 18:41
"thumbnail_type",
"thumbnail_length",
rows = cast(
List[Tuple[int, int, str, str, int]],
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These seem to weirdly be all nullable?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"thumbnail_type",
"thumbnail_length",
rows = cast(
List[Tuple[int, int, str, str, int]],
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These seem to be all nullable.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

retcols=("room_id", "joined_via"),
desc="get_server_which_served_partial_join",
rows = cast(
List[Tuple[str, str]],
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

joined_via is nullable?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think I added that column after the fact.

ISTR that recent postgresses have made it so that adding a new column with a non-null default is much faster than it used to be. Maybe we don't need to add nullable columns when extending a table these days?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is as far as I reviewed up to

Copy link
Contributor

@DMRobertson DMRobertson Oct 19, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ISTR that recent postgresses have made it so that adding a new column with a non-null default is much faster than it used to be. Maybe we don't need to add nullable columns when extending a table these days?

PG 11's changelog says

Many other useful performance improvements, including the ability to avoid a table rewrite for ALTER TABLE ADD COLUMN with a non-null column default

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this table is usually small so we can also probably just change it on the fly? I've filed an issue for now: #16547

"is_verified",
"session_data",
rows = cast(
List[Tuple[str, str, int, int, int, str]],
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

first_message_index and forwarded_count and is_verified is nullable. (The last doesn't matter we cast to a bool below.)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See #16548.

@clokep
Copy link
Member Author

clokep commented Oct 18, 2023

@DMRobertson Any thoughts on this? Should I just file issues for the above?

synapse/storage/database.py Outdated Show resolved Hide resolved
synapse/storage/database.py Outdated Show resolved Hide resolved
retcols=("room_id", "joined_via"),
desc="get_server_which_served_partial_join",
rows = cast(
List[Tuple[str, str]],
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think I added that column after the fact.

ISTR that recent postgresses have made it so that adding a new column with a non-null default is much faster than it used to be. Maybe we don't need to add nullable columns when extending a table these days?

retcols=("room_id", "joined_via"),
desc="get_server_which_served_partial_join",
rows = cast(
List[Tuple[str, str]],
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is as far as I reviewed up to

@DMRobertson
Copy link
Contributor

@DMRobertson Any thoughts on this? Should I just file issues for the above?

My preference would be to update the type hints that doesn't cause new type errors. If there are new type errors, then yeah let's file an issue. (Also happy for you to punt it and just file an issue :))

Copy link
Contributor

@DMRobertson DMRobertson left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure if me double checking these types is useful, but those are the discrepancies I could spot.

@clokep clokep marked this pull request as ready for review October 25, 2023 18:06
@clokep clokep requested a review from a team as a code owner October 25, 2023 18:06
@clokep clokep requested review from DMRobertson and removed request for a team October 25, 2023 18:06
@clokep clokep force-pushed the clokep/axe-cursor-to-dict-5 branch from 9c880ad to 5153b9a Compare October 26, 2023 11:49
self.mock_txn.__iter__ = Mock(return_value=iter([(1,), (2,), (3,)]))
self.mock_txn.fetchall.return_value = [(1,), (2,), (3,)]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To check: I think that this is needed because we've changed the implementation to use fetchall (rather than a dictionary comprehension that loops over the psycopg cursor). Do I understand correctly?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, exactly! 👍

Copy link
Contributor

@DMRobertson DMRobertson left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for bearing with me and seeing this through. LGTM!

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants