Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

perf(duckdb): speed up memtable registration #9419

Merged

Conversation

cpcloud
Copy link
Member

@cpcloud cpcloud commented Jun 21, 2024

Speeds up memtable registration by a factor of some of multiple of N, where N is the number of tables in the database. In other words, the time to register a memtable is no longer dependent on the number of tables in the database.

@cpcloud cpcloud added this to the 9.2 milestone Jun 21, 2024
@cpcloud cpcloud added performance Issues related to ibis's performance duckdb The DuckDB backend labels Jun 21, 2024
@cpcloud cpcloud requested review from gforsyth and jcrist June 21, 2024 11:08
@cpcloud cpcloud force-pushed the duckdb-speed-up-memtable-registration branch from f8e5bee to 76620ed Compare June 21, 2024 11:14
@cpcloud cpcloud requested a review from ncclementi June 21, 2024 11:17
@cpcloud cpcloud force-pushed the duckdb-speed-up-memtable-registration branch from 76620ed to 9805034 Compare June 21, 2024 12:01
Copy link
Member

@jcrist jcrist left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fine by me to merge for now. We should really apply this kind of change universally across all our backends. Someone raised a similar issue in #9216 also caused by this, but on the bigquery backend instead.

ibis/backends/duckdb/__init__.py Show resolved Hide resolved
@cpcloud cpcloud merged commit 7878d8c into ibis-project:main Jun 21, 2024
75 checks passed
@cpcloud cpcloud deleted the duckdb-speed-up-memtable-registration branch June 21, 2024 18:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
duckdb The DuckDB backend performance Issues related to ibis's performance
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants