Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: asyncpg adapter #177

Closed
wants to merge 13 commits into from
Closed

feat: asyncpg adapter #177

wants to merge 13 commits into from

Conversation

cofin
Copy link

@cofin cofin commented Oct 13, 2024

This PR implements the asyncpg adapter mentioned in #172

I've left the current psycopg adapter in as a reference, but I can remove (if needed) based on feedback here.

saq/queue/postgres_asyncpg.py Outdated Show resolved Hide resolved
saq/queue/postgres_asyncpg.py Outdated Show resolved Hide resolved
@tobymao
Copy link
Owner

tobymao commented Oct 13, 2024

i just merged my commit, can you rebase?

@tobymao
Copy link
Owner

tobymao commented Oct 13, 2024

i can't really compare the two implementations like this, can you redo it so that you just rewrite the existing postgres adapter?

@cofin
Copy link
Author

cofin commented Oct 13, 2024

i can't really compare the two implementations like this, can you redo it so that you just rewrite the existing postgres adapter?

Yep, can do. I'll also bring in your updated changes.

Comment on lines -64 to -68
min_size: minimum pool size. (default 4)
The minimum number of Postgres connections.
max_size: maximum pool size. (default 20)
If greater than 0, this limits the maximum number of connections to Postgres.
Otherwise, maintain `min_size` number of connections.
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Moved to the from_url method since it can't be modified after initial creation.

@cofin cofin marked this pull request as ready for review October 13, 2024 20:00
@cofin
Copy link
Author

cofin commented Oct 13, 2024

I've made the updates for the group and priority settings. I think it's ready for you to take a look at this point.

)
result = await cursor.fetchone()
self.assertIsNone(result)

async def test_cron_job_close_to_target(self) -> None:
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This test is a bit flaky. It passes about 50-60% of the time for me.

)
).format(
jobs_table=self.jobs_table,
expire_at=SQL(",expire_at = %(expire_at)s" if expire_at != -1 else ""),
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is cool syntax sugar I wasn't aware of from psycopg.

Since there isn't a direct equivalent to this, the easiest thing to do was to break this into 2 statements.

@tobymao
Copy link
Owner

tobymao commented Oct 14, 2024

benchmarks got 8.5x slower, it needs to be fixed before this is considered,

benchmarks/simple.py saq_pg

@tobymao
Copy link
Owner

tobymao commented Oct 14, 2024

i'm also not a big fan of the api of asyncio compared to psycopg, looks like it's not dbapi.

given the benchmarks, i'm so far not inclined to accept it, considering lgpl v3 only means users cannot modify the source, it doesn't seem to be that detrimental, but i do understand some companies cannot adhere to it.

@cofin
Copy link
Author

cofin commented Oct 14, 2024

That's quite a surprising difference on performance. I definitely would not have expected that great of a difference.

Re: the dbapi comment, you are correct. It does not have a dbapi compatible interface. Given this difference, do you want me to spend time investigating the performance gap?

@tobymao
Copy link
Owner

tobymao commented Oct 14, 2024

@cofin i've been thinking about it a bit and i apologize but i don't think i have much appetite for this at the moment, so i think we can close this for now. sorry about that and thanks for the effort.

@tobymao tobymao closed this Oct 14, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants