Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor: Replace lib/pq with pgx in a PostgreSQL scaler #4705

Merged

Conversation

a-narsudinov
Copy link
Contributor

@a-narsudinov a-narsudinov commented Jun 17, 2023

The development of lib/pq is over and the library doesn't receive any fixes or new features. This is sad because nowadays lib/pq lacks a of lot of features we are get used to in the C version of libpq and its derivatives.

I've experienced troubles with this unimplemented stuff in lib/pq:

  • Multi-host connection, which is important for a high availability
  • Support of parsing CA certificate of PostgreSQL server stored in a ~/.postgresql folder that led to a negative user experience

I believe there are more missing features.

These things will be never fixed in lib/pq, so in this patch, we replace the abandoned lib/pq with an actively developing pgx.

I tested this patch in my environment and got that I needed: high avaliabillity with help of multihost connection which just works in pgx driver.

I think that it mustn't have a breaking changes and may be merged easily.

Checklist

Fixes #4704

@a-narsudinov a-narsudinov requested a review from a team as a code owner June 17, 2023 21:03
@github-actions
Copy link

Thank you for your contribution! 🙏 We will review your PR as soon as possible.

While you are waiting, make sure to:

Learn more about:

@a-narsudinov a-narsudinov changed the title refactor: Replace lib/pq with pgx in postgresql_scaler.go refactor: Replace lib/pq with pgx in a PostgreSQL scaler Jun 17, 2023
@a-narsudinov a-narsudinov force-pushed the refactor/pg-scaler/replace-libpq branch from beffda5 to 6bc73e4 Compare June 17, 2023 21:27
Copy link
Member

@zroubalik zroubalik left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@a-narsudinov nice, thanks for the contribution. Could you please add a test to cover the multihost scenario which is discussed in the issue?

@a-narsudinov
Copy link
Contributor Author

a-narsudinov commented Jun 18, 2023

@zroubalik I believe I can add test for this scenario, but could you please help me figure out how to implement it better? I guess the proper way to test it is to modify the current end-to-end postgresql_test.go test and make it run a master-replica PostgreSQL setup. Am I right?

@zroubalik
Copy link
Member

@zroubalik I believe I can add test for this scenario, but could you please help me figure out how to implement it better? I guess the proper way to test it is to modify the current end-to-end postgresql_test.go test and make it run a master-replica PostgreSQL setup. Am I right?

Yeah, that's what I was thinking about https://github.com/kedacore/keda/blob/main/tests/scalers/postgresql/postgresql_test.go

Also, do you think it makes sense to add unit test for parsing the multihost setup?

@a-narsudinov
Copy link
Contributor Author

To be honest, I didn't even test the case when we parse the data from separate fields and use it to build a connection string via strings joiner. By the way it should work nicely right now, but I suppose we should add a unit test for this scenario just so we won't forget about this very nice feature during the next refactoring and won't break it. This is not so much work compared to the e2e test, so we just may do it, even if it isn't essential now.

@a-narsudinov
Copy link
Contributor Author

I'm really interested in merging this PR into the mainline until the upcoming release, how much time do I have to do the tests? :D

@zroubalik
Copy link
Member

I'm really interested in merging this PR into the mainline until the upcoming release, how much time do I have to do the tests? :D

😄 We plan to do release later this week (Wed-Thu). The best would be to include the test by Tuesday EOD.

@a-narsudinov a-narsudinov force-pushed the refactor/pg-scaler/replace-libpq branch 2 times, most recently from 8c64786 to 3fb4e21 Compare June 20, 2023 10:17
@a-narsudinov
Copy link
Contributor Author

Hi @zroubalik! I think that this PR looks good now with a new tests. Could you please have a look? :)

@zroubalik
Copy link
Member

zroubalik commented Jun 20, 2023

/run-e2e postgresql
Update: You can check the progress here

@a-narsudinov a-narsudinov force-pushed the refactor/pg-scaler/replace-libpq branch from 3fb4e21 to 07d6cf2 Compare June 20, 2023 13:27
@a-narsudinov
Copy link
Contributor Author

I'm sorry, I made a typo in my email and I had to re-push one more time :(

The development of `lib/pq` is over and the library doesn't receive any fixes
or new features. This is sad because nowadays `lib/pq` lacks a of lot of
features we are get used to in the C version of `libpq` and its derivatives.

I've experienced troubles with this unimplemented stuff in `lib/pq`:
- Multi-host connection, which is important for a high availability
- Support of parsing CA certificate of PostgreSQL server stored in
a `~/.postgresql` folder that led to a negative user experience

I believe there are more missing features.

These things will be never fixed in `lib/pq`, so in this patch, we replace the
abandoned `lib/pq` with an actively developing `pgx`.

Signed-off-by: Alexander Narsudinov <a.narsudinov@gmail.com>
Signed-off-by: Alexander Narsudinov <a.narsudinov@gmail.com>
We are going to introduce a new test for HA PostgreSQL, so this patch moves
all reusable code from the test code to the helper.

In the next patch we'll use this helper to build a new test case.

Signed-off-by: Alexander Narsudinov <a.narsudinov@gmail.com>
This patch allows to check if the KEDA operator able to use multihost connection
to PostgreSQL HA setup.

Signed-off-by: Alexander Narsudinov <a.narsudinov@gmail.com>
Signed-off-by: Alexander Narsudinov <a.narsudinov@gmail.com>
@a-narsudinov a-narsudinov force-pushed the refactor/pg-scaler/replace-libpq branch from 07d6cf2 to 3749348 Compare June 20, 2023 13:34
@zroubalik
Copy link
Member

zroubalik commented Jun 20, 2023

/run-e2e postgresql
Update: You can check the progress here

Copy link
Member

@zroubalik zroubalik left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

thanks for the contribution!

@zroubalik zroubalik merged commit 26a82b3 into kedacore:main Jun 21, 2023
SpiritZhou pushed a commit to SpiritZhou/keda that referenced this pull request Jun 30, 2023
Signed-off-by: Alexander Narsudinov <a.narsudinov@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Provide a way to use multihost connection in a PostgreSQLScaler
2 participants