synapse_port_db fails because 'ascii' codec can't encode character #8993
Comments
|
It would be interesting to see what in Something like the following might work (although I'm not sure about the number of backslashes tto use there 😄). SELECT * FROM room_memberships WHERE display_name LIKE "%\\xfc%";Note that the result may leak some private info. If you can confirm that's the issue then I suspect it is a bug in the port DB script. |
|
Yes, I confirm that |
|
and there are no other columns with that character. |
clokep
added
z-bug
|
I cannot reproduce this. The character exists in my sqlite database: The port script runs successfully and the character appears in my Postgres database: |
Test case for matrix-org/synapse#8993
|
@haslersn I was wondering if you could try running synapse_port_db against the snapshot in https://github.com/callahad/synapse-8993? That's a copy of my SQLite database from trying to reproduce as above. There are two users in the database, |
|
With your database snapshot I get the same error. |
|
Initially I was wrong in that it's Debian 10. It's actually Debian 9. I updated my original post. |
|
Fascinating! So it's something with your environment... Could you please check what the encoding is for your postgres database? `psql -l` should do it, or `SHOW SERVER_ENCODING;` when connected to the database.
…On Sat, Jan 9, 2021, at 13:00, haslersn wrote:
Initially I was wrong in that it's Debian 10. It's actually Debian 9. I updated my original post.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub <#8993 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAF5ALSJ26WBCD2XAH6Z23SZBHO5ANCNFSM4VLQIFLQ>.
|
|
|
The PostgreSQL version I used was 9.6. Now I am trying to migrate to a PostgreSQL 12 cluster running outside the Debian 9 VM, and that seems to work. (It's still running.) |
|
I suspect this is https://bugs.python.org/issue19846, which was fixed in Python 3.7 Can you try re-running the port_db script, prepending E.g., |
|
Migrating to PostgreSQL 12 worked, and I did it with Python 3.5.3. Do you still think it could be that Python issue19846? |
|
That's a fair point, and I can't for the life of me reproduce this issue on my end, even using Python 3.5.3 and both Postgres 9.5 and 9.6. I'm going to close this issue as you were able to port out by targeting PostgreSQL 12, but I'd love to reopen and resolve it if we can find reliable steps to reproduce the bug. I really appreciate your help throughout. |
|
It looks like I'm currently being bitten by this bug... |
|
/me frowns, clicks the "Reopen and comment" button |
callahad
added
T-Defect
If you'd like to start a DM with me to try and resolve, @ed:geraghty.london |
|
So the original reporter was seeing issues encoding |
|
I'm currently attempting to migrate from sqlite on an MS Azure Debian 10 instance to a "postgresql for Azure", which means I'm not 100% in control of the configs on both sides. Synapse: I've tried setting the locale to all sorts of things with no success, it's currently I've also tried using synapse server version 12 Also, it dies on different unicode chars each time I run it, I'm guessing the porting script isn't deterministic in its ordering? 🤔 |
|
Sorry for the delay here; I'm still unable to reproduce this... I'm curious if you can successfully port the test database from my earlier comment? |
|
|
I don't want to mess around with this box too much! I'm going to try migrating to a local postgres instance, take one step out of the chain |
|
It would appear that's done the trick 🤷♂️ guess you can close the ticket again...? |
|
Strange! Thank you for checking. |
I wanted to port my db from sqlite to postgres, but
synapse_port_dbfails with below-mentioned error.Steps to reproduce
These are the steps I performed.
homeserver.dbtohomeserver.db.snapshot.homeserver.yamltohomeserver-postgres.yaml.homeserver-postgres.yamlto use postgres instead of sqlite.synapse_port_db --sqlite-database homeserver.db.snapshot --postgres-config homeserver-postgres.yamlThe error
Version information
{"python_version":"3.5.3","server_version":"1.24.0"}The text was updated successfully, but these errors were encountered: