sqlite: dropping fetch result without reading to the end breaks connection
#1147
Comments
|
I reduced the test to: Add it to the end of |
|
The problem seems to be that the statement handle is not dropped at the end of "Break the reader" call, because the statement is cached. And it is not even finalized as long as the handle is in the cache. I have been able to work around the problem with |
link2xt
added a commit
to deltachat/deltachat-core-rust
that referenced
this issue
Apr 2, 2021
This is a workaround for launchbadge/sqlx#1147 Draining statement results is no longer needed.
dignifiedquire
added a commit
to deltachat/deltachat-core-rust
that referenced
this issue
Apr 6, 2021
* refactor: move sql into own mod * refactor: resultify sql interface * fixup * start usage of sqlx * compile again * migrate exists * migrate query_row and query_row_optional * migrate table_exists * migrate with_conn * migrate rowid * migrate most of query_map * finish query_map migration * remove rusqlite and fixup examples * improve migrations and ensure closing * fix rust tests * wip * fixup rebase and get things to compile * start fixing tests * use sqlx fork * debugging * fix receive imf insert * fix message loading * fix grpid bug * exists() expects COUNT() in sql-statement * fix two more exists() statements * fix test_audit_log_view_without_daymarker * Fix clippy warnings * fix test_fetch_existing dc_get_config() must never return NULL, instead return "" * ffi: make render_info accept non-option BTreeMap * fix a typo * Restore changes from PR #2258 They have been accidentally reverted during sqlx rebasing * Remove useless cast that was not there before switch to sqlx * Split long query into several lines so rustfmt does not fail * Remove outdated comment DC_CONTACT_ID_SELF is bound into the query now. * Split long lines to make rustfmt work * Restore accidentally removed part of PR #2132 * Make rustfmt work * Use exists() instead of count() in is_group_explicitly_left * Make rustfmt work * Fix a typo ("foreing") * Fix rustfmt * Fix formatting * Replace 0x100 constant with Origin::IncomingReplyTo * Replace constant 100 with Chattype::Single * Remove commented out code * peerstate: make rustfmt work * Move the code to make msg immutable * do not re-use bind parameters sqlx does not bind the first `?` implicitly to `?1`, therefore the sql-statement fails. we could use `?1` several times, however, i did not find good documentation about that, and even if that works now, that could fail with other db or even with sqlite in newer versions. therefore, we just bind every parameter explicitly. * Do not use read only sqlite mode * Factor out database pool creation * Open reader pool in readonly mode if database is opened in readonly mode * Add a test for reopening the database * sql::table_exists: drain response stream to avoid breaking connection * Drain the stream in Sql.col_exists * sqlx: fix failing TestOnlineAccount::test_saved_mime_on_received_message (#2323) * add failing tests for save_mime_headers * fix saving mime if an encrypted message is received, and we do not have the decoded data, for whatever reason, save the raw data. * let dc_get_mime_headers return NULL on empty string * let test_save_mime_headers_*() tests return anyhow::Result<()> * make clippy happy * Remove 5 unwraps * Remove another unwrap * Replace {:?} with {:#} for error handling * Remove all dbg! statements * set max_connections=10 (default), show num_cpus in get_info() setting the number of connections to the number of cpus seems to be to low in general: - on a mac i5 from 2017, that would result in 4 connections - on an iphone7, that would result in only 2 connections the sqlx-default is 10 connections, so, this pr uses that default (plus 1 connection for the writer tag). on the iphone7, that change was directly noticable - sqlx with 2 connections feelt not much faster that rusqlite with 10 connections, however, using sqlx with the 10+1 connection was a real boost, at least at a first glance :) might be, fewer connections would be sufficient as well, however, i do not see much reasons not to go with the default here. * Disable sqlx statement cache This is a workaround for launchbadge/sqlx#1147 Draining statement results is no longer needed. * Make clippy happy * Do not pass NULL to mime_headers column nor expect it from there. Treat existing NULL as "empty string". * Increase SQL busy timeout to 100 seconds No need to abort the connection if all connections are stalled for 10 seconds. This did not happen, just to make sure each timeout is a result of an actual deadlock. * README: specify log target to get rid of sqlx query logging * Remove 3 unwrap()'s * Remove two more unwrap()'s Co-authored-by: B. Petersen <r10s@b44t.com> Co-authored-by: Hocuri <hocuri@gmx.de> Co-authored-by: link2xt <link2xt@testrun.org>
|
I have debugged this further. The problem seems to be in the I found a previous issue #662 exactly about Same thing happens in I made a fix and will prepare a similar test case to #912, but with |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Starting to read the
fetchresult and then dropping the stream instead of reading to the end leaves the connection in some broken state when it does not see any changed made to the database anymore, as if someone started a transaction withBEGINand did not finalize it. Maybe this is what happens when you start stepping through sqlite statement and don't finish it.Here is the code which reproduces the bug:See the second post for shorter version.If
PRAGMAresult (can be any other query probably) is not read to the end, reader connection ends up in a broken state where it does not see the data written on the connection from the writer pool.Corresponding
cargo.tomlthat I used:cargo.lockfor reference:The bug was encountered here: deltachat/deltachat-core-rust#2089
The architecture with two pools comes from #459.
I managed to run
DATABASE_URL="sqlite:$PWD/tests/sqlite/sqlite.db" cargo test --features any,macros,migrate,sqlite,all-types,runtime-async-std-native-tls --test sqlitebut don't see how to easily integrate this test into test suite. Maybe it will even be closed with WONTFIX, so I did not try yet. If it is a bug and not a misuse of the API on our side, then it's of course worth integrating into regression test suite.The text was updated successfully, but these errors were encountered: