Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
fix(mssql): dont yield from inside a cursor
Continuing in the efforts against this heisenbug. Previous fix didn't do the trick. We noticed two failure modes: 1. the LHS is wrong (expr.execute()) 2. the RHS is wrong (wrote the wrong number of rows to a file) both of these point, at their core, to some issue with the cursor, since the mssql backend relies on a SQLAlchemy cursor for feeding both `execute` and `to_pyarrow_batches` (which itself feeds `to_parquet` and `to_csv`). We found this: http://www.pymssql.org/en/stable/pymssql_examples.html#important-note-about-cursors which lets us know that we _cannot_ have multiple cursors on the same connection. Attempting to close the cursor after the `yield` loop doesn't work. So instead, for now, this terrible workaround. We fetch the entire contents, then re-batch it using `toolz`. We should noodle on this some more, but this is one possible "fix"
- Loading branch information