BUG: Escape single quote in DICOM database SQL statement #881

cpinter · 2019-08-29T01:57:38Z

Re https://issues.slicer.org/view.php?id=4709

lassoan · 2019-08-29T02:02:15Z

Looks good, but why don't we do it with all the other fields? For example, physician's name can contain single-quote, too.

cpinter · 2019-08-29T02:06:58Z

It occurred to me but didn't seem likely even for the physician name. If you give me a list of fields you want me to escape the same way I'll add those.

lassoan · 2019-08-29T02:20:01Z

It is very common in person names (O'Leary...) and can easily occur in any description fields. Are there any other restricted characters? Does \ have to be encoded as \?

pieper · 2019-08-29T14:00:59Z

I don't think this is the source of the issue - the data was imported correctly in the first place, but the error happened when it converted to the new schema. Can you point to the commits involved in the database update?

cpinter · 2019-08-29T14:17:14Z

Sorry I had only a little time in the evening and didn't pay enough attention. I'll track this down now.

cpinter · 2019-08-29T14:18:31Z

Actually it would help a lot if I had that data and could debug into it. Without it it's just guesswork.

cpinter · 2019-08-29T14:24:22Z

Anyway, I did some guesswork and pushed a new change :) It escapes the single quote character in every field that is being processed during update.

pieper · 2019-08-29T16:13:52Z

Thanks for digging into this @cpinter

I still don't understand why this became an issue for converting when the data imported correctly before. Do you have any ideas on that? Without that it makes it hard to know if the fix is correct.

I can't share the exact dataset (PHI) and I haven't had time to check, but I can do more debugging/testing at some point as needed. For now I'd be happy just understanding what changed - can you point to the commits that handle the database update?

cpinter · 2019-08-29T16:23:30Z

I also don't understand how the initial insertion can be fine without having to sanitize, but then fails later.

This is the original commit
9c2af28
And a few follow-up commits - basically whatever I committed to CTK lately. This is quite a big change, and I'm not sure how reading it would help. If you have specific questions I'm happy to answer though.

pieper · 2019-08-29T19:05:30Z

Yes, I was looking through and it seems that this squashed commit has everything lumped together and I don't see what would have led to this issue.

Specifically though if you look at the error message it's in an UPDATE to the Studies table - do you know where that would happening?

 Bad SQL: UPDATE Studies SET DisplayedNumberOfSeries='3', InstitutionName='BRIGHAM & WOMEN'S HOSPITAL', PatientsUID=4, ReferringPhysician='x^x^^^', StudyDate='2019xxxx', StudyDescription='MRI BRAIN FUNCTIONAL', StudyInstanceUID='1.2.xx' WHERE StudyInstanceUID='1.2.x'

cpinter · 2019-08-29T19:55:09Z

This PR should fix the issue.

The updated values are set here: https://github.com/commontk/CTK/blob/master/Libs/DICOM/Core/ctkDICOMDatabase.cpp#L929
And then the update command is called here:
https://github.com/commontk/CTK/blob/master/Libs/DICOM/Core/ctkDICOMDatabase.cpp#L1066

cpinter · 2019-08-29T19:58:51Z

Maybe this PR is not enough after all. We need to make sure the displayed field generator does not override the original value with some problematic value. It's here:
https://github.com/commontk/CTK/blob/master/Libs/DICOM/Core/ctkDICOMDisplayedFieldGenerator.cpp#L116

pieper · 2019-08-29T20:07:55Z

Oh well yes, this is where the problem is: https://github.com/commontk/CTK/blob/master/Libs/DICOM/Core/ctkDICOMDatabase.cpp#L1066

This uses QString formatting to build the SQL query instead of the usual prepare and bind that is used elsewhere. It should be changed to match the rest of the QSqlQuery usage and the problem should go away.

pieper · 2019-08-29T20:10:31Z

Also line 1144 and anywhere else where sql statements are created.

cpinter · 2019-08-29T20:10:37Z

OK. I'll change the queries to use prepare everywhere in this class.

cpinter · 2019-08-29T20:11:29Z

Line 1144 seems to be just a closing brace...

pieper · 2019-08-29T20:14:28Z

Line 1144 seems to be just a closing brace...

Sorry, meant 1104, but you have the idea. 😄

I don't have the exact commit handy but I ran into this exact issue with the code when it was first written back seven or eight years ago when first testing it with data from Brigham and Women's Hospital and that's why we changed to using the prepare and bind calls.

cpinter · 2019-08-29T20:15:36Z

One question, maybe you know this by heart. The prepare/bind way just replaces strings, or is there something "more intelligent" going on there? Because the displayStudiesFieldUpdateList and its similar counterparts are longer strings involving multiple fields, and if it's just string replacement then it's fine, but if it somehow forces to bind values for fields, then it probably won't work like this.

cpinter · 2019-08-29T21:49:06Z

I updated the PR. It is not working yet (crashes), but it will be something like this. I have to go now though, sop I'll fix this tomorrow.

cpinter · 2019-08-29T23:48:07Z

This should do it. @pieper can you please test with your dataset?

pieper · 2019-09-01T21:57:41Z

Hmm, sorry, no, still not working. But it's closer. Now the error is:

SQL failed
 Bad SQL: UPDATE Studies SET DisplayedNumberOfSeries = ? , InstitutionName = ? , PatientsUID = ? , ReferringPhysician = ? , StudyDate = ? , StudyDescription = ? , StudyInstanceUID = ? StudyInstanceUID UID = ? ;
Error text:  Parameter count mismatch

pieper · 2019-09-01T22:07:00Z

Libs/DICOM/Core/ctkDICOMDatabase.cpp

-        QString("UPDATE Studies SET %1 WHERE StudyInstanceUID='%2';").arg(displayStudiesFieldUpdateList).arg(currentStudy["StudyInstanceUID"]);
-      this->loggedExec(updateDisplayStudyStatement, updateDisplayStudyStatementString);
+      QSqlQuery updateDisplayStudyStatement(this->Database);
+      updateDisplayStudyStatement.prepare( QString("UPDATE Studies SET %1 StudyInstanceUID UID = ? ;").arg(displayStudiesFieldUpdateString) );


Doesn't this sql statement need a WHERE clause?

pieper · 2019-09-01T22:08:12Z

I took a look but didn't see anything obvious here about the parameter counts, but maybe it's related to the comment I added in the code about a missing WHERE clause? We can chat again about it if you want to walk through the logic together.

cpinter · 2019-09-01T23:54:38Z

Interesting. I tried a simple import then update and it worked for me. Based on the error message, a WHERE seems to be missing. Not sure what's going on. Is it OK if I check it out on Tuesday?

pieper · 2019-09-01T23:58:43Z

Oh yes, Tuesday sounds great!

Re https://issues.slicer.org/view.php?id=4709

cpinter · 2019-09-02T00:00:47Z

I changed the statement. I think I accidentally pasted the StudyInstanceUID on the WHERE instead the UID... Not sure how it worked when I tried it. I hope it's going to be good now.

pieper · 2019-09-02T21:47:31Z

Tested and it looks good to me - thanks @cpinter! 👍

Addresses an issue when there's an apostrophe in a field name (e.g. Briham and Women's) See: commontk/CTK#881 $ git shortlog 6a8d584d3eb0ed750f6a850ab4f9c1089820f701..9e89c22d8599763bc20960d1b22175e465fac65c --no-merges Csaba Pinter (1): BUG: Escape single quote in DICOM database SQL statement Steve Pieper (1): STYLE: remove ^M characters From: Steve Pieper <pieper@isomics.com> git-svn-id: http://svn.slicer.org/Slicer4/trunk@28474 3bd1e089-480b-0410-8dfb-8563597acbee

pieper · 2019-09-02T22:10:28Z

Okay, I updated Slicer to use this, so we should be good to go. I'll try to check in the nightly build.

cpinter · 2019-09-03T12:42:34Z

Thank you very much! Sorry for the problems!

jcfr · 2019-09-03T12:55:38Z

🎉

Addresses an issue when there's an apostrophe in a field name (e.g. Briham and Women's) See: commontk/CTK#881 $ git shortlog 6a8d584d3eb0ed750f6a850ab4f9c1089820f701..9e89c22d8599763bc20960d1b22175e465fac65c --no-merges Csaba Pinter (1): BUG: Escape single quote in DICOM database SQL statement Steve Pieper (1): STYLE: remove ^M characters From: Steve Pieper <pieper@isomics.com> git-svn-id: http://svn.slicer.org/Slicer4/trunk@28474 3bd1e089-480b-0410-8dfb-8563597acbee

cpinter force-pushed the fix-dicom-database-sql-statement branch from db30c43 to ec9b9f1 Compare August 29, 2019 14:23

cpinter force-pushed the fix-dicom-database-sql-statement branch from ec9b9f1 to 729b1f7 Compare August 29, 2019 21:48

cpinter force-pushed the fix-dicom-database-sql-statement branch from 729b1f7 to f0ea905 Compare August 29, 2019 23:47

pieper reviewed Sep 1, 2019

View reviewed changes

BUG: Escape single quote in DICOM database SQL statement

ebd2b26

Re https://issues.slicer.org/view.php?id=4709

cpinter force-pushed the fix-dicom-database-sql-statement branch from f0ea905 to ebd2b26 Compare September 1, 2019 23:59

pieper merged commit be071a0 into commontk:master Sep 2, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

BUG: Escape single quote in DICOM database SQL statement #881

BUG: Escape single quote in DICOM database SQL statement #881

cpinter commented Aug 29, 2019

lassoan commented Aug 29, 2019

cpinter commented Aug 29, 2019

lassoan commented Aug 29, 2019

pieper commented Aug 29, 2019

cpinter commented Aug 29, 2019

cpinter commented Aug 29, 2019

cpinter commented Aug 29, 2019

pieper commented Aug 29, 2019

cpinter commented Aug 29, 2019

pieper commented Aug 29, 2019

cpinter commented Aug 29, 2019

cpinter commented Aug 29, 2019

pieper commented Aug 29, 2019

pieper commented Aug 29, 2019

cpinter commented Aug 29, 2019

cpinter commented Aug 29, 2019

pieper commented Aug 29, 2019

cpinter commented Aug 29, 2019

cpinter commented Aug 29, 2019

cpinter commented Aug 29, 2019

pieper commented Sep 1, 2019

pieper Sep 1, 2019

pieper commented Sep 1, 2019

cpinter commented Sep 1, 2019

pieper commented Sep 1, 2019

cpinter commented Sep 2, 2019

pieper commented Sep 2, 2019

pieper commented Sep 2, 2019

cpinter commented Sep 3, 2019

jcfr commented Sep 3, 2019

BUG: Escape single quote in DICOM database SQL statement #881

BUG: Escape single quote in DICOM database SQL statement #881

Conversation

cpinter commented Aug 29, 2019

lassoan commented Aug 29, 2019

cpinter commented Aug 29, 2019

lassoan commented Aug 29, 2019

pieper commented Aug 29, 2019

cpinter commented Aug 29, 2019

cpinter commented Aug 29, 2019

cpinter commented Aug 29, 2019

pieper commented Aug 29, 2019

cpinter commented Aug 29, 2019

pieper commented Aug 29, 2019

cpinter commented Aug 29, 2019

cpinter commented Aug 29, 2019

pieper commented Aug 29, 2019

pieper commented Aug 29, 2019

cpinter commented Aug 29, 2019

cpinter commented Aug 29, 2019

pieper commented Aug 29, 2019

cpinter commented Aug 29, 2019

cpinter commented Aug 29, 2019

cpinter commented Aug 29, 2019

pieper commented Sep 1, 2019

pieper Sep 1, 2019

Choose a reason for hiding this comment

pieper commented Sep 1, 2019

cpinter commented Sep 1, 2019

pieper commented Sep 1, 2019

cpinter commented Sep 2, 2019

pieper commented Sep 2, 2019

pieper commented Sep 2, 2019

cpinter commented Sep 3, 2019

jcfr commented Sep 3, 2019