Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Line separator extended character causes query to silently fail and prevents further queries on iOS #147

Closed
subblue opened this issue Oct 17, 2014 · 15 comments

Comments

@subblue
Copy link

subblue commented Oct 17, 2014

I'm using a prepopulated sqlite database in our iOS app exported from a separate system.
I've found that for some queries the database would never return a result or even error and leave the app in a state where it couldn't recover to make any further queries.

Digging into the entries that were failing I found that the content (which was a block of JSON stored as a string in a TEXT field) contained extended characters such as the unicode line separator U+2028.
I'm guessing that at the point where the content from the database is converted to a form to send to the embedded browser it is tripping up on this. It would be good if some sort of check could be added in the plugin to warn of potential issues extracting the data.

Anyway, in case others have similar weird issues this is worth checking for. I've now added an extra stage in the sqlite.db creation script to remove these extended characters.

@brodycj
Copy link
Contributor

brodycj commented Oct 30, 2014

This issue definitely needs investigation (on all three platforms Android/iOS/WP(8)), and hopefully I will reproduce this (or a similar) issue with the \u2028 character in a unit test.

@brodycj
Copy link
Contributor

brodycj commented Feb 28, 2015

This scenario is now reproduced for the iOS version. The test causes a silent failure (no success or error callback) for WP(8) version. This is not an issue for the Android version.

@brodycj
Copy link
Contributor

brodycj commented Mar 5, 2015

From some further testing I now suspect the Objective-C JSON encoding of the query results.

brodycj pushed a commit that referenced this issue Mar 10, 2015
Fixes:
- #193: workaround for Android db locking/closing issue
- #144: convert array parameters to string to match Web SQL
- #199: fix double-precision REAL values in result for iOS version
- #150/#153: close Android db before removing from map
- Fix truncation in iOS query result in case of UNICODE NULL (\0 or \u0000) (ref: PR #170)
- Some fixes for error handling to be consistent with Web SQL (ref: PR #170)

Testing ONLY:
- #147: testing with UNICODE line separator
- #195: Reproduce issue with double-precision REAL number on WP(8) ONLY
brodycj pushed a commit that referenced this issue Mar 10, 2015
Fixes:
- #193: workaround for Android db locking/closing issue
- #144: convert array parameters to string to match Web SQL
- #199: fix double-precision REAL values in result for iOS version
- #150/#153: close Android db before removing from map
- Fix truncation in iOS query result in case of UNICODE NULL (\0 or \u0000) (ref: PR #170)
- Some fixes for error handling to be consistent with Web SQL (ref: PR #170)

Testing ONLY:
- #147: testing with UNICODE line separator
- #195: Reproduce issue with double-precision REAL number on WP(8) ONLY
@brodycj
Copy link
Contributor

brodycj commented May 21, 2015

I just discovered this article: http://timelessrepo.com/json-isnt-a-javascript-subset that helps explain the problem with the UNICODE line separator character. Suspect this bug is with cordova-ios.

@brodycj brodycj added the testing label Jul 7, 2015
@brodycj
Copy link
Contributor

brodycj commented Jul 7, 2015

Also need to test UNICODE \u2029 - paragraph separator

UPDATE: When testing the following code against cordova-ext-echo:

        var Echo = cordova.require('org.apache.cordova.plugins.echo.Echo');
        Echo.echo('first\u2027second', function(v) {
          console.log('cb 1: ' + v);
        });
        Echo.echo('third\u2028fourth', function(v) {
          console.log('cb 2: ' + v);
        });
        Echo.echo('fifth\u2029six', function(v) {
          console.log('cb 3: ' + v);
        });

only shows the following output:

cb 1: first‧second

Definitely a bug in cordova-ios.

@brodycj
Copy link
Contributor

brodycj commented Jul 29, 2015

In addition, this can be an issue when using lz-string to compress data, as described here: https://groups.google.com/d/msgid/Cordova-SQLitePlugin/5a31f7d7-143f-45fe-979f-e2ec5a02f44d%40googlegroups.com

Original description:

I found a weird issue reading a string encoded in UTF16.

I'm using lzstring to compress big XML blocks as json elements and store it
in the db. The insert or update works file, but when I retrieve the record,
the plugin crash immediatly after the cordova.js call

return iOSExec.nativeFetchMessages();

returning a

SyntaxError: Unexpected EOF.

This is caused by the unicode char E2 80 A9.

The following workaround was given:

I fixed the problem by replacing all occurrences of both characters after compression and before storage, then after retrieval and before decompression. Something like this...
before storage:

var dbvalue = LZString.compressToUTF16(value);
dbvalue = dbvalue.replace(/\u2028/g, '\\u2028').replace(/\u2029/g, '\\u2029');

after retrieval

var utfdata = dbvalue.replace(/\\u2028/g,'\u2028').replace(/\\u2029/g,'\u2029');
var value = LZString.decompressFromUTF16(utfdata);

But it has two problems:

  • It could be more efficient (see (#2890) - websql: avoid hex() for binaries pouchdb/pouchdb#2900 to see how they deal with a similar problem with \0 characters in a "binary" string)
  • In case the characters \ u 2 0 2 8 or \ u 2 0 2 9 would ever show up in the output of lz-string, they would be replaced after retrieval and then LZString.decompressFromUTF16(utfdata) would not give you the original string value.

Here is a corrected version which should work properly (if I am not mistaken):

before storage:

var dbvalue = LZString.compressToUTF16(value);
dbvalue = dbvalue
  .replace(/\u007f/g, '\u007f\u007f')
  .replace(/\u2028/g, '\u007f\u0028').replace(/\u2029/g, '\u007f\u0029');

after retrieval

var utfdata = dbvalue
  .replace(/\u007f\u0028/g,'\u2028').replace(/\u007f\u0029/g,'\u2029')
  .replace(/\u007f\u007f/g,'\u007f');
var value = LZString.decompressFromUTF16(utfdata);

(It should be possible to use any other "Basic Latin" (Unicode block) character in the range of \u0001 ... \u007e with the exception of \u0028 and \u0029 in place of \u007f, if necessary for some reason.)

Another alternative from pouchdb/pouchdb#2900 (comment):

COBS would be more efficient for preventing embedded nulls. Use a 16-bit version of it, if a null is a 0000.

Some references:

@brodycj
Copy link
Contributor

brodycj commented Jul 31, 2015

I just filed Apache Cordova CB-9435 to report the problem with U+2028 and U+2029 on cordova-ios.

@assyme
Copy link

assyme commented Dec 5, 2015

Hi,

I am currently struggling with the same problem and unfortunately, cordova has asked to find a work around in CB-9435. I tried replacing the U+0028 & U+0029 characters from my json string before inserting in the database, but the problem still persists. I am not too comfortable with Objective C and unsure what i need to do in the plugin side to escape these characters. Will highly appreciate if you could guide me on this one.

@brodycj
Copy link
Contributor

brodycj commented Feb 14, 2016

This is solved in litehelpers / Cordova-sqlite-enterprise-free, available under a different licensing scheme.

@brodycj brodycj closed this as completed Feb 14, 2016
@CookieCookson
Copy link

@brodybits There are other characters which also make this plugin fail on open. I have had issues with the following character:

http://www.fileformat.info/info/unicode/char/2019/index.htm

@brodycj
Copy link
Contributor

brodycj commented Apr 8, 2016

Thanks @CookieCookson! I will document this and add it to the test for the other platforms when I get a chance.

@brodycj
Copy link
Contributor

brodycj commented Apr 10, 2016

@brodybits There are other characters which also make this plugin fail on open. I have had issues with the following character:

http://www.fileformat.info/info/unicode/char/2019/index.htm

I raised #459 with this since it seems to be quite different from the OP.

FYI a normal string test with the \u2019 (as well as \u2018) works OK on Android.

@brodycj
Copy link
Contributor

brodycj commented Nov 18, 2016

FYI this is also an issue with cordova-android 6.0.0 which is currently installed by the latest Cordova CLI (6.4.0), raised in apache/cordova-discuss#57 for discussion.

@GeeKanJi
Copy link

GeeKanJi commented Dec 7, 2016

Hi,

Is there a workaround?

@GeeKanJi
Copy link

GeeKanJi commented Dec 7, 2016

workaround:
WP-API/WP-API#2923

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

5 participants