Fix #357 - invalid characters in path/query

[michalbundyra]

Fixes #357
Fixes #218

[michalbundyra]

@weierophinney I was looking for right solution of this issue, and I think I have found one finally.
In my opinion we need to do something with invalid characters - in that case encode - what I mean we can't just throw exception, as these can be user input.
So if invalid UTF-8 character is detected in the string, then I am checking the string letter by letter to encode that invalid character. This solution run regexp on one character in the loop, but I couldn't find anything better. Any suggestions appreciated.

/cc @YellowPepper @krowinski @freyr @oroszlanyzsolt

[weierophinney]

Noting here, because I had to look it up - str_split() splits into bytes instead of characters when it encounters a multibyte string.

[weierophinney]

I don't understand what this is doing, exactly.

I've tried it locally with a variety of strings from your query provider below, and every single one of them results in a positive match, which means the logic below never gets hit. Maybe you're not testing invalid UTF-8 characters anywhere?

[michalbundyra]

How come?

See: https://3v4l.org/TntQf

var_dump(preg_match('//u', "\x21\x92"));
var_dump(preg_match('//u', "\x21"));
var_dump(preg_match('//u', "\x92"));

result:

bool(false)
int(1)
bool(false)

Somehow tests I've added work.

[weierophinney]

Turned out I was using single quotes instead of double quotes when I was doing my tests here. Once I changed the quoting style, I was able to observe the same behaviour finally.

Might be good to drop a note in indicating what this is doing, though, so future contributors/maintainers know the purpose.

[krowinski]

tx for fix!