Exceptions

[maxhaton]

Exceptions are overused at the moment however they are not particularly inefficient when used for genuinely exceptional circumstances (if inefficient at all by some measure i.e. the runtime cost can easily be nil)

[LightBender]

I vaguely remember that I got Walter to admit that this is more aspirational and not a hard rule. I can't see how we can realistically do everything we need to do without them so we can use, we just need to be judicious about it. I should probably confirm with @WalterBright and open a PR for it.

[pbackus]

For some interfaces, it might make sense to provide two versions: one that uses exceptions, and one that doesn't.

An example of this in current Phobos is match and tryMatch in std.sumtype.

[jmdavis]

std.conv very much falls in that camp. For years, I've thought that we should have something like tryTo which returns a Nullable rather than throwing so that you can choose which you want to do, but I've never taken the time to implement it.

[Herringway]

While I am generally in favour of exceptions, std.utf.validate is very much using them badly. That really should have just been a function that returns a bool...

[Herringway]

std.utf.validate does not do that, and it is not often useful for it to do so. Non-unicode text is not exceptional, unfortunately. All that extra information does not tell the user what the text is actually encoded with (it couldn't, even if it wanted to) and is just noise that gets in the way when falling back to a non-UTF encoding.

An alternate function can provide that information if it's desired. Call it enforceUTF or something. Or better yet, assertUTF since non-UTF strings are supposed to be illegal anyway.

[jmdavis]

validate clearly needs to go. I'm pretty sure that the reason that it even exists over something like isValidUTF is because of how decode is implemented. It originally just operated on arrays and threw a UTFException, and making that work with a function that returns bool requires some series rework to the implementation - or code duplication. The code was reworked at some point to add the option to optionally use the replacement character instead of throwing, and it was reworked to work with ranges, but that's just made it even more of a mess due to what it's forced to do to avoid code duplication. And when I tried at one point to write isValidUTF, Andrei shot it down on the grounds that the implementation was too complicated.

As it stands, it's simple to write a function that just uses UseReplacementDchar.yes with decode and return bool if it encounters the replacement character, but then you have the problem of whether the replacement character was in the original input, or whether it just replaced an invalid code point. As I understand it, the replacement character itself is valid Unicode, but it's only ever used if there was invalid Unicode in the data at some point, so whether it should be treated as invalid is debatable. Probably, it shouldn't be, but treating it as such is the only quick and easy way to implement something like isValidUTF.

So, the reason that we have a function throwing an exception rather than returning a bool very much comes down to historical and implementation details, and it's clearly something that should be fixed and not how exceptions should be used (if anything, it's the poster child for how not to use exceptions). Regardless, we'll need to figure out how to rework std.utf in Phobos v3 to solve the problem.

[Herringway]

Perhaps building it all around a function like T[] getValidUTF(T[]); that simply returns the slice of the input that's valid is the way to go. Then a theoretical isValid function is as simple as

bool isValid(T)(T[] input) {
  return getValidUTF(input).length == input.length;
}

and a theoretical enforceUTF is as simple as

void enforceUTF(T)(T[] input) {
  doSomeThrowyDecoding(input[getValidUTF(input).length .. $]);
}

The latter function could require two attempts at decoding any offending text, but that cost will be minimal compared to the exception cost anyways.

[jmdavis]

Honestly, I'm not sure what use that would have. If you're decoding strings, what you'd typically want to do (and I think that the standard might even require it) would be either

1. Convert all invalid code points to the replacement character so that the program is able to operate on the data just fine in spite of not all of it being valid.
2. Throw an exception, because there were invalid code points in the data, and your program doesn't want to ignore that.

std.utf.decode actually does these both just fine right now. The main problem is that the default is throwing rather than using the replacement character, and that's what auto-decoding then does.

If you're not decoding the string as part of what your program does to do its job, but you want to be sure that you have valid data, then you need something like validate or isValidUTF which iterates through the entire string and makes sure that it doesn't contain invalid code points (though obviously returning a bool is better design than throwing an exception in a case like this). This can make good sense if you want to make sure that you're dealing with good input but then have the rest of the program not worry about whether the Unicode is valid or not - in which case, you would use the version of decode which uses the replacement character when it encounters an invalid code point, but you wouldn't actually ever get the replacement character unless it was in the original input, or your program did some incorrect string manipulation, because you would have verified up front that the input was valid.

The problem implementation-wise with regards to isValidUTF just comes from how complicated decode is internally in order to be efficient with both strings and multiple categories of ranges and in order to be able to handle both using the replacement character or throwing depending on the template arguments. And making the current code return bool for validity rather than returning a dchar adds another layer to the complexity which arguably takes it to the point that it should just duplicate the core logic rather than trying to reuse the same code like decode currently tries to do for all of the combinations of what it currently does. Obviously, duplicating logic is not ideal, but the current implementation is pretty complex - though maybe if I tackled the problem again now, I'd be able to find a better way to make it work.

Trying to implement a function that skips the invalid code points would not simplify things at all. It would require a completely separate implementation, because decode does not provide that kind of functionality. And I have no clue what the use case for such a function would be. The replacement character already provides a standard way to get valid Unicode from invalid Unicode so that you can process bad data, and skipping the invalid code points so that the bad input just disappears just munges the strings even more. It's also an inefficient way to check for validity, since if you're checking whether the string is valid, you really want to stop iterating through the string as soon as you encounter an invalid code point.

In any case, I don't think that there's anyone arguing that we should have validate rather than isValidUTF in Phobos v3. It's just a question of figuring out how to do a clean implementation, which gets complicated if we want to avoid duplicating the decoding logic. And we'll need to do that one way or another.

[Herringway]

Honestly, I'm not sure what use that would have. If you're decoding strings, what you'd typically want to do (and I think that the standard might even require it) would be either

1. Convert all invalid code points to the replacement character so that the program is able to operate on the data just fine in spite of not all of it being valid.
2. Throw an exception, because there were invalid code points in the data, and your program doesn't want to ignore that.

3. Fall back to a different encoding.

Non-unicode text is unfortunately still very common. This is not an exceptional case. It's the only use case I've personally found for validate(), in fact.