[#21] Count double-width Asian characters as two characters for line-…

…breaking purposes.

Closes #21

RELEASE-NOTES.md

@@ -3,15 +3,18 @@
 # <a name="4.0.0-alpha-2"></a> Picocli 4.0.0-alpha-2 (UNRELEASED)
 The picocli community is pleased to announce picocli 4.0.0-alpha-2.
 
-This release contains bugfixes for argument groups. See the [4.0.0-alpha-1 New and Noteworthy section](#4.0.0-alpha-1-new) below for more details on argument groups.
+Thanks to great feedback from the picocli community on the 4.0.0-alpha-1 release, this release contains many argument group-related bugfixes.
+See the [4.0.0-alpha-1 New and Noteworthy section](#4.0.0-alpha-1-new) below for more details on argument groups.
 
 _Please try this and provide feedback. We can still make changes._
 
 _What do you think of the annotations API? What about the programmatic API? Does it work as expected? Are the input validation error messages correct and clear? Is the documentation clear and complete? Anything you want to change or improve? Any other feedback?_
 
-Many thanks to the picocli community members who contributed!
+This release also has support for variable expansion and improved support for Chinese, Japanese and Korean.
 
-This is the fifty-first public release.
+Many thanks to the picocli community for the contributions!
+
+This is the fifty-third public release.
 Picocli follows [semantic versioning](http://semver.org/).
 
 ## <a name="4.0.0-alpha-2"></a> Table of Contents
@@ -22,15 +25,27 @@ Picocli follows [semantic versioning](http://semver.org/).
 
 ## <a name="4.0.0-alpha-2-new"></a> New and Noteworthy
 
+### Variable Expansion
+
+TODO
+
+### Improved Support for Chinese, Japanese and Korean
+Picocli will align the usage help message to fit within some user-defined width (80 columns by default).
+A number of characters in Chinese, Japanese and Korean (CJK) are wider than others.
+If those characters are treated to have the same width as other characters, the usage help message may extend past the right margin.
+
+From this release, picocli will use 2 columns for these wide characters when calculating where to put line breaks, resulting in better usage help message text.
+
 
 ## <a name="4.0.0-alpha-2-fixes"></a> Fixed issues
+- [#21] Count double-width Asian characters as two characters for line-breaking purposes.
 - [#660] Added `@java.lang.annotation.Inherited` to the `@picocli.CommandLine.Command` annotation. Thanks to [Devin Smith](https://github.com/devinrsmith) for the suggestion.
-- [#661] Bugfix for stack overflow when option in an `@ArgGroup` had a default value. Thanks to [Andreas Deininger](https://github.com/deining) for reporting this.
-- [#656] Bugfix for issue where synopsis for composite groups did not expand for n..* (n > 1). Thanks to Arno Tuomainen for finding this issue.
+- [#661] Bugfix for stack overflow when option in an argument group had a default value. Thanks to [Andreas Deininger](https://github.com/deining) for reporting this.
+- [#656] Bugfix for issue where synopsis for composite argument groups did not expand for n..* (n > 1). Thanks to Arno Tuomainen for finding this issue.
 - [#654] Bugfix: argument group heading text was not retrieved from ResourceBundle. Thanks to [Andreas Deininger](https://github.com/deining) for raising this.
-- [#635] Bugfix in validation: did not show an error if some but not all parts of a co-occurring group were specified. Thanks to [Philipp Hanslovsky](https://github.com/hanslovsky) for the pull request. 
-- [#653] Bugfix: validation should be skipped if help was requested. Thanks to [Andreas Deininger](https://github.com/deining) for raising this.
-- [#655] Bugfix: ArgGroup validation silently accepts missing subgroup with multiplicity=1.
+- [#635] Bugfix in argument group validation: did not show an error if some but not all parts of a co-occurring group were specified. Thanks to [Philipp Hanslovsky](https://github.com/hanslovsky) for the pull request. 
+- [#653] Bugfix: argument group validation should be skipped if help was requested. Thanks to [Andreas Deininger](https://github.com/deining) for raising this.
+- [#655] Bugfix: argument group validation silently accepts missing subgroup with multiplicity=1.
 - [#652] Documentation: fixes in user manual. Thanks to [Andreas Deininger](https://github.com/deining) for the pull request.
 - [#651] Documentation: fixes in user manual. Thanks to [Andreas Deininger](https://github.com/deining) for the pull request.
 

src/main/java/picocli/CommandLine.java

@@ -11841,31 +11841,76 @@ public Cell putValue(int row, int col, Text value) {
                 throw new IllegalStateException(column.overflow.toString());
             }
             private static int length(Text str) {
-                return str.length; // TODO count some characters as double length
+                return length(str, str.from, str.length);
+            }
+            private static int length(Text str, int from, int length) {
+                int result = 0;
+                for (int i = from; i < str.from + length; i++) {
+                    result += isCharCJK(str.plain.charAt(i)) ? 2 : 1;
+                }
+                return result;
             }
 
+            /**
+             * Given a character, is this character considered to be a CJK character?
+             * Shamelessly stolen from
+             * <a href="http://stackoverflow.com/questions/1499804/how-can-i-detect-japanese-text-in-a-java-string">StackOverflow</a>
+             * where it was contributed by user Rakesh N. (Upvote! :-) )
+             * @param c Character to test
+             * @return {@code true} if the character is a CJK character
+             */
+            static boolean isCharCJK(char c) {
+                Character.UnicodeBlock unicodeBlock = Character.UnicodeBlock.of(c);
+                return (unicodeBlock == Character.UnicodeBlock.HIRAGANA)
+                        || (unicodeBlock == Character.UnicodeBlock.KATAKANA)
+                        || (unicodeBlock == Character.UnicodeBlock.KATAKANA_PHONETIC_EXTENSIONS)
+                        || (unicodeBlock == Character.UnicodeBlock.HANGUL_COMPATIBILITY_JAMO)
+                        || (unicodeBlock == Character.UnicodeBlock.HANGUL_JAMO)
+                        || (unicodeBlock == Character.UnicodeBlock.HANGUL_SYLLABLES)
+                        || (unicodeBlock == Character.UnicodeBlock.CJK_UNIFIED_IDEOGRAPHS)
+                        || (unicodeBlock == Character.UnicodeBlock.CJK_UNIFIED_IDEOGRAPHS_EXTENSION_A)
+                        || (unicodeBlock == Character.UnicodeBlock.CJK_UNIFIED_IDEOGRAPHS_EXTENSION_B)
+                        || (unicodeBlock == Character.UnicodeBlock.CJK_COMPATIBILITY_FORMS)
+                        || (unicodeBlock == Character.UnicodeBlock.CJK_COMPATIBILITY_IDEOGRAPHS)
+                        || (unicodeBlock == Character.UnicodeBlock.CJK_RADICALS_SUPPLEMENT)
+                        || (unicodeBlock == Character.UnicodeBlock.CJK_SYMBOLS_AND_PUNCTUATION)
+                        || (unicodeBlock == Character.UnicodeBlock.ENCLOSED_CJK_LETTERS_AND_MONTHS)
+                        //The magic number here is the separating index between full-width and half-width
+                        || (unicodeBlock == Character.UnicodeBlock.HALFWIDTH_AND_FULLWIDTH_FORMS && c < 0xFF61);
+            }
+
+            static class Count {
+                int charCount;
+                int columnCount;
+            }
             private int copy(BreakIterator line, Text text, Text columnValue, int offset) {
                 // Deceive the BreakIterator to ensure no line breaks after '-' character
                 line.setText(text.plainString().replace("-", "\u00ff"));
-                int done = 0;
+                Count count = new Count();
                 for (int start = line.first(), end = line.next(); end != BreakIterator.DONE; start = end, end = line.next()) {
                     Text word = text.substring(start, end); //.replace("\u00ff", "-"); // not needed
-                    if (columnValue.maxLength >= offset + done + length(word)) {
-                        done += copy(word, columnValue, offset + done); // TODO messages length
+                    if (columnValue.maxLength >= offset + count.columnCount + length(word)) {
+                        copy(word, columnValue, offset + count.charCount, count);
                     } else {
                         break;
                     }
                 }
-                if (done == 0 && length(text) + offset > columnValue.maxLength) {
+                if (count.charCount == 0 && length(text) + offset > columnValue.maxLength) {
                     // The value is a single word that is too big to be written to the column. Write as much as we can.
-                    done = copy(text, columnValue, offset);
+                    copy(text, columnValue, offset, count);
                 }
-                return done;
+                return count.charCount;
             }
             private static int copy(Text value, Text destination, int offset) {
+                Count count = new Count();
+                copy(value, destination, offset, count);
+                return count.charCount;
+            }
+            private static void copy(Text value, Text destination, int offset, Count count) {
                 int length = Math.min(value.length, destination.maxLength - offset);
                 value.getStyledChars(value.from, length, destination, offset);
-                return length;
+                count.columnCount += length(value, value.from, length);
+                count.charCount += length;
             }
 
             /** Copies the text representation that we built up from the options into the specified StringBuilder.

src/test/java/picocli/CommandLineHelpTest.java

@@ -3795,4 +3795,43 @@ class App {
                         "Using raw String: '%n' format strings have not been replaced with newlines. " +
                         "Please ensure to escape '%' characters with another '%'."));
     }
+
+    @Test
+    public void testCharCJKDoubleWidthText() {
+        @Command(name = "cjk", mixinStandardHelpOptions = true, description = {
+                "12345678901234567890123456789012345678901234567890123456789012345678901234567890",
+                "CUI\u306fGUI\u3068\u6bd4\u3079\u3066\u76f4\u611f\u7684\u306a\u64cd\u4f5c\u304c\u3067\u304d\u306a\u3044\u306a\u3069\u306e\u6b20\u70b9\u304c\u3042\u308b\u3082\u306e\u306e\u3001\u30e1\u30a4\u30f3\u30e1\u30e2\u30ea\u304a\u3088\u3073\u30d3\u30c7\u30aa\u30e1\u30e2\u30ea\u306a\u3069\u306e\u30b3\u30f3\u30d4\u30e5\u30fc\u30bf\u8cc7\u6e90\u306e\u6d88\u8cbb\u304c\u5c11\u306a\u3044\u3053\u3068\u304b\u3089\u3001" +
+                        "\u6027\u80fd\u306e\u4f4e\u3044\u521d\u671f\u306e\u30b3\u30f3\u30d4\u30e5\u30fc\u30bf\u3067\u306fCUI\u306b\u3088\u308b\u5bfe\u8a71\u74b0\u5883\u304c\u4e3b\u6d41\u3060\u3063\u305f\u3002\u305d\u306e\u5f8c\u3001\u30b3\u30f3\u30d4\u30e5\u30fc\u30bf\u306e\u6027\u80fd\u304c\u5411\u4e0a\u3057\u3001\u30de\u30a6\u30b9\u306a\u3069\u306e\u30dd\u30a4\u30f3\u30c6\u30a3\u30f3\u30b0\u30c7\u30d0\u30a4\u30b9\u306b\u3088\u3063\u3066" +
+                        "\u76f4\u611f\u7684\u306a\u64cd\u4f5c\u306e\u3067\u304d\u308bGUI\u74b0\u5883\u304cMacintosh\u3084Windows 95\u306a\u3069\u306b\u3088\u3063\u3066\u30aa\u30d5\u30a3\u30b9\u3084\u4e00\u822c\u5bb6\u5ead\u306b\u3082\u666e\u53ca\u3057\u3001" +
+                        "CUI\u306f\u65e2\u5b9a\u306e\u30a4\u30f3\u30bf\u30fc\u30d5\u30a7\u30a4\u30b9\u3067\u306f\u306a\u304f\u306a\u3063\u3066\u3044\u3063\u305f\u3002\u30d1\u30fc\u30bd\u30ca\u30eb\u30b3\u30f3\u30d4\u30e5\u30fc\u30bf (PC) \u5411\u3051\u3084\u30b5\u30fc\u30d0\u30fc\u5411\u3051\u306e\u30aa\u30da\u30ec\u30fc\u30c6\u30a3\u30f3\u30b0\u30b7\u30b9\u30c6\u30e0 (OS) " +
+                        "\u306b\u306f\u3001\u65e2\u5b9a\u306e\u30a4\u30f3\u30bf\u30fc\u30d5\u30a7\u30a4\u30b9\u304cGUI\u3067\u3042\u3063\u3066\u3082\u30b3\u30de\u30f3\u30c9\u30e9\u30a4\u30f3\u30bf\u30fc\u30df\u30ca\u30eb\u306a\u3069\u306eCUI\u74b0\u5883\u304c\u4f9d\u7136\u3068\u3057\u3066\u7528\u610f\u3055\u308c\u3066\u3044\u308b\u304c\u3001\u30b9\u30de\u30fc\u30c8\u30d5\u30a9\u30f3\u306a\u3069\u306e\u30e2\u30d0\u30a4\u30eb" +
+                        "\u7aef\u672b\u5411\u3051OS\u306b\u306f\u6a19\u6e96\u3067\u7528\u610f\u3055\u308c\u3066\u304a\u3089\u305a\u3001GUI\u304c\u4e3b\u6d41\u3068\u306a\u3063\u3066\u3044\u308b\u3002"})
+        class App {
+            @Option(names = {"-x", "--long"}, description = "\u3057\u304b\u3057\u3001GUI\u74b0\u5883\u3067\u3042\u308c\u3070\u30dd\u30a4\u30f3\u30c6\u30a3\u30f3\u30b0\u30c7\u30d0\u30a4\u30b9\u306b\u3088\u308b\u64cd\u4f5c\u3092\u7e70\u308a\u8fd4\u3057\u884c\u308f\u306a\u3051\u308c\u3070\u306a\u3089\u306a\u3044\u3088\u3046\u306a\u7169\u96d1\u306a\u4f5c\u696d\u3092\u3001CUI\u74b0\u5883\u3067\u3042\u308c\u3070\u7c21\u5358\u306a\u30b3\u30de\u30f3\u30c9\u306e\u7d44\u307f\u5408\u308f\u305b\u3067\u5b9f\u884c\u3067\u304d\u308b\u5834\u5408\u3082\u3042\u308b\u306a\u3069\u3001CUI\u306b\u3082\u4f9d\u7136\u3068\u3057\u3066\u5229\u70b9\u306f\u3042\u308b\u3002\u7279\u306b\u30a8\u30f3\u30c9\u30e6\u30fc\u30b6\u30fc\u3060\u3051\u3067\u306a\u304f\u30cd\u30c3\u30c8\u30ef\u30fc\u30af\u7ba1\u7406\u8005\u3084\u30bd\u30d5\u30c8\u30a6\u30a7\u30a2\u958b\u767a\u8005\u3082\u5229\u7528\u3059\u308bPC\u3084\u30b5\u30fc\u30d0\u30fc\u306b\u304a\u3044\u3066\u306f\u3001GUI\u3068CUI\u306f\u7528\u9014\u306b\u3088\u3063\u3066\u4f4f\u307f\u5206\u3051\u3066\u4f75\u5b58\u3057\u3066\u3044\u308b\u3082\u306e\u3067\u3042\u308a\u3001CUI\u306f\u5b8c\u5168\u306b\u5ec3\u308c\u305f\u308f\u3051\u3067\u306f\u306a\u3044\u3002")
+            int x;
+        }
+        String usageMessage = new CommandLine(new App()).getUsageMessage();
+        String expected = String.format("" +
+                "Usage: cjk [-hV] [-x=<x>]%n" +
+                "12345678901234567890123456789012345678901234567890123456789012345678901234567890%n" +
+                "CUI\u306fGUI\u3068\u6bd4\u3079\u3066\u76f4\u611f\u7684\u306a\u64cd\u4f5c\u304c\u3067\u304d\u306a\u3044\u306a\u3069\u306e\u6b20\u70b9\u304c\u3042\u308b\u3082\u306e\u306e\u3001\u30e1\u30a4\u30f3\u30e1\u30e2\u30ea\u304a\u3088\u3073%n" +
+                "\u30d3\u30c7\u30aa\u30e1\u30e2\u30ea\u306a\u3069\u306e\u30b3\u30f3\u30d4\u30e5\u30fc\u30bf\u8cc7\u6e90\u306e\u6d88\u8cbb\u304c\u5c11\u306a\u3044\u3053\u3068\u304b\u3089\u3001\u6027\u80fd\u306e\u4f4e\u3044\u521d\u671f\u306e\u30b3\u30f3%n" +
+                "\u30d4\u30e5\u30fc\u30bf\u3067\u306fCUI\u306b\u3088\u308b\u5bfe\u8a71\u74b0\u5883\u304c\u4e3b\u6d41\u3060\u3063\u305f\u3002\u305d\u306e\u5f8c\u3001\u30b3\u30f3\u30d4\u30e5\u30fc\u30bf\u306e\u6027\u80fd\u304c\u5411\u4e0a\u3057\u3001%n" +
+                "\u30de\u30a6\u30b9\u306a\u3069\u306e\u30dd\u30a4\u30f3\u30c6\u30a3\u30f3\u30b0\u30c7\u30d0\u30a4\u30b9\u306b\u3088\u3063\u3066\u76f4\u611f\u7684\u306a\u64cd\u4f5c\u306e\u3067\u304d\u308bGUI\u74b0\u5883\u304cMacintosh%n" +
+                "\u3084Windows 95\u306a\u3069\u306b\u3088\u3063\u3066\u30aa\u30d5\u30a3\u30b9\u3084\u4e00\u822c\u5bb6\u5ead\u306b\u3082\u666e\u53ca\u3057\u3001CUI\u306f\u65e2\u5b9a\u306e\u30a4\u30f3\u30bf\u30fc\u30d5\u30a7\u30a4%n" +
+                "\u30b9\u3067\u306f\u306a\u304f\u306a\u3063\u3066\u3044\u3063\u305f\u3002\u30d1\u30fc\u30bd\u30ca\u30eb\u30b3\u30f3\u30d4\u30e5\u30fc\u30bf (PC) \u5411\u3051\u3084\u30b5\u30fc\u30d0\u30fc\u5411\u3051\u306e\u30aa\u30da\u30ec\u30fc%n" +
+                "\u30c6\u30a3\u30f3\u30b0\u30b7\u30b9\u30c6\u30e0 (OS) \u306b\u306f\u3001\u65e2\u5b9a\u306e\u30a4\u30f3\u30bf\u30fc\u30d5\u30a7\u30a4\u30b9\u304cGUI\u3067\u3042\u3063\u3066\u3082\u30b3\u30de\u30f3\u30c9\u30e9\u30a4\u30f3%n" +
+                "\u30bf\u30fc\u30df\u30ca\u30eb\u306a\u3069\u306eCUI\u74b0\u5883\u304c\u4f9d\u7136\u3068\u3057\u3066\u7528\u610f\u3055\u308c\u3066\u3044\u308b\u304c\u3001\u30b9\u30de\u30fc\u30c8\u30d5\u30a9\u30f3\u306a\u3069\u306e\u30e2\u30d0\u30a4%n" +
+                "\u30eb\u7aef\u672b\u5411\u3051OS\u306b\u306f\u6a19\u6e96\u3067\u7528\u610f\u3055\u308c\u3066\u304a\u3089\u305a\u3001GUI\u304c\u4e3b\u6d41\u3068\u306a\u3063\u3066\u3044\u308b\u3002%n" +
+                "  -h, --help       Show this help message and exit.%n" +
+                "  -V, --version    Print version information and exit.%n" +
+                "  -x, --long=<x>   \u3057\u304b\u3057\u3001GUI\u74b0\u5883\u3067\u3042\u308c\u3070\u30dd\u30a4\u30f3\u30c6\u30a3\u30f3\u30b0\u30c7\u30d0\u30a4\u30b9\u306b\u3088\u308b\u64cd\u4f5c\u3092\u7e70\u308a\u8fd4\u3057%n" +
+                "                     \u884c\u308f\u306a\u3051\u308c\u3070\u306a\u3089\u306a\u3044\u3088\u3046\u306a\u7169\u96d1\u306a\u4f5c\u696d\u3092\u3001CUI\u74b0\u5883\u3067\u3042\u308c\u3070\u7c21\u5358\u306a\u30b3%n" +
+                "                     \u30de\u30f3\u30c9\u306e\u7d44\u307f\u5408\u308f\u305b\u3067\u5b9f\u884c\u3067\u304d\u308b\u5834\u5408\u3082\u3042\u308b\u306a\u3069\u3001CUI\u306b\u3082\u4f9d\u7136\u3068\u3057\u3066%n" +
+                "                     \u5229\u70b9\u306f\u3042\u308b\u3002\u7279\u306b\u30a8\u30f3\u30c9\u30e6\u30fc\u30b6\u30fc\u3060\u3051\u3067\u306a\u304f\u30cd\u30c3\u30c8\u30ef\u30fc\u30af\u7ba1\u7406\u8005\u3084\u30bd\u30d5%n" +
+                "                     \u30c8\u30a6\u30a7\u30a2\u958b\u767a\u8005\u3082\u5229\u7528\u3059\u308bPC\u3084\u30b5\u30fc\u30d0\u30fc\u306b\u304a\u3044\u3066\u306f\u3001GUI\u3068CUI\u306f\u7528\u9014\u306b%n" +
+                "                     \u3088\u3063\u3066\u4f4f\u307f\u5206\u3051\u3066\u4f75\u5b58\u3057\u3066\u3044\u308b\u3082\u306e\u3067\u3042\u308a\u3001CUI\u306f\u5b8c\u5168\u306b\u5ec3\u308c\u305f\u308f\u3051\u3067%n" +
+                "                     \u306f\u306a\u3044\u3002%n");
+        assertEquals(expected, usageMessage);
+    }
 }