Unmappable character for encoding ASCII

[leliamesteban]

picocli won't compile

CommandLine.java:3491: error: unmappable character for encoding ASCII
         * The synopsis would show {@code --[no-]force} and {@code -XX:??PrintGCDetails}, respectively.
CommandLine.java:3491: error: unmappable character for encoding ASCII
         * The synopsis would show {@code --[no-]force} and {@code -XX:??PrintGCDetails}, respectively.

[remkop]

I've added the following to the build:

    compileJava.options.encoding = "UTF-8"
    compileTestJava.options.encoding = "UTF-8"

Thanks for raising this!

[remkop]

@leliamesteban Did this solve the problem?

[leliamesteban]

Unfortunately not. It fails with the same output:

> Task :javadoc FAILED
/home/liam/picocli/src/main/java/picocli/CommandLine.java:3491: error: unmappable character for encoding ASCII
         * The synopsis would show {@code --[no-]force} and {@code -XX:??PrintGCDetails}, respectively.
                                                                       ^
/home/liam/picocli/src/main/java/picocli/CommandLine.java:3491: error: unmappable character for encoding ASCII
         * The synopsis would show {@code --[no-]force} and {@code -XX:??PrintGCDetails}, respectively.

I'll show what I'm running so you can see if I'm doing something wrong

git show

commit 4f2c086a02243ac221a13579baaaa879874a14a7
Author: Remko Popma <remkop@yahoo.com>
Date:   Wed Jun 5 06:35:36 2019 +0900

    [#717] Fix unmappable character for encoding ASCII by setting compiler encoding to UTF8 explicitly.

diff --git a/RELEASE-NOTES.md b/RELEASE-NOTES.md
index 4da14308..f282c49d 100644
--- a/RELEASE-NOTES.md
+++ b/RELEASE-NOTES.md
@@ -217,6 +217,7 @@ If the negated form of the option is found, for example `--no-verbose`, the valu
 - [#694] API: `negatable` boolean options. Thanks to [Michael D. Adams](https://github.com/adamsmd) for the feature request.
 - [#712] Boolean options should not toggle by default, to be consistent with negatable options
 - [#709] Fix scrambled characters for the `<C2><B1>` character when running on system with non-UTF8 encoding
+- [#717] Fix unmappable character for encoding ASCII by setting compiler encoding to UTF8 explicitly. Thanks to [Liam Esteban Prince](https://github.com/leliamesteban) for raising this.
 - [#697] Option sort in usage help should ignore option name prefix; long options without short name should be inserted alphabetically, instead of always appear at the top.
 - [#695] Fix runtime warnings about illegal reflective access to field `java.io.FilterOutputStream.out`. Thanks to [gitfineon](https://github.com/gitfineon) for reporting this issue.
 - [#698] Reduce `reflect-config.json` used by GraalVM native-image generation
diff --git a/build.gradle b/build.gradle
index 1133671e..52d83473 100644
--- a/build.gradle
+++ b/build.gradle
@@ -35,6 +35,9 @@ allprojects {
     targetCompatibility = !org.gradle.api.JavaVersion.current().isJava9Compatible() ?
             1.5 : org.gradle.api.JavaVersion.current().isJava11Compatible() ? 1.7 : 1.6
 
+    compileJava.options.encoding = "UTF-8"
+    compileTestJava.options.encoding = "UTF-8"
+
     repositories {
         jcenter()
     }

./gradlew build

> Configure project :
applying jacoco build file for Java version 1.8

> Task :javadoc FAILED
/home/liam/picocli/src/main/java/picocli/CommandLine.java:3491: error: unmappable character for encoding ASCII
         * The synopsis would show {@code --[no-]force} and {@code -XX:??PrintGCDetails}, respectively.
                                                                       ^
/home/liam/picocli/src/main/java/picocli/CommandLine.java:3491: error: unmappable character for encoding ASCII
         * The synopsis would show {@code --[no-]force} and {@code -XX:??PrintGCDetails}, respectively.
                                                                        ^
2 errors

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':javadoc'.
> Javadoc generation failed. Generated Javadoc options file (useful for troubleshooting): '/home/liam/picocli/build/tmp/javadoc/javadoc.options'

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.

* Get more help at https://help.gradle.org

Deprecated Gradle features were used in this build, making it incompatible with Gradle 5.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/4.10.2/userguide/command_line_interface.html#sec:command_line_warnings

BUILD FAILED in 1s
7 actionable tasks: 1 executed, 6 up-to-date

javac src/main/java/picocli/CommandLine.java

src/main/java/picocli/CommandLine.java:3491: error: unmappable character for encoding ASCII
         * The synopsis would show {@code --[no-]force} and {@code -XX:??PrintGCDetails}, respectively.
                                                                       ^
src/main/java/picocli/CommandLine.java:3491: error: unmappable character for encoding ASCII
         * The synopsis would show {@code --[no-]force} and {@code -XX:??PrintGCDetails}, respectively.
                                                                        ^
2 errors

[remkop]

@leliamesteban Thanks for the detailed response.
I've replaced the non-ascii character in the code with its HTML entity, and also explicitly set the source encoding in the javadoc gradle task to ensure the build is repeatable even if non-ascii characters are introduced again later.

Either of these should enable you to build with gradle, the first fix should also enable running the javac tool without specifying an encoding. Can you verify?

[remkop]

Can I ask you a favor?

What do you see when you run this program on your system?

public class NegatableTest {

    @Option(names = "-X:-option", negatable = true, description = "java-style option")
    private String option;
    
    public static void main(String[] args) {
        new CommandLine(new NegatableTest()).usage(System.out);
    }
}

I suspect that the usage help may not show correctly on systems that have ascii as the platform encoding, but I would like to confirm.

[leliamesteban]

I managed to compile CommandLine.java individually, although it complains about unchecked or unsafe operations.

CommandLine.java:9162: warning: [unchecked] Possible heap pollution from parameterized vararg type Class<? extends Annotation>
            private static void assertNoDuplicateAnnotations(TypedMember member, Class<? extends Annotation> myAnnotation, Class<? extends Annotation>... forbidden) {
                                                                         ^
1 warning

I can't get NegatableTest to compile because it can't find the symbol for the "@option" annotation, even when I use the jar.

javac NegatableTest.java CommandLine.java

NegatableTest.java:3: error: cannot find symbol
    @Option(names = "-X:-option", negatable = true, description = "java-style option")
     ^
  symbol:   class Option
  location: class NegatableTest
NegatableTest.java:7: error: cannot access CommandLine
        new CommandLine(new NegatableTest()).usage(System.out);
            ^
  bad class file: ./CommandLine.class
    class file contains wrong class: picocli.CommandLine
    Please remove or make sure it appears in the correct subdirectory of the classpath.
2 errors

I also had to change this line

<         new CommandLine(new NegatableTest()).usage(System.out);
---
>         new picocli.CommandLine(new NegatableTest()).usage(System.out);

javac NegatableTest.java -classpath ".:picocli-4.0.0-beta-1b.jar"

NegatableTest.java:3: error: cannot find symbol
    @Option(names = "-X:-option", negatable = true, description = "java-style option")
     ^
  symbol:   class Option
  location: class NegatableTest

Sorry to respond with a compilation error different to the one you expected but I can't figure out how to get java to recognise the annotations, even when CommandLine compiles

[remkop]

Sorry my bad, I should have also pasted the import statements...

import picocli.CommandLine;
import picocli.CommandLine.Option;

public class NegatableTest {

    @Option(names = "-X:-option", negatable = true, description = "java-style option")
    private String option;

    public static void main(String[] args) {
        new CommandLine(new NegatableTest()).usage(System.out);
    }
}

(To compile the CommandLine.java source file, it should be placed in a picocli subdirectory from the directory that you run javac (because it is in the picocli package), but it's easier to just compile NegatableTest.java with the .:picocli-4.0.0-beta-1b.jar classpath.)

[remkop]

The issue is the ± character that is displayed in the synopsis for negatable Java-style options like -XX:+PrintGCDetails -> shown as -XX:±PrintGCDetails in the usage help.

I have tested and it appears that the ± character is rendered scrambled or as '?' characters if the encoding is ASCII, as well as in Cygwin and MinGW.

I am changing this to use the (+|-) character sequence instead. That should allow compilation as well as execution on ASCII systems without problems.

Thanks for raising this!

[remkop]

FYI, 4.0.0-beta-2 has been released with the fix for this issue. It may take a few hours for Maven mirrors to update.