Bad encoding on default picocli.CommandLine#getOut and picocli.CommandLine#getErr

[charphi]

Depending on your console settings, the default PrintWriter of picocli.CommandLine#getOut may use a bad encoding.

Picocli code:

public PrintWriter getOut() {
    if (this.out == null) {
        this.setOut(new PrintWriter(System.out, true));
    }

    return this.out;
}

This code creates a PrintWriter using Charset.defaultCharset() which can be different from the current console encoding.
A possible solution would be to use Charset.forName(System.getProperty("sun.stdout.encoding")).
See https://github.com/keerath/openjdk-8-source/blob/5f6e9d42a9f6b6736100c9c6f43f5f5ea1570cfb/jdk/src/share/classes/java/lang/System.java#L1189

Here is a code to reproduce (Windows 10, command prompt, encoding cp437):

///usr/bin/env jbang "$0" "$@" ; exit $?
//REPOS jitpack
//DEPS com.github.remkop.picocli:picocli:4.6.1
//DEPS com.github.remkop.picocli:picocli-codegen:4.6.1

import picocli.CommandLine;

import java.io.OutputStreamWriter;
import java.nio.charset.Charset;
import java.util.concurrent.Callable;

@CommandLine.Command(name = "EncodingIssue")
class EncodingIssue implements Callable<Void> {

    @CommandLine.Spec
    private CommandLine.Model.CommandSpec spec;

    public static void main(String... args) {
        System.exit(new CommandLine(new EncodingIssue()).execute(args));
    }

    @Override
    public Void call() throws Exception {
        String problem = "[abcµ]" + System.lineSeparator();
        // ko
        spec.commandLine().getOut().append("1" + spec.commandLine().getColorScheme().errorText(problem)).flush();
        spec.commandLine().getOut().append("2" + problem).flush();
        new OutputStreamWriter(System.out, Charset.defaultCharset()).append("3" + problem).flush();
        // ok
        System.out.append("4" + problem).flush();
        new OutputStreamWriter(System.out, getStdoutEncoding()).append("5" + problem).flush();
        return null;
    }

    static Charset getStdoutEncoding() {
        String charsetName = System.getProperty("sun.stdout.encoding");
        if (charsetName != null) return Charset.forName(charsetName);
        return Charset.defaultCharset();
    }
}

[charphi]

Screenshot of the problem:

[remkop]

Interesting, I was not aware of this.
Thank you for raising this!

Will you be able to provide a pull request for this?

[remkop]

In the example screenshot, what is the encoding used by your console/terminal? (Update: understood about cp437.)
There is still a ? question mark displayed in front of the µ glyph in lines 4 and 5. Why do you think this is?

I assume that your source code (that contains the µ character) is in unicode, and this is different from the encoding used by your console/terminal. What would you see if your source code also uses the same encoding used by your console/terminal? (you may need to specify the encoding to javac in order to compile.)

[charphi]

My code doesn't contain special characters but deal with .csv files which might contain some.

For the question mark, I don't know 😕 It might come from the font used by the terminal.

[charphi]

I've done some research and the question mark is the result of the use of an unmappable-character.

When an unmappable-character is encountered, the default behavior of PrintWriter is to replace the problematic character with a replacement value.

Here is some code to test:

String problem = "[abcµ]";
CharsetEncoder cp437 = Charset.forName("cp437").newEncoder();
System.out.println(cp437.canEncode(problem)); // returns false

cp437.onUnmappableCharacter(CodingErrorAction.REPLACE);
cp437.encode(CharBuffer.wrap(problem)); // doesn't throw exception

cp437.onUnmappableCharacter(CodingErrorAction.REPORT);
cp437.encode(CharBuffer.wrap(problem)); // throws exception

[remkop]

Thanks for the PR!
Merged and documented in the release notes.