All console output goes to the error output stream

[mjfreelancing]

I'm using D2 to generate diagrams from an internal utility. I have found that, on Windows at least, all of the console output is being sent to the error output stream, including all "success:..." messages. Is there any reason why the success messages cannot be sent to the standard output stream?

[bo-ku-ra]

> d2 foobar.d2 - > output.svg

"output.svg" is not broken.

[mjfreelancing]

I probably should have provided more context on our utility being a C# application.

I have attached a sample visual studio console application using ClipWrap (rather than my own code to remove that as a cause). The C# code looks like this:

var stdOutBuffer = new StringBuilder();
var stdErrBuffer = new StringBuilder();

_ = await Cli.Wrap("d2.exe")
    .WithArguments(new[] { "foobar.d2", "foobar.svg" })
    .WithStandardOutputPipe(PipeTarget.ToStringBuilder(stdOutBuffer))
    .WithStandardErrorPipe(PipeTarget.ToStringBuilder(stdErrBuffer))
    .ExecuteAsync();

var stdOut = stdOutBuffer.ToString();
Console.WriteLine($"Standard: {stdOut}");

var stdErr = stdErrBuffer.ToString();
Console.WriteLine($"Error: {stdErr}");

And the output is:

Standard:
Error: success: successfully compiled foobar.d2 to foobar.svg in 383.1654ms

D2OutputUsingCliWrap.zip

[alixander]

@mjfreelancing that's correct. @bo-ku-ra 's example is saying that if a CLI program's success message output were to go to stdout, there'd be no way to pipe the actual SVG output to stdout.

Unix programs need to be interoperable, there needs to be a way to write to stdin and write to stdout. Therefore, both are reserved. The only other channel for debug/info messages is stderr.

[mjfreelancing]

That's a shame considering stdout / stderr typically have usage expectations. The only way I can see this being handled on my side is to have a custom stderr handler for D2 and my regular handler for everything else. To do this though, I'm going to have to make an assumption that all success messages will be prefixed with "success" and all error output is prefixed with "err" and then apply a strategy to forward onto my regular handlers.

I'm not keen on making assumptions as anything can change at any time, but is this a feasible assumption or are there other prefixes I need to be aware of?

Edit: I'm currently looking for a prefix of "err" to handle errors and let everything else fall through to a standard handler that I can see is picking up "info" and "success". If I can rely on "err:" for all error output then this will work for me.

[alixander]

That's a shame considering stdout / stderr typically have usage expectations.

AFAIK, no popular/standard/good command line tools output debugging/info messages to stdout when their primary function is to write contents. For example, wget outputs success messages to stderr. If you have a counterexample where the tool's primary function is to write a payload (and not just displaying info), and the success message doesn't go to stderr, I'd be interested to know it.

The only way I can see this being handled on my side is to have a custom stderr handler

You can look at the exit code. Any nonzero exit code is a failure (https://tldp.org/LDP/abs/html/exitcodes.html)

[bo-ku-ra]

in windows, the concept should be the same as in UNIX, except that 'CON' is used instead of '-'.

[mjfreelancing]

@alixander I wasn't aware until your explanation that the output is being generated via stdout (as opposed to explicitly creating a file, for example). I can see the dilemna now.

In my case I am logging all output as some of my diagram generation takes quite a long time to process and I have noticed 'info' items being emitted during these times so I can't use the exit code. For now, looking at the content of the output will have to do me.

Thanks for the replies. I'll work with it as-is knowing the constraints.

On a side note, and this is how I worked out why I wasn't receiving any logs via stdout, is there a limit on how big a PNG diagram can be? I have one diagram that generates SVG files just fine, albeit quite big, but when I try to create a PNG it fails with an error invalid PNG: "data:,". I'll create another issue if you think I need to but I'll first need to string replace some proprietory content.

[bo-ku-ra]

cmd.com

c:\>d2 foobar.d2 > log.txt 2>&1

[alixander]

@mjfreelancing I don't understand why those things would prevent looking at exit code. CLI APIs should handle it for you actually. See https://pkg.go.dev/os/exec#Cmd.Run (exit status).

As for the PNG limit, unfortunately I do think there is. To know for sure, can you try decreasing the size by doing --scale=0.5?

[alixander]

related #463

[mjfreelancing]

@alixander I can't use the exit code from the point of view this is only available at the end of processing. I have one diagram that takes so long to generate there are dozens of 'info' messages sent that I would never receive. I am capturing everything as a way of knowing it is still in progress. Furthermore, the final log messages (error or success) would have already been sent and lost by the time the error code is received.

As for the PNG issue I'll give that scaling option a try.

We can park this. What I have in place meets my requirements for now. Thanks.