Emit request body only

[jayvdb]

I am using dummyhttp as a fake https://opentelemetry.io/ listener , receiving POST for events in our system, and have been using -vv to see the incoming events.

I am less interested in the headers going in or out. Only the incoming POST body interests me. The incoming POST bodies are JSON. What would be even better is if the incoming JSON could be parsed into JSON and then emitted as a single line JSONL. i.e. removing all the newlines from the incoming POST ;-)

Obviously the design needs to be generic so that it would be useful for others also.

I'm happy to do all the work, provided it has a chance of being accepted.

[svenstaro]

So there are two feature requests here: You want to have more accurate control over the logging and you want to add some kind of transformer for incoming requests? I would be down for both.

What's your design proposal (flags primarily) for the logging part? We could follow the convention of httpie.

[jayvdb]

I like the idea of httpie's --print - it looks ideal .

What about also adding --body & --headers ? In httpie they are for responses , whereas in dummyhttp the response is less interesting than the request. Perhaps more explicit --request-body and --response-body ? IMO they are unnecessary if --print is provided.

Interestingly https://httpie.io/docs/cli/verbose-output -v is roughtly dummyhttp's -vv , and httpie's -vv has even more info like "Elapsed time: 1.099171542s".

If I have --print working, then I believe the output will be entirely JSON , so I could feed the stdout to another tool to do the transformer part of this request, and dont need that feature in dummyhttp.

[svenstaro]

How will --print and -v, -vv interact?

[jayvdb]

yea, I am playing around with this, and that is one of the significant issues.

My initial thinking is that --print and -v are made to be mutually exclusive using clap arg "groups". Looks to be easy; havent tried it yet.

Another is that I havent found a way for clap "derive" to do --print similar to httpie.

Currently I have

#[derive(clap::ValueEnum, Clone, Debug, PartialEq)]
pub enum PrintOption {
    #[clap(name = "B")]
    RequestBody,
    #[clap(name = "H")]
    RequestHeaders,
    #[clap(name = "b")]
    ResponseBody,
    #[clap(name = "h")]
    ResponseHeaders,
}

pub struct Args {
    ..
    #[arg(long, value_delimiter = ',', num_args = 1..)]
    pub print: Vec<PrintOption>,
    ..
}

and I am running with that atm, using args.print.contains.

I'd like to be using something like https://docs.rs/enumset/latest/enumset/

My rough thinking is that the comma separators are annoying, but not a deal breaker, and once everything else is built, we can ask the clap team if there is an approach which will avoid the commas.

[svenstaro]

I think here it would be reasonable to do a custom parser function to the get behavior from httpie.

[jayvdb]

https://github.com/jayvdb/dummyhttp/tree/print-arg has a first cut with one commit .

With that I can do dummyhttp --quiet --print=B --body '' --port 5080 | jq '..' .

Still a lot of work needed to determine how to integrate this to avoid duplication. As "colored" detects tty using is_terminal , the --print code can use colored decorations , which will be discarded when piped to jq .

If a user did --print=HB, what should be the separators between headers and body. IMO it should look like a HTTP stream of the headers and bodies. I wouldnt expect "┌─Incoming request" to be constantly appearing.

So I am thinking that "┌─Incoming request" and "┌─Outgoing response" would only be used if --print toggles are used to print something from both request and response.

I do wonder whether --print b makes any sense as it is a constant. I guess it could become more dynamic if dummyhttp used something like https://github.com/cksac/fake-rs , but then how would this work if dummyhttp was returning a binary format such as an image. Sentry logs request bodies, but has some config to control this, and typically only logs JSON and form bodies.

I checked apache logging, and it has no way to make LogFormat include the POST body, so that wouldnt work for me. Their dumpio module seems to be the recommended approach for logging POST bodies. traefik also has no plugin for logging request bodies

Kong has a way to achieve POST logging, but that doesnt seem like it helps with a design.

expressjs's morgan doesnt support bodies but can be extended to do this

It looks like haproxy does have a mechanism for logging request bodies https://discourse.haproxy.org/t/log-http-request-body-and-performance-impact/6894

[svenstaro]

I think --print b still makes sense as the body argument supports some mild templating. I would think that changing the way to do logging for binary data in bodies is useful (and we could just go [5243 bytes of binary data]) but isn't it out of scope for this particular issue? I'd be happy to tackle that in a separate issue.

About the separators: Yeah no idea come to think of it. Maybe get rid of the separators altogether in case of --print and keep them for -vv? I really like them as they break up the dense information into easily skimmable chunks.