How to parse a value of custom type from OsStr in Clap 4?

[pkolaczk]

Previously it was quite simple - I could use #[clap(parse(from_os_str))], but now it doesn't work.
It also doesn't work automatically - I'm getting:

33   | pub struct Path {
     | ---------------
     | |
     | doesn't satisfy `path::Path: FromStr`
     | doesn't satisfy `path::Path: ValueEnum`
     | doesn't satisfy `path::Path: ValueParserFactory`
...

on my custom type. So it looks like I need to create my own ValueParserFactory, but apparently I haven't found any docs explaining how to do it.

But my type already supports From<&OsStr> through this blanket impl:

impl<P> From<P> for Path
where
    P: AsRef<std::path::Path>,
{

Why clap couldn't just use existing standard conversions like TryFrom<&OsStr> or From<&OsStr> if they are defined?
Is there any way to tell it to use my From conversion?

[epage]

Yes, simple conversions for OsStr were one thing that we have not yet carried over from parse(...). For explicit functions (#[clap(parse(from_os_str = ...)]), that requires more than one blanket impl for ValueParser::from which we can't do. I also tried to get the auto-type selection working with clap::value_parser! but last time I did so, I had lifetime issues I couldn't resolve. It looks like I've got it working this time, so hopefully a new release will include it.

In the mean time, you should be able to do #[clap(value_parser = clap::builder::OsStringValueParser::new().map(|s| ...))]. ValueParserFactoryshould only be needed if you want to make it convenient for others to get the right value parser, otherwise you can just set the value parser directly with#[clap(value_parser = ...)]`.

[epage]

I just ran into this issue in my most recent experiment with migrating off Clap v3 and, because my parser is also a validator, I tried using .try_map, only to run into a constraint failure wall that I don't have time to bang my head against:

value_parser!(PathBuf) returns an optimized value parser rather than something that implements TypedValueParser and needs to be boxed.

If you use PathBufValueParser, it should work.

Can we get a guaranteed-to-compile example for this use-case in the tutorial?

I always hesitate on these because if I added an example to the tutorial for each problem people ran into and then asked for it, it would be large enough and incohesive enough to make it hard to find anything.

Also, the example for try_map isn't too different than the solution here (using a core value parser directly rather than using the macro).

[ssokolow]

If you use PathBufValueParser, it should work.

Can we say that explicitly in the docs? I'm struggling with Clap 4's documentation being a giant wall of traits and types and "Useful for composing new TypedValueParsers" was about as meaningful to me as Mandarin Chinese in that context.

What does it mean to be 'useful for' composing? When would I want it? Why can't I use something else? etc. etc. etc.

I tried to start from value_parser!(PathBuf) because it was the only thing I could find in the declarative API examples that conclusively would enable whatever secret magic sauce was necessary to process arguments as &OsStr, not &str... which, by the way, is the major reason I use clap: It's the only thing I've found that combines a derive-based library with not-broken-by-design processing of POSIX paths.

All in all, this whole Clap 3 to Clap 4 migration feels like a giant footgun for lack of a sufficiently front-and-center migration guide and, in some cases, I was so burned out that I just dropped constraints like empty_values = false and quietly wished I'd distrusted clap 3.x enough to write regression tests for my schema like I've done in the past in Python.

I always hesitate on these because if I added an example to the tutorial for each problem people ran into and then asked for it, it would be large enough and incohesive enough to make it hard to find anything.

"How do I get the type system to let me have custom parsing without forcing POSIX paths to go through a &str step?" seems to be a pretty important special case. If I weren't so determined to avoid having a second set of structs and a hand-written TryFrom, I'd just have treated it as impossible and done it in code afterward.

Also, the example for try_map isn't too different than the solution here (using a core value parser directly rather than using the macro).

By the time I found it, I was so lost that I took one look at it and checked out, mentally, because "It's using the builder API. That won't help me." I was just too overwhelmed to try to construct a complete enough mental model to translate back and forth between the two APIs.

To be honest, Clap's 4's redesign has front-loaded so much API complexity (eg. I just followed the compiler errors without knowing what the difference between #[command(...)] and #[arg(...)] is or why the old #[clap(...)] for both wasn't good enough) that, for my use I'm honestly starting to wonder if I might save time in the long run by offering to help the developer of Gumdrop rewrite it to support OsStr arguments.

(At the very least, it feels like there's a growing need in the ecosystem for a declarative argument parser for simple cases that still handles POSIX paths correctly... a situation not helped by the absent source links in the docs.rs documentation... though, granted, I noticed that more when I was trying to figure out if there was a way to fully match the Clap 3.x styling by uppercasing the headers.)

[epage]

That sounds like a frustrating experience. A lot of it seems to stem from not finding the right documentation in our changelog which put you on a bad path for finding how to migrate.

I'm going to highlight a couple of points, not to say that it absolves us or that you should have found it, but to see if we can figure out why it wasn't found and if there are things to improve.

It also sounds like a lot of the frustration might have been short circuited if questions were asked in discussions or elsewhere (while discord is a contentious topic, I do know that there are people active in #cli-and-tui that quickly respond to people's questions). I'd be curious why this wasn't done earlier to see if there are ways we can lower the barrier.

All in all, this whole Clap 3 to Clap 4 migration feels like a giant footgun for lack of a sufficiently front-and-center migration guide and, in some cases, I was so burned out that I just dropped constraints like empty_values = false and quietly wished I'd distrusted clap 3.x enough to write regression tests for my schema like I've done in the past in Python.

Our migration guide is in the 4.0.0 announcement, like it was for 3.0.0.

The following step

Run cargo check --features clap/deprecated and resolve all deprecation warnings

would have caused the following output to be reported for

Replaced with Arg::forbid_empty_values

Which updating to that would have reported:

Replaced with Arg::value_parser(NonEmptyStringValueParser::new())

Derive: replace #[clap(forbid_empty_values = true)] with #[clap(value_parser = NonEmptyStringValueParser::new())]

Builder: replace arg.forbid_empty_values(true) with arg.value_parser(NonEmptyStringValueParser::new())

That double step is annoying. I updated a lot of prior deprecations to go straight to the new ones but I didn't for the ones dating back to 3.0.0 because I had assumed people had already responded to them as that was when deprecations were on by default.

As for why deprecations are behind a feature flag, it is to allow people to more easily choose when to respond to them without adding -Adeprecated. See #3822.

I noticed that more when I was trying to figure out if there was a way to fully match the Clap 3.x styling by uppercasing the headers.

From the changelog

(help) Help headings are now title cased, making any user-provided help headings inconsistent. To get the old behavior, see Command::help_template, Arg::help_heading, and Command::subcommand_help_heading (#4132)

[epage]

As for the value parser documentation

What does it mean to be 'useful for' composing? When would I want it? Why can't I use something else? etc. etc. etc.

Could you point me to where you found that phrase? I'm not finding it at the moment to see about fixing it.

I tried to start from value_parser!(PathBuf) because it was the only thing I could find in the declarative API examples that conclusively would enable whatever secret magic sauce was necessary to process arguments as &OsStr, not &str... which, by the way, is the major reason I use clap: It's the only thing I've found that combines a derive-based library with not-broken-by-design processing of POSIX paths.

If you don't mind sharing, do you have a rough guess as to the path through the documentation that you took? That way I can see about finding the right place to touch up to clarify the path for solving this problem.

[ssokolow]

It also sounds like a lot of the frustration might have been short circuited if questions were asked in discussions or elsewhere (while discord is a contentious topic, I do know that there are people active in #cli-and-tui that quickly respond to people's questions).

I habitually assume that individual libraries smaller than GTK or Qt don't have a big enough community to manage a real-time chat for help (and, by the time I realize I can't solve something myself, I'm more likely to report a bug than to reconsider any forums I passed by), so it's rare for me to look for chat channels.

As for Discord, I also don't like how cluttered my Discord is becoming with channels I do intend to use more than once or how every new "server" I join and forget to leave is rolling more dice for, at some point in the future, getting me locked out of chatting with my friends with an unfixable "We thought your account looked suspicious so, until you sign up for a mobile plan so you can give us an SMS number, your account will be locked" message that they explicitly say they will make no exceptions to... I have no mobile plan and have no intent to get one.)

Our migration guide is in the 4.0.0 announcement, like it was for 3.0.0.

That explains the miscommunication.

I didn't see the 4.0.0 announcement and would have forgotten to look in it for a migration guide if I did, because, before Reddit killed the RSS feed for /r/rust/ and drove me off Reddit, I tended to skim the comments before following links (a habit developed in more clickbait-y subreddits), and I'm pretty sure I saw the "color support is temporarily gone" part before I ever had reason to read the announcement.

I don't even know if had the opportunity to see the deprecations if I'd noticed the option. I skipped from 3.x to 4.4.7 because of the lack of an API-stable way to keep the color scheme from clap 3.x and, when I managed to fit in time to migrate, I didn't think to check if the changelog contained anything helpful and went straight to https://docs.rs/clap/latest/clap/ where I didn't see any link to any migration guides, so I started treating it as learning a new API from scratch.

(My experience with the 2.x to 3.x migration was that I already knew you'd just subsumed StructOpt more or less verbatim from the /r/rust/ comments, so I didn't check the changelog there either and my strategy was much more effective.)

Overall, my only exposure to the clap changelog has been through excerpts GitHub Dependabot presents to me. (And most of the 4.x lifecycle I haven't been receiving even those because I'd tell Dependabot to "ignore this minor version" each time I found that there was still no stable theming API.)

(help) Help headings are now title cased, making any user-provided help headings inconsistent. To get the old behavior, see Command::help_template, Arg::help_heading, and Command::subcommand_help_heading (#4132)

I'll just live with it. I was hoping I wouldn't have to resort to the nuclear option, since I wasn't a fan of having to maintain a custom template just to inject an additional newline for help2man compatibility.

It'd be nice if there were a way to specify a custom closure to act as a styling modifier.

Could you point me to where you found that phrase? I'm not finding it at the moment to see about fixing it.

The page you linked to.

https://docs.rs/clap/latest/clap/builder/struct.PathBufValueParser.html

Its non-default documentation appears to consist solely of the struct docstring being this...

Implementation for ValueParser::path_buf

Useful for composing new TypedValueParsers

...and a docstring for new() that could have been machine-generated as a default from a template.

Implementation for ValueParser::path_buf

If you don't mind sharing, do you have a rough guess as to the path through the documentation that you took? That way I can see about finding the right place to touch up to clarify the path for solving this problem.

I don't mind.

1. I landed at https://docs.rs/clap/latest/clap/ and didn't see anything like a migration guide.
2. I went into https://docs.rs/clap/latest/clap/_derive/index.html to adapt the derives that were failing to compile. (Speaking of which, when I discovered that #[clap(...)] was no longer the way and went to see some larger-scale functioning code, I discovered that the chapter_X links in the sidebar of https://docs.rs/clap/latest/clap/_derive/_tutorial/chapter_0/index.html (as linked from the front page) are broken, but manually editing the address bar URL showed me that the rest of the chapters were too verbose to be an ideal choice for efficiently identifying the information I needed.)
3. I can't remember exactly how I wound up determining that value_parser was the correct way to switch on the OsStr machinery, but I think Google led me to derive: default to parse_os_str for OsString or PathBuf types #3496 and then to a commit message indicating that allow_invalid_utf8 was being replaced with magic value_parser behaviour.)
4. I tried that and slammed into "Expected a function that takes a str" type system messages.
5. I went back to the reference to double-check that I wasn't still operating from outdated docs and managed to follow the link to https://docs.rs/clap/latest/clap/macro.value_parser.html where the mentions of things like From<&OsStr> types convinced me I was indeed dealing with "magic behaviour".
6. I started searching for details on what characteristics my incantation must have to get clap to switch on the magic "parse from OsStr instead of str" behaviour for a fn(&OsStr) or Fn(&OsStr) (fn, I think? I don't think I remember clap accepting closures there).
7. I went to check the cookbook I'd seen (https://docs.rs/clap/latest/clap/_derive/_cookbook/index.html) to see if such a thing was in there and "Typed arguments: derive" with its "Custom parse()" made me realize that maybe I could work from PathBuf instead of OsStr.
8. It was around this point that I got too frustrated and confused to form a clear memory of the rest of the process.

(It didn't help that I was in a mindstate of "I just got off clearing out several dozen Dependabot PRs. I need to upgrade half a dozen projects which use Clap. I don't want to RTFM a new API from scratch in a language I came to for improved maintainability. Why do I have to "choose three of four" for automatic --help generation, a derive-based API, proper handling of POSIX paths, and a simple, stable API suitable for 'shell scripts written in Rust'?". Heck, that (and better compile times) is why I choose Gumdrop over clap whenever I'm doing something I don't expect to ever need to pass to find [...] -exec, such as feeding config paths to actix-web projects. Gumdrop is simple to use, fast to compile, doesn't change much, has a derive API, and generates --help for me... it's just built around the assumption that all valid argument values can be represented in a String.)

I think that part of the reason I expected the migtation guide to be on docs.rs is that, if you're bending rustdoc to start to act more like mdBook with modules like _derive, my intuition is that you're trying to make the Python manual, which includes that sort of version-related information in the same Sphinx (formerly LaTeX) project as the API docs. (Often right in the API docs, as with the subprocess module's section on replacing uses of other APIs.)

[Dude4Linux]

value_parser!(PathBuf) returns an optimized value parser rather than something that implements TypedValueParser and needs to be boxed.

If you use PathBufValueParser, it should work.

When I tried substituting PathBufValueParser for PathBuf, I get an error:

error[E0412]: cannot find type `PathBufValueParser` in this scope
  --> src/main.rs:50:51
   |
50 |                 .value_parser(clap::value_parser!(PathBufValueParser))
   |                                                   ^^^^^^^^^^^^^^^^^^ not found in this scope
   |
help: consider importing this struct
   |
1  + use clap::builder::PathBufValueParser;
   |

When I followed the suggestion, I got a different error:

error[E0599]: the method `value_parser` exists for reference `&&&&&&_AutoValueParser<PathBufValueParser>`, but its trait bounds were not satisfied
    --> src/main.rs:51:31
     |
51   |                 .value_parser(clap::value_parser!(PathBufValueParser))
     |                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ method cannot be called on `&&&&&&_AutoValueParser<PathBufValueParser>` due to unsatisfied trait bounds
     |
    ::: /home/john/.cargo/registry/src/index.crates.io-6f17d22bba15001f/clap_builder-4.4.18/src/builder/value_parser.rs:1033:1
     |
1033 | pub struct PathBufValueParser {}
     | -----------------------------
     | |
     | doesn't satisfy `PathBufValueParser: From<&'s std::ffi::OsStr>`
     | doesn't satisfy `PathBufValueParser: From<&'s str>`
     | doesn't satisfy `PathBufValueParser: From<OsString>`
     | doesn't satisfy `PathBufValueParser: From<std::string::String>`
     | doesn't satisfy `PathBufValueParser: FromStr`
     | doesn't satisfy `PathBufValueParser: ValueEnum`
     | doesn't satisfy `PathBufValueParser: ValueParserFactory`
...
2462 | pub struct _AutoValueParser<T>(std::marker::PhantomData<T>);
     | ------------------------------ doesn't satisfy `_: _ValueParserViaParse`
     |
     = note: the following trait bounds were not satisfied:
             `PathBufValueParser: ValueEnum`
             which is required by `&&&&&_AutoValueParser<PathBufValueParser>: clap::builder::via_prelude::_ValueParserViaValueEnum`
             `PathBufValueParser: ValueParserFactory`
             which is required by `&&&&&&_AutoValueParser<PathBufValueParser>: clap::builder::via_prelude::_ValueParserViaFactory`
             `PathBufValueParser: From<OsString>`
             which is required by `&&&&_AutoValueParser<PathBufValueParser>: clap::builder::via_prelude::_ValueParserViaFromOsString`
             `PathBufValueParser: From<&'s std::ffi::OsStr>`
             which is required by `&&&_AutoValueParser<PathBufValueParser>: clap::builder::via_prelude::_ValueParserViaFromOsStr`
             `PathBufValueParser: From<std::string::String>`
             which is required by `&&_AutoValueParser<PathBufValueParser>: clap::builder::via_prelude::_ValueParserViaFromString`
             `PathBufValueParser: From<&'s str>`
             which is required by `&_AutoValueParser<PathBufValueParser>: clap::builder::via_prelude::_ValueParserViaFromStr`
             `PathBufValueParser: FromStr`
             which is required by `_AutoValueParser<PathBufValueParser>: clap::builder::via_prelude::_ValueParserViaParse`
     = note: this error originates in the macro `clap::value_parser` (in Nightly builds, run with -Z macro-backtrace for more info)

Do you have any other thoughts or suggestions?

[epage]

PathBufValueParser needs to be used without the macro. The macro helps find a value parser.