Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Repeating options with sub-options/arguments #635

Closed
hanslovsky opened this issue Feb 19, 2019 · 14 comments
Closed

Repeating options with sub-options/arguments #635

hanslovsky opened this issue Feb 19, 2019 · 14 comments
Labels
status: duplicate 👨🏻‍🤝‍👨🏻 A duplicate of another issue theme: parser An issue or change related to the parser
Milestone

Comments

@hanslovsky
Copy link

hanslovsky commented Feb 19, 2019

I would like to add an option multiple times and add sub-options/parameters to it. In my scenario, I have a viewer application and, among other options and one positional argument, I would like to add datasets, somewhat like this:

viewer <positional-argument> \
    --add-dataset --container=c1 --dataset=d1 \
    --add-dataset --dataset=<d2 --type=label \
    --fallback-container=fbc \ # use this as fallback, if no container is specified 
    [other options]

For this call, I would add two datasets to my container:

  • d1 in container c1 (type is auto-detected from meta data)
  • d2 in container fbc as label dataset

As far as I can tell, this is not (yet) supported in picocli (please correct me if I am wrong), and my - kind of hackish way - to achieve that, is to add an Option

@CommandLine.Option(names = arrayOf("--add-dataset"), arity = "+")
var addDatasetStrings: List<String> = mutableListOf()

that collects greedily all Strings that follow that option into a list. Users need to append a split-char after the last option of each --add-dataset (I chose _ here). I then split the list at _, and parse each of the sublists with a separate parser. This works well for me but it has a few downsides, as far as I can tell:

  • I do not think it is possible to have overlapping arguments between the main arguments/options and the options/arguments for --add-dataset.
  • Need delimiter string (I used _)
  • help message for sub option (--add-dataset) not included automatically

I added my (kotlin) code and an invocation example at the very end of this comment for reference.

I think that this would be a useful addition. There are several issues that found that are related but not the same in my understanding:

Out of these, #454 seems to be the most closely related issues. For my real-world use case (viewer application with one positional argument, and more options), I could turn the one positional argument into an option and #454 could be a working solution for me. For anything that requires positional arguments, I would be concerned that the positional arguments might clash with the subcommands.

import org.apache.commons.lang3.builder.ToStringBuilder
import picocli.CommandLine
import java.util.concurrent.Callable

@CommandLine.Command

class AddDataset(val fallbackContainer: String?) : Callable<AddDataset> {

	@CommandLine.Option(names = arrayOf("--container"))
	var container: String? = null

	@CommandLine.Option(names = arrayOf("--dataset"), required=true)
	var dataset: String? = null

	@CommandLine.Option(names = arrayOf("--help", "-h"),  usageHelp = true)
	var helpRequested = false

	override fun call(): AddDataset {
		return this
	}

	override fun toString(): String {
		return ToStringBuilder(this)
				.append("container", container?: fallbackContainer)
				.append("dataset", dataset)
				.toString()
	}

}


class Args : Callable<List<AddDataset>> {

	@CommandLine.Option(names = arrayOf("--default-container"))
	var defaultContainer: String? = null

	@CommandLine.Option(names = arrayOf("--add-dataset"), arity = "+")
	var addDatasetStrings: List<String> = mutableListOf()

	@CommandLine.Option(names = arrayOf("--help", "-h"),  usageHelp = true)
	var helpRequested = false

	override fun call(): List<AddDataset> {

		val indices = addDatasetStrings
				.withIndex()
				.filter { it.value.equals("_", ignoreCase = true) }
				.map { it.index }

		val subLists = mutableListOf<List<String>>()
		var nextStartingIndex = 0
		for (i in 0 until indices.size) {
			subLists.add(addDatasetStrings.subList(nextStartingIndex, indices[i]))
			nextStartingIndex = indices[i] + 1
		}

		return subLists.map { CommandLine.call(AddDataset(defaultContainer), *it.toTypedArray()) }
	}

}

fun main(argv: Array<String>) {

	val args = Args()
	val datasets = CommandLine.call(args, *argv)
	println(datasets)

}
$ Command --add-dataset --container=123 --dataset=456 _ --default-container=abracadabra --add-dataset --dataset=789 _
[AddDataset@5b87ed94[container=123,dataset=456], AddDataset@6e0e048a[container=abracadabra,dataset=789]]
@remkop
Copy link
Owner

remkop commented Feb 20, 2019

Hi @hanslovsky , the plan is that picocli 4.0 will add support for composite repeating groups.

The goal is that argument groups will support:

For example, it should be possible to have a command with a synopsis like this:

viewer [-a -d=DATASET [-c=CONTAINER] [-t=TYPE]]... [-f=FALLBACK] <positional>

or, if the group needs to be specified at least once:

viewer (-a -d=DATASET [-c=CONTAINER] [-t=TYPE])... [-f=FALLBACK] <positional>

(Where -a is short for --add-dataset, -c is short for --container, etc)

I believe this will meet your requirements, but please give it a try when early versions become available.

This is currently work in progress, and an early (but insuffient - lacks support for repeating composite arguments) version is currently in master. The #199 ticket has some comments that describe the current state of things and follow-up plans. This month I have other commitments but I will resume working on this from March.

@hanslovsky
Copy link
Author

That looks very much like what I need! I'll close this issue as soon as I have tried and confirmed that it works for me.

Thanks for keeping up the great work, have been waiting for a good Java command line parser for a long time and picocli exceeds expecations. I keep spreading the word and now my whole lab uses it.

remkop added a commit that referenced this issue Mar 18, 2019
* mutually exclusive options (#199)
* option that must co-occur (#295)
* option grouping in the usage help message (#450)
* repeating composite arguments (#358 and #635) (this should also cover the use cases presented in #454 and #434 requests for repeatable subcommands)
@remkop remkop added this to the 4.0 milestone Mar 18, 2019
remkop added a commit that referenced this issue Mar 28, 2019
TODO: validation logic needs to be reviewed; docs for programmatic API
remkop added a commit that referenced this issue Mar 28, 2019
TODO: validation logic needs to be reviewed; docs for programmatic API
@remkop remkop modified the milestones: 4.0, 4.0-alpha-1 Mar 30, 2019
@remkop
Copy link
Owner

remkop commented Mar 30, 2019

Picocli 4.0.0-alpha-1 has been released which includes support for repeating composite groups.
See https://picocli.info/#_argument_groups for details.

Please try this and provide feedback. We can still make changes.

What do you think of the annotations API? What about the programmatic API? Does it work as expected? Are the input validation error messages correct and clear? Is the documentation clear and complete? Anything you want to change or improve? Any other feedback?

@hanslovsky
Copy link
Author

hanslovsky commented Mar 30, 2019

Great work, thanks for releasing 4.0.0-alpha-1!
Repeating composite argument groups (7.4 in the user guide) is intuitive and works as expected. I currently do not understand how to trigger a composite group by an option. In the minimum working example below, I can create a CLI with synopsis

test-composite [--option1=<option1> --option2=<option2>]...

but it is unclear to me how I could create a CLI with synopsis

test-composite [--add-group [--option1=<option1> --option2=<option2>]]...

I could add an --add-group option to the group that also contains --option1 and --option2 but then the order would not be respected. Also, semantically, it would not make a lot of sense, in my opinion.

Minimum working example: I tried to annotate groups with @CommandLine.Option but @Option and @ArgGroup are mutually exclusive (see commented line):

import picocli.CommandLine;

import java.util.List;

public class OptionGroupsMWE {

	@CommandLine.Command(name = "test-composite")
	static class Args implements Runnable {
//		@CommandLine.Option(names = "--add-group")
		@CommandLine.ArgGroup(multiplicity = "0..*", exclusive = false)
		List<Group> groups = null;

		@CommandLine.Option(names = "--help", usageHelp = true)
		boolean helpRequested;

		@Override
		public String toString() {
			return String.format("%s[groups=%s,helpRequested=%s]", getClass().getSimpleName(), groups, helpRequested);
		}

		@Override
		public void run() {}
	}

	static class Group {
		@CommandLine.Option(names = "--option1", required = true)
		String option1 = null;

		@CommandLine.Option(names = "--option2", required = true)
		String option2 = null;

		@Override
		public String toString() {
			return String.format("%s[option1=%s,option2=%s]", getClass().getSimpleName(), option1, option2);
		}
	}

	public static void main(String[] argv) {
		final Args args = new Args();
		CommandLine.run(args, argv);
		System.out.println(args);
	}
}

@remkop
Copy link
Owner

remkop commented Mar 31, 2019

Any option in the group will "trigger" the group. After a group is started, the parser verifies that all required elements are present. The second time that a required element is specified, it triggers another multiple of that group.

To get the following synopsis:

test-composite [--add-group (--option1=<option1> --option2=<option2>)]...

you would create a nested group:

@Command(name = "test-composite")
class TestComposite {

  @ArgGroup(exclusive = false, multiplicity = "0..*")
  List<OuterGroup> outerList;

  static class OuterGroup {
    @Option(names = "--add-group", required = true) boolean addGroup;

    @ArgGroup(exclusive = false, multiplicity = "1")
    Inner inner;

    static class Inner {
      @Option(names = "--option1", required = true) String option1;
      @Option(names = "--option2", required = true) String option2;
    }
  }
}

By creating an "inner" group, you make them a single element, so users can specify --add-group either before or after --option1 and --option2, but not in between. Within the inner group, elements may be specified in any order, so option2 may precede option1, as long as they are both specified. The following are valid input for the above group definition:

# valid input:
cmd --add-group --option1=1 --option2=1 --add-group --option1=2 --option2=2
cmd --add-group --option2=1 --option1=1 --option1=2 --option2=2 --add-group 
cmd --option2=1 --option1=1 --add-group --add-group --option1=2 --option2=2

By contrast, if all options are in a single group, for example, if OuterGroup was defined as a single flat group, like this:

  static class OuterGroup {
    @Option(names = "--add-group", required = true) boolean addGroup;
    @Option(names = "--option1", required = true) String option1;
    @Option(names = "--option2", required = true) String option2;
  }

Then it would be valid to specify --add-group between --option1 and --option2.

@remkop
Copy link
Owner

remkop commented Mar 31, 2019

I reopened #454, because it may be more natural to model your use case as a repeating subcommand.

@hanslovsky
Copy link
Author

hanslovsky commented Mar 31, 2019

Nested groups is a clever solution! --add-group after the inner options should not be a problem, in practice, in particular as the synopsis encourages use as intended.

For some reason, I can omit required options from the inner group without error, e.g. with the TestComposite class (the last example only fails to produce an error with CommandLine.run):

--add-group --option1=1
--add-group --option2=1
--add-group

For completeness, the entire example:

import picocli.CommandLine;

import java.util.List;

@CommandLine.Command(name = "test-composite")
public class TestComposite {

	@CommandLine.ArgGroup(exclusive = false, multiplicity = "0..*")
	List<OuterGroup> outerList;

	static class OuterGroup {
		@CommandLine.Option(names = "--add-group", required = true) boolean addGroup;

		@CommandLine.ArgGroup(exclusive = false, multiplicity = "1")
		Inner inner;

		static class Inner {
			@CommandLine.Option(names = "--option1", required = true) String option1;
			@CommandLine.Option(names = "--option2", required = true) String option2;
		}
	}

	public static void main(String[] argv) {
		final TestComposite args = new TestComposite();
		new CommandLine(args).parse(argv);
	}
}

@remkop
Copy link
Owner

remkop commented Mar 31, 2019

I’ll try this when I get to my PC. It’s possible that the validation logic needs more work.

@remkop
Copy link
Owner

remkop commented Mar 31, 2019

By the way, you can get insight into what the parser is doing by enabling tracing with system property -Dpicocli.trace=DEBUG. Just FYI. I’ll look at this later today.

@hanslovsky
Copy link
Author

Debug output for args --add-group --option2=1:

[picocli DEBUG] Creating CommandSpec for object of class org.janelia.saalfeldlab.paintera.TestComposite with factory picocli.CommandLine$DefaultFactory
[picocli INFO] Picocli version: 4.0.0-alpha-1, JVM: 1.8.0_212 (Oracle Corporation OpenJDK 64-Bit Server VM 25.212-b01), OS: Linux 5.0.4-arch1-1-ARCH amd64
[picocli INFO] Parsing 2 command line args [--add-group, --option2=1]
[picocli DEBUG] Parser configuration: posixClusteredShortOptionsAllowed=true, stopAtPositional=false, stopAtUnmatched=false, separator=null, overwrittenOptionsAllowed=false, unmatchedArgumentsAllowed=false, expandAtFiles=true, atFileCommentChar=#, useSimplifiedAtFiles=false, endOfOptionsDelimiter=--, limitSplit=false, aritySatisfiedByAttachedOptionParam=false, toggleBooleanFlags=true, unmatchedOptionsArePositionalParams=false, collectErrors=false,caseInsensitiveEnumValuesAllowed=false, trimQuotes=false, splitQuotedStrings=false
[picocli DEBUG] (ANSI is disabled by default: isatty=false, XTERM=null, OSTYPE=null, isWindows=false, JansiConsoleInstalled=false, ANSICON=null, ConEmuANSI=null, NO_COLOR=null, CLICOLOR=null, CLICOLOR_FORCE=null)
[picocli DEBUG] Initializing org.janelia.saalfeldlab.paintera.TestComposite: 3 options, 0 positional parameters, 0 required, 1 groups, 0 subcommands.
[picocli DEBUG] Processing argument '--add-group'. Remainder=[--option2=1]
[picocli DEBUG] '--add-group' cannot be separated into <option>=<option-parameter>
[picocli DEBUG] Found option named '--add-group': field boolean org.janelia.saalfeldlab.paintera.TestComposite$OuterGroup.addGroup, arity=0
[picocli INFO] Adding multiple to MatchedGroup [--add-group (--option1=<option1> --option2=<option2>)]...={} (group=1 [--add-group (--option1=<option1> --option2=<option2>)]...).
[picocli DEBUG] Creating new user object of type class org.janelia.saalfeldlab.paintera.TestComposite$OuterGroup for group [--add-group (--option1=<option1> --option2=<option2>)]...
[picocli DEBUG] Created org.janelia.saalfeldlab.paintera.TestComposite$OuterGroup@6fffcba5, invoking setter FieldBinding(java.util.List org.janelia.saalfeldlab.paintera.TestComposite.outerList) with scope Scope(value=org.janelia.saalfeldlab.paintera.TestComposite@34340fab)
[picocli DEBUG] Initializing --add-group in group [--add-group (--option1=<option1> --option2=<option2>)]...: setting scope to user object org.janelia.saalfeldlab.paintera.TestComposite$OuterGroup@6fffcba5 and initializing initial and default values
[picocli DEBUG] Initial value not available for field boolean org.janelia.saalfeldlab.paintera.TestComposite$OuterGroup.addGroup
[picocli DEBUG] Setting scope for subgroup (--option1=<option1> --option2=<option2>) with setter=FieldBinding(org.janelia.saalfeldlab.paintera.TestComposite$OuterGroup$Inner org.janelia.saalfeldlab.paintera.TestComposite$OuterGroup.inner) in group [--add-group (--option1=<option1> --option2=<option2>)]... to user object org.janelia.saalfeldlab.paintera.TestComposite$OuterGroup@6fffcba5
[picocli DEBUG] Initialization complete for group [--add-group (--option1=<option1> --option2=<option2>)]...
[picocli INFO] Setting field boolean org.janelia.saalfeldlab.paintera.TestComposite$OuterGroup.addGroup to 'true' (was 'false') for option --add-group
[picocli DEBUG] Processing argument '--option2=1'. Remainder=[]
[picocli DEBUG] Separated '--option2' option from '1' option parameter
[picocli DEBUG] Found option named '--option2': field String org.janelia.saalfeldlab.paintera.TestComposite$OuterGroup$Inner.option2, arity=1
[picocli INFO] Adding multiple to MatchedGroup (--option1=<option1> --option2=<option2>)={} (group=1.1 (--option1=<option1> --option2=<option2>)).
[picocli DEBUG] Creating new user object of type class org.janelia.saalfeldlab.paintera.TestComposite$OuterGroup$Inner for group (--option1=<option1> --option2=<option2>)
[picocli DEBUG] Created org.janelia.saalfeldlab.paintera.TestComposite$OuterGroup$Inner@2b80d80f, invoking setter FieldBinding(org.janelia.saalfeldlab.paintera.TestComposite$OuterGroup$Inner org.janelia.saalfeldlab.paintera.TestComposite$OuterGroup.inner) with scope Scope(value=org.janelia.saalfeldlab.paintera.TestComposite$OuterGroup@6fffcba5)
[picocli DEBUG] Initializing --option1=<option1> in group (--option1=<option1> --option2=<option2>): setting scope to user object org.janelia.saalfeldlab.paintera.TestComposite$OuterGroup$Inner@2b80d80f and initializing initial and default values
[picocli DEBUG] Initial value not available for field String org.janelia.saalfeldlab.paintera.TestComposite$OuterGroup$Inner.option1
[picocli DEBUG] Initializing --option2=<option2> in group (--option1=<option1> --option2=<option2>): setting scope to user object org.janelia.saalfeldlab.paintera.TestComposite$OuterGroup$Inner@2b80d80f and initializing initial and default values
[picocli DEBUG] Initial value not available for field String org.janelia.saalfeldlab.paintera.TestComposite$OuterGroup$Inner.option2
[picocli DEBUG] Initialization complete for group (--option1=<option1> --option2=<option2>)
[picocli INFO] Setting field String org.janelia.saalfeldlab.paintera.TestComposite$OuterGroup$Inner.option2 to '1' (was 'null') for option --option2

and or --add-group only:

[picocli DEBUG] Creating CommandSpec for object of class org.janelia.saalfeldlab.paintera.TestComposite with factory picocli.CommandLine$DefaultFactory
[picocli INFO] Picocli version: 4.0.0-alpha-1, JVM: 1.8.0_212 (Oracle Corporation OpenJDK 64-Bit Server VM 25.212-b01), OS: Linux 5.0.4-arch1-1-ARCH amd64
[picocli INFO] Parsing 1 command line args [--add-group]
[picocli DEBUG] Parser configuration: posixClusteredShortOptionsAllowed=true, stopAtPositional=false, stopAtUnmatched=false, separator=null, overwrittenOptionsAllowed=false, unmatchedArgumentsAllowed=false, expandAtFiles=true, atFileCommentChar=#, useSimplifiedAtFiles=false, endOfOptionsDelimiter=--, limitSplit=false, aritySatisfiedByAttachedOptionParam=false, toggleBooleanFlags=true, unmatchedOptionsArePositionalParams=false, collectErrors=false,caseInsensitiveEnumValuesAllowed=false, trimQuotes=false, splitQuotedStrings=false
[picocli DEBUG] (ANSI is disabled by default: isatty=false, XTERM=null, OSTYPE=null, isWindows=false, JansiConsoleInstalled=false, ANSICON=null, ConEmuANSI=null, NO_COLOR=null, CLICOLOR=null, CLICOLOR_FORCE=null)
[picocli DEBUG] Initializing org.janelia.saalfeldlab.paintera.TestComposite: 3 options, 0 positional parameters, 0 required, 1 groups, 0 subcommands.
[picocli DEBUG] Processing argument '--add-group'. Remainder=[]
[picocli DEBUG] '--add-group' cannot be separated into <option>=<option-parameter>
[picocli DEBUG] Found option named '--add-group': field boolean org.janelia.saalfeldlab.paintera.TestComposite$OuterGroup.addGroup, arity=0
[picocli INFO] Adding multiple to MatchedGroup [--add-group (--option1=<option1> --option2=<option2>)]...={} (group=1 [--add-group (--option1=<option1> --option2=<option2>)]...).
[picocli DEBUG] Creating new user object of type class org.janelia.saalfeldlab.paintera.TestComposite$OuterGroup for group [--add-group (--option1=<option1> --option2=<option2>)]...
[picocli DEBUG] Created org.janelia.saalfeldlab.paintera.TestComposite$OuterGroup@6fffcba5, invoking setter FieldBinding(java.util.List org.janelia.saalfeldlab.paintera.TestComposite.outerList) with scope Scope(value=org.janelia.saalfeldlab.paintera.TestComposite@34340fab)
[picocli DEBUG] Initializing --add-group in group [--add-group (--option1=<option1> --option2=<option2>)]...: setting scope to user object org.janelia.saalfeldlab.paintera.TestComposite$OuterGroup@6fffcba5 and initializing initial and default values
[picocli DEBUG] Initial value not available for field boolean org.janelia.saalfeldlab.paintera.TestComposite$OuterGroup.addGroup
[picocli DEBUG] Setting scope for subgroup (--option1=<option1> --option2=<option2>) with setter=FieldBinding(org.janelia.saalfeldlab.paintera.TestComposite$OuterGroup$Inner org.janelia.saalfeldlab.paintera.TestComposite$OuterGroup.inner) in group [--add-group (--option1=<option1> --option2=<option2>)]... to user object org.janelia.saalfeldlab.paintera.TestComposite$OuterGroup@6fffcba5
[picocli DEBUG] Initialization complete for group [--add-group (--option1=<option1> --option2=<option2>)]...
[picocli INFO] Setting field boolean org.janelia.saalfeldlab.paintera.TestComposite$OuterGroup.addGroup to 'true' (was 'false') for option --add-group
Exception in thread "main" picocli.CommandLine$MissingParameterException: Error: Missing required argument(s): [--add-group (--option1=<option1> --option2=<option2>)]...

remkop added a commit that referenced this issue Apr 1, 2019
@remkop
Copy link
Owner

remkop commented Apr 1, 2019

Thanks for raising this!
I fixed one arg group validation bug in master, but found another one. Added a failing test but not able to fix it yet, I suspect the validation logic needs to be restructured.

@remkop
Copy link
Owner

remkop commented Apr 2, 2019

I raised #655 for the new issue I found.
If you find any other issues, would you mind raising separate tickets for them? That makes it easier for me to track the remaining todo items.

@hanslovsky
Copy link
Author

If you find any other issues, would you mind raising separate tickets for them?

Will do. Unfortunately, I am rather busy writing up my thesis right now, so I will most likely not have a lot of time for extensive testing.

@remkop
Copy link
Owner

remkop commented Apr 3, 2019

No problem. Good luck with the thesis!

@remkop remkop added the theme: parser An issue or change related to the parser label Apr 2, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
status: duplicate 👨🏻‍🤝‍👨🏻 A duplicate of another issue theme: parser An issue or change related to the parser
Projects
None yet
Development

No branches or pull requests

2 participants