Json skeleton

[kyleknap]

This pull request made a couple changes and feature additions.
I added two new custom arguments:

• --generate-cli-skeleton: This uses botocore ArgumentGenerator to print a JSON skeleton to standard output that can be used as input to the cli. This JSON can be redirected to a file and then filled out and used as input to the CLI.
• --cli-input-json: This takes JSON (typically generated from --generate-cli-skeleton) from either a string or some file and uses that as the input to the command line operation. Additional arguments can be added to command line. If there are collisions between the two, the value from argument is used over the value from the input JSON

To create these features, I made the following changes to the cli driver:

• enable/disable_call_operation: These are methods that can be called to prevent the ServiceOperation from performing the botocore service operation call.
• emit the event: building-argument-table-parser. This gives us the ability to determine all of the arguments that are going to be passed to the parser (including arguments that were added through a hook). This was needed to ensure the parser does not throw errors for missing required args.
• emit the event: calling-service-operation. This allows us to see the entire configuration of the ServiceOperation before a call is made to botocore. That way we can edit the parameters that are going to be included in the call or even determine if we want the call to happen.

cc @jamesls @danielgtaylor

[danielgtaylor]

May want to add a comment about why we return 0 here, and determine whether zero should be used over e.g. None. Does it matter?

[kyleknap]

I think it is the return code, but I should check to make sure. Also, I agree that a comment would be good to explain why zero is returned.

[kyleknap]

I checked. The invoke() method returns zero at the end.
https://github.com/aws/aws-cli/blob/develop/awscli/clidriver.py#L567

[danielgtaylor]

Overall I think this looks really good, just had a few questions about some edge cases and some uncovered use-cases. 👍

[kyleknap]

Other than the strange build failure, the code is ready to get looked at again. I incorporated all the feedback given except for the extra feature to --generate-cli-skeleton

[jamesls]

I was just trying this out, and it looks like we have a bug for some existing CLI arguments where we can't set the required property:

$ $ aws ec2 run-instances --generate-cli-skeleton

2014-10-24 16:35:36,270 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.ec2.run-instances: calling handler <function add_generate_skeleton at 0x101698938>
2014-10-24 16:35:36,271 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.ec2.run-instances: calling handler <function add_cli_input_json at 0x1016985f0>
2014-10-24 16:35:36,271 - MainThread - botocore.hooks - DEBUG - Event building-argument-table-parser.ec2.run-instances: calling handler <bound method GenerateCliSkeletonArgument.override_required_args of <awscli.customizations.generatecliskeleton.GenerateCliSkeletonArgument object at 0x101c84f10>>
2014-10-24 16:35:36,271 - MainThread - awscli.clidriver - DEBUG - Exception caught in main()
Traceback (most recent call last):
  File "aws-cli/awscli/clidriver.py", line 206, in main
    return command_table[parsed_args.command](remaining, parsed_args)
  File "aws-cli/awscli/clidriver.py", line 354, in __call__
    return command_table[parsed_args.operation](remaining, parsed_globals)
  File "aws-cli/awscli/clidriver.py", line 462, in __call__
    self._emit(event, argument_table=self.arg_table, args=args)
  File "aws-cli/awscli/clidriver.py", line 557, in _emit
    return session.emit(name, **kwargs)
  File "botocore/botocore/session.py", line 712, in emit
    return self._events.emit(event_name, **kwargs)
  File "botocore/botocore/hooks.py", line 152, in emit
    response = handler(**kwargs)
  File "aws-cli/awscli/customizations/arguments.py", line 53, in override_required_args
    argument_table[arg_name].required = False
AttributeError: can't set attribute
2014-10-24 16:35:36,272 - MainThread - awscli.clidriver - DEBUG - Exiting with rc 255

can't set attribute

I'm guessing this is because some customizations create their own custom argument classes they may not adhere to the CLIArgument interface.

[jamesls]

It was a little confusing at first to call this event build-argument-table-parser, but we don't actually send the parser in the event. Compare to events like building-command-table and building-argument-table where we send the object being built. Maybe we can just rename this?

[kyleknap]

Yeah that makes sense.

[jamesls]

I wrote some some integration tests that test we get valid JSON output for any command supports generate-cli-skeleton: jamesls@c350ae3

Feel free to pull it into your branch to help with testing. The things it found were mostly:

• input_shape of None generating an error
• any customization that created a required property that was read only.

[kyleknap]

I updated the code such that it passes all of the generate cli skeleton integration tests. Changes include making it capable to set required for all customizations and able to handle customized arguments that made the assumption that they would always receive a non-None value. Should be good to look at again.

[kyleknap]

I further refactored the clidriver code to make the events more general. Ready to be looked at again.

cc @jamesls @danielgtaylor

[coveralls]

Coverage increased (+0.07%) when pulling a16c878 on kyleknap:json-skeleton into 4ac66b2 on aws:develop.

[jamesls]

Looks good. I like the generalized events to clidriver.