Adds support for submitting jobs with gpus

[dposada]

Changes proposed in this PR

• adding --gpus flag to cs submit
• adding an integration test for --gpus

Why are we making these changes?

This allows users to submit jobs with gpus using cs. See issue #793 for more context.

[dposada]

@dPeS - Any chance you can take this for a spin and see if it works in your environment?

[dPeS]

@dposada sure, will do next week and report here when done

[dPeS]

@dposada works as expected... but I see 1 issue here:

cs submit --gpus 1 sleep 1 works ok now, but:

cs submit sleep 1 waits infinitely since there is no host without GPU

is it expected?

[dposada]

Currently, non-gpu jobs won't get scheduled on agents with gpus (@pschorf correct me if I'm wrong)

[pschorf]

@dPeS That's correct. Currently, we have logic to prevent scheduling jobs without GPUs on GPU-enabled hosts to prevent tying up the non-GPU resources on the host. If that's an issue in your environment we can look into making that configurable.

[pschorf]

@DaoWen any feedback before I merge this?

[dPeS]

@pschorf hi, that would be useful for my case... it's not a bloker since I could spawn p2.xlarge + some non gpu instance, but in future configuration option will be cool

[dposada]

I created #803 to track this.

Thanks for testing this PR, @dPeS!