[ADAM-864] Don't force shuffle if reducing partition count.

[fnothaft]

Resolves #864. In Spark, coalescing will reduce the number of partitions in an
RDD without performing a shuffle, but coalescing will only increase the number
of partitions if a shuffle is performed. This PR modifies Transform and Vcf2ADAM
to check whether the coalesce option will increase or decrease the partition
count. Additionally, it adds a flag that allows the user to force a shuffle;
this may be desirable as this causes a HashPartitioned shuffle, which may
improve the balance of records across partitions.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/ADAM-prb/1000/
Test PASSed.

[heuermh]

Could you add ADAM2Fasta in this pull request? It uses -partitions and -shuffle as argument names instead of -coalesce and -force_shuffle_coalesce, respectively.

[fnothaft]

@heuermh Done!

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/ADAM-prb/1001/
Test PASSed.

[heuermh]

LGTM, thanks.

[heuermh]

Can I merge this? If so, is the Merge github button still ok? If I remember correctly, the commit-pr.sh script won't work for me because I'm using git over https.

[fnothaft]

I thought I'd pushed a fix for the commit-pr.sh issue? Let me rebase this first... or, do you just want me to merge it?

[heuermh]

commit-pr.sh hasn't been revised yet.

Rebase and I'll hit the button, no need to bend the don't-merge-your-own-pull-requests rule. :)

[fnothaft]

Rebased!

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/ADAM-prb/1010/
Test PASSed.

[heuermh]

Thanks!