[SPARK-3934] [SPARK-3918] [mllib] Bug fixes for RandomForest, DecisionTree

[jkbradley]

SPARK-3934: When run with a mix of unordered categorical and continuous features, on multiclass classification, RandomForest fails. The bug is in the sanity checks in getFeatureOffset and getLeftRightFeatureOffsets, which use the wrong indices for checking whether features are unordered.
Fix: Remove the sanity checks since they are not really needed, and since they would require DTStatsAggregator to keep track of an extra set of indices (for the feature subset).

Added test to RandomForestSuite which failed with old version but now works.

SPARK-3918: Added baggedInput.unpersist at end of training.

Also:

• I removed DTStatsAggregator.isUnordered since it is no longer used.
• DecisionTreeMetadata: Added logWarning when maxBins is automatically reduced.
• Updated DecisionTreeRunner to explicitly fix the test data to have the same number of features as the training data. This is a temporary fix which should eventually be replaced by pre-indexing both datasets.
• RandomForestModel: Updated toString to print total number of nodes in forest.
• Changed Predict class to be public DeveloperApi. This was necessary to allow users to create their own trees by hand (for testing).

CC: @mengxr @manishamde @chouqin @codedeft Just notifying you of these small bug fixes.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amp.lab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21695/
Test FAILed.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21700/
Test FAILed.

[chouqin]

@jkbradley Thanks for the PR! It looks good to me.

[manishamde]

@jkbradley Thanks. LGTM!

[mengxr]

test this please

[SparkQA]

QA tests have started for PR 2785 at commit e116473.

• This patch merges cleanly.

[SparkQA]

QA tests have finished for PR 2785 at commit e116473.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • class Predict(

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21715/
Test PASSed.

[mengxr]

Could you try to merge master? This is already covered in #2775.

[jkbradley]

@chouqin @manishamde @mengxr Thanks for taking a look! I think stuff is fixed.

[SparkQA]

QA tests have started for PR 2785 at commit 9132321.

• This patch merges cleanly.

[SparkQA]

QA tests have finished for PR 2785 at commit 9132321.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • class Predict(

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21864/
Test PASSed.

[mengxr]

LGTM. Merged into master. Thanks!