This library extracts pyspark decision tree paths for a given input.
Spark decision tree models such as random forest and gbdt classifiers are implemented in Scala. No APIs are provided to extract decision paths for a given input. In some applications, explicit decision tree paths are useful for non-expert users to understand how decisions are made.
In this repository, we provide a simple utility to convert spark decision tree (pyspark.ml.classification.DecisionTreeClassificationModel or pyspark.ml.regression.DecisionTreeRegressionModel) in scala to tree in python and extract decision paths for a given input. It is also useful if you like to port spark tree model into non-spark environment.
This library can be used to extract decision paths for pyspark.ml.classification.{RandomForestClassificationModel, GBTClassificationModel} and pyspark.ml.regression.{RandomForestRegressionModel, GBTRegressionModel}.
1. Create wheel package
python setup.py bdist_wheel
2. Upload the wheel package into databrick cluster
3. Run the example notebook in databricks
```
from pyspark_tree_decision_paths import tree, decision_path
python_tree_model = tree.build_tree(spark_tree_model)
predicted_paths = decision_path.extract_decision_tree_paths(python_tree_model, x)
print_decision_paths(predicted_paths)
#decison path outputs look like for binary classification ( <=50K, >50K) on [UCI Adult dataset](https://archive.ics.uci.edu/ml/datasets/adult)
# feature_col_name(feature_value)
Feature relationship( Husband) is not in [" Not-in-family", " Own-child", " Unmarried", " Other-relative"]
Feature education-num(10.0) <= 12.5
Feature capital-gain(7688.0) > 6808.0
Leaf prediction: >50K
```