Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Added per-class probability prediction for random forests #138

Closed
wants to merge 8 commits into from
Closed

Added per-class probability prediction for random forests #138

Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
663db03
Added per-class probability prediction for random forests
AlanRace Jul 11, 2022
2603a1f
Add test
Mec-iS Aug 24, 2022
61db4eb
Add test
Mec-iS Aug 24, 2022
b6fb819
Merge pull request #1 from smartcorelib/alanrace-predict-probs
AlanRace Aug 29, 2022
d46b830
Merge branch 'development' into predict-probability
AlanRace Aug 29, 2022
7f7b2ed
Fixed test by transposing matrix
AlanRace Aug 29, 2022
28c81eb
Test case now passing without transpose
AlanRace Aug 30, 2022
e9ed9e8
Merge remote-tracking branch 'sm/development' into predict-probability
morenol Sep 22, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
102 changes: 100 additions & 2 deletions src/ensemble/random_forest_classifier.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -55,7 +55,8 @@ use serde::{Deserialize, Serialize};

use crate::api::{Predictor, SupervisedEstimator};
use crate::error::{Failed, FailedError};
use crate::linalg::Matrix;
use crate::linalg::naive::dense_matrix::DenseMatrix;
use crate::linalg::{BaseMatrix, Matrix};
use crate::math::num::RealNumber;
use crate::rand::get_rng_impl;
use crate::tree::decision_tree_classifier::{
Expand Down Expand Up @@ -553,6 +554,37 @@ impl<T: RealNumber> RandomForestClassifier<T> {
which_max(&result)
}

/// Predict the per-class probabilties for each observation.
/// The probability is calculated as the fraction of trees that predicted a given class
pub fn predict_probs<M: Matrix<T>>(&self, x: &M) -> Result<DenseMatrix<f64>, Failed> {
Mec-iS marked this conversation as resolved.
Show resolved Hide resolved
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

in scikit it is called predict_proba, I think that it is better to keep the same name

let mut result = DenseMatrix::<f64>::zeros(x.shape().0, self.classes.len());

let (n, _) = x.shape();

for i in 0..n {
let row_probs = self.predict_probs_for_row(x, i);

for (j, item) in row_probs.iter().enumerate() {
result.set(i, j, *item);
}
}

Ok(result)
}

fn predict_probs_for_row<M: Matrix<T>>(&self, x: &M, row: usize) -> Vec<f64> {
let mut result = vec![0; self.classes.len()];

for tree in self.trees.iter() {
result[tree.predict_for_row(x, row)] += 1;
}

result
.iter()
.map(|n| *n as f64 / self.trees.len() as f64)
.collect()
}

fn sample_with_replacement(y: &[usize], num_classes: usize, rng: &mut impl Rng) -> Vec<usize> {
let class_weight = vec![1.; num_classes];
let nrows = y.len();
Expand All @@ -578,7 +610,7 @@ impl<T: RealNumber> RandomForestClassifier<T> {
}

#[cfg(test)]
mod tests {
mod tests_prob {
use super::*;
use crate::linalg::naive::dense_matrix::DenseMatrix;
use crate::metrics::*;
Expand Down Expand Up @@ -742,4 +774,70 @@ mod tests {

assert_eq!(forest, deserialized_forest);
}

#[cfg_attr(target_arch = "wasm32", wasm_bindgen_test::wasm_bindgen_test)]
#[test]
fn fit_predict_probabilities() {
let x = DenseMatrix::<f64>::from_2d_array(&[
&[5.1, 3.5, 1.4, 0.2],
&[4.9, 3.0, 1.4, 0.2],
&[4.7, 3.2, 1.3, 0.2],
&[4.6, 3.1, 1.5, 0.2],
&[5.0, 3.6, 1.4, 0.2],
&[5.4, 3.9, 1.7, 0.4],
&[4.6, 3.4, 1.4, 0.3],
&[5.0, 3.4, 1.5, 0.2],
&[4.4, 2.9, 1.4, 0.2],
&[4.9, 3.1, 1.5, 0.1],
&[7.0, 3.2, 4.7, 1.4],
&[6.4, 3.2, 4.5, 1.5],
&[6.9, 3.1, 4.9, 1.5],
&[5.5, 2.3, 4.0, 1.3],
&[6.5, 2.8, 4.6, 1.5],
&[5.7, 2.8, 4.5, 1.3],
&[6.3, 3.3, 4.7, 1.6],
&[4.9, 2.4, 3.3, 1.0],
&[6.6, 2.9, 4.6, 1.3],
&[5.2, 2.7, 3.9, 1.4],
]);
let y = vec![
0., 0., 0., 0., 0., 0., 0., 0., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
];

let classifier = RandomForestClassifier::fit(
&x,
&y,
RandomForestClassifierParameters {
criterion: SplitCriterion::Gini,
max_depth: None,
min_samples_leaf: 1,
min_samples_split: 2,
n_trees: 100,
m: Option::None,
keep_samples: false,
seed: 87,
},
)
.unwrap();

println!("{:?}", classifier.classes);

let results = classifier.predict_probs(&x).unwrap();
println!("{:?}", x.shape());
println!("{:?}", results);
println!("{:?}", results.shape());

assert_eq!(
results,
DenseMatrix::<f64>::from_array(
20,
2,
&[
1.0, 0.0, 0.78, 0.22, 0.95, 0.05, 0.82, 0.18, 1.0, 0.0, 0.92, 0.08, 0.99, 0.01,
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are these the expected values?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes. those are the results as returned by the test. they match among all the targets except Wasm.

Copy link
Collaborator

@morenol morenol Sep 24, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

They are failing for me locally, and they also failed in the CI for x86_64-unknown-linux-gnu,

I think that the green checks in the CI are only in the CI jobs that builds the crate but not run tests (32 bits arch)

0.96, 0.04, 0.36, 0.64, 0.33, 0.67, 0.02, 0.98, 0.02, 0.98, 0.0, 1.0, 0.0, 1.0,
0.0, 1.0, 0.0, 1.0, 0.03, 0.97, 0.05, 0.95, 0.0, 1.0, 0.02, 0.98
]
)
);
}
}