Skip to content

Commit

Permalink
Calculate token usage in parallel via Rayon
Browse files Browse the repository at this point in the history
What?
=====

This swaps token usage calculation with Rayon to further speed up
processing.

|            | before | after | speedup |
| Codebase 1 |    778 |   621 |     25% |
| Codebase 2 |   1214 |   985 |     23% |
| Codebase 3 |  37980 | 36950 |      3% |
|  discourse |   1309 |   949 |     38% |

All times are measured in milliseconds and benchmarks are run with
[bench](https://hackage.haskell.org/package/bench) against release mode.
  • Loading branch information
joshuaclayton committed May 25, 2020
1 parent 0092fab commit 8b4f822
Show file tree
Hide file tree
Showing 3 changed files with 7 additions and 4 deletions.
1 change: 1 addition & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

3 changes: 2 additions & 1 deletion crates/token_analysis/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,8 @@ project_configuration = { path = "../../crates/project_configuration/" }
serde_json = "1.0.50"
serde = { version = "1.0.105", features = ["derive"] }
itertools = "0.9"
indicatif = {version = "0.14"}
rayon = "1.1"
indicatif = {version = "0.14", features = ["with_rayon"]}

[dev-dependencies]
totems = "0.2.7"
7 changes: 4 additions & 3 deletions crates/token_analysis/src/token_usage.rs
Original file line number Diff line number Diff line change
@@ -1,9 +1,10 @@
use super::analysis_filter::{AnalysisFilter, OrderField, SortOrder};
use super::occurrence_count::FileTypeCounts;
use super::usage_likelihood::UsageLikelihood;
use indicatif::ProgressIterator;
use indicatif::ParallelProgressIterator;
use itertools::{rev, Itertools};
use project_configuration::ProjectConfiguration;
use rayon::prelude::*;
use serde::Serialize;
use token_search::{TokenSearchConfig, TokenSearchResult, TokenSearchResults};

Expand Down Expand Up @@ -46,9 +47,9 @@ impl TokenUsageResults {
let size = &unwrapped_results.len();

let results = unwrapped_results
.into_iter()
.par_iter()
.progress_with(token_search_config.toggleable_progress_bar("🧐 Analyzing...", *size))
.map(|r| TokenUsage::new(config, r))
.map(|r| TokenUsage::new(config, r.clone()))
.collect::<Vec<_>>();
TokenUsageResults(results)
}
Expand Down

0 comments on commit 8b4f822

Please sign in to comment.