Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove stratified group k fold #101

Merged
merged 2 commits into from
Apr 2, 2024
Merged

Conversation

L-M-Sherlock
Copy link
Member

According to the recent result of benchmark, splitting dataset into several subsets, optimizing FSRS on them and averaging the parameters don't perform better than non-splitting. So I removed it in this PR. It also makes the optimization faster.

Model: FSRS-4.5-split
Total number of users: 19993
Total number of reviews: 728883020
Weighted average by reviews:
FSRS-4.5 LogLoss (mean±std): 0.3293±0.1538
FSRS-4.5 LogLoss: 0.33±0.006
FSRS-4.5 RMSE(bins) (mean±std): 0.0569±0.0358
FSRS-4.5 RMSE(bins): 0.057±0.0011

Weighted average by log(reviews):
FSRS-4.5 LogLoss (mean±std): 0.3544±0.1723
FSRS-4.5 LogLoss: 0.354±0.0031
FSRS-4.5 RMSE(bins) (mean±std): 0.0783±0.0506
FSRS-4.5 RMSE(bins): 0.078±0.0009

Weighted average by users:
FSRS-4.5 LogLoss (mean±std): 0.3583±0.1749
FSRS-4.5 LogLoss: 0.358±0.0031
FSRS-4.5 RMSE(bins) (mean±std): 0.0817±0.0527
FSRS-4.5 RMSE(bins): 0.082±0.0010

weights: [0.6646, 1.7213, 5.18, 12.4195, 5.1978, 1.2412, 0.8733, 0.0467, 1.617, 0.1342, 1.0166, 2.0894, 0.0871, 0.3185, 1.5606, 0.219, 2.8747]

Model: FSRS-4.5-non-split
Total number of users: 19993
Total number of reviews: 728883020
Weighted average by reviews:
FSRS-4.5-old LogLoss (mean±std): 0.3293±0.1538
FSRS-4.5-old LogLoss: 0.33±0.006
FSRS-4.5-old RMSE(bins) (mean±std): 0.0567±0.0359
FSRS-4.5-old RMSE(bins): 0.057±0.0011

Weighted average by log(reviews):
FSRS-4.5-old LogLoss (mean±std): 0.3544±0.1726
FSRS-4.5-old LogLoss: 0.354±0.0031
FSRS-4.5-old RMSE(bins) (mean±std): 0.0780±0.0507
FSRS-4.5-old RMSE(bins): 0.078±0.0009

Weighted average by users:
FSRS-4.5-old LogLoss (mean±std): 0.3583±0.1752
FSRS-4.5-old LogLoss: 0.358±0.0032
FSRS-4.5-old RMSE(bins) (mean±std): 0.0815±0.0528
FSRS-4.5-old RMSE(bins): 0.081±0.0010

weights: [0.6646, 1.7213, 5.18, 12.4195, 5.201, 1.2504, 0.8733, 0.041, 1.6164, 0.1336, 1.0205, 2.0841, 0.0865, 0.3184, 1.5798, 0.219, 2.8852]

@L-M-Sherlock L-M-Sherlock added the enhancement New feature or request label Apr 2, 2024
@L-M-Sherlock L-M-Sherlock merged commit e81789e into main Apr 2, 2024
@L-M-Sherlock L-M-Sherlock deleted the remove-StratifiedGroupKFold branch April 2, 2024 02:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant