Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

core(metrics): update lantern coefficients #5120

Merged
merged 3 commits into from
May 8, 2018

Conversation

patrickhulce
Copy link
Collaborator

@patrickhulce patrickhulce commented May 4, 2018

This PR updates the lantern coefficients to the values that offer the lowest absolute error of the 9-run accuracy/variance dataset, they haven't ever been updated so the jump is expected. this PR also turns on the flexibleOrdering feature for the optimistic estimate which improved accuracy ~5% on FCP/FMP 🎉

In the ideal case if lantern were perfect, we would expect the optimistic/pessimistic coefficients to both be positive, sum to ~1, and the intercept to be 0. The good news is we've made considerable progress toward this goal for FCP/FMP, intercepts are significantly lower and coefficients are positive and sum to ~1.

Progress is positive on TTI, the coefficients now sum to ~1 instead of ~1.5 and the intercept is roughly the same.

Speed Index is just weird, the optimistic estimate being real speed index throws things off quite a bit. The lowest absolute error is coefficients at ~2x and then a negative intercept. This actually doesn't seem that bad since on the low end we'll be using the FCP estimate for speed index anyhow, and if it improves accuracy on the high end of speed index, we'll take it, but still not an ideal situation.

MAPE State of the World

Aside: once I add the GCP test suite I think we should shift away from measuring based on MAPE and do the search approach, % of results that are Good/OK/Terrible which is much easier to interpret and judge success

Metric MAPE MAPE on middle 50%
FCP 24.4% 9.8%
FMP 27.4% 11.0%
FCPUI 29.9% 16.2%
TTI 30.9% 16.1%
SI 39.6% 18.7%

Copy link
Member

@brendankenny brendankenny left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is all great, and important to get in for first users of 3.0. LGTM!

we should shift away from measuring based on MAPE and do the search approach, % of results that are Good/OK/Terrible which is much easier to interpret and judge success

distribution of residuals 4eva

@patrickhulce patrickhulce merged commit b69c8f6 into master May 8, 2018
@patrickhulce patrickhulce deleted the update_lantern_coefficients branch May 8, 2018 18:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants