SmartFlank: Use average test time for new tests when possible

[CristianGM]

Fixes #977

Test Plan

How do we know the code works?
I've modified the previous tests to show it's behavior change

Checklist

• Documented
• Unit tested
• release_notes.md updated

[jan-goral]

Hey, great thanks for your contribution! This improvement looks really smart. We have to consider if we can merge it as is or integrate as an additional option because it is changing default behavior a little.

@bootstraponline #635 (comment) still we don't want an option for it?

[CristianGM]

Adding a flag for average would be nice, and IMHO adding a parameter for the default was a must.
I understand trying to limit the number of parameters but it seems the difference on what's the "default" for different devs is too big to just say 10s or 120s.

I'm assuming here that for a project all the tests are kind of the same (for sure there will be a variance, but I don't expect it to be minutes). Having a slightly bigger shard shouldn't be an issue, even smaller is fine if it's not under 1 minute so we have the $$$ issue (just paying for unused seconds, no drama but it's still $).

[bootstraponline]

@bootstraponline #635 (comment) still we don't want an option for it?

@jan-gogo Thanks for this comment Jan! I think you're correct we should have an option. I hoped that we could select one value for all the users, however an option makes the most sense.

[bootstraponline]

Adding a flag for average would be nice, and IMHO adding a parameter for the default was a must.

Agreed that we should have a parameter for the default. Thanks for sharing your use case so we can improve Flank!

[CristianGM]

So...
Parameter for the default?
boolean parameter for using average "prediction"?

Do you want me to do the changes? or is it something you want to do? (first time i look at Flank code, but I guess adding the parameter won't be too complex, naming may be)

[bootstraponline]

Do you want me to do the changes? or is it something you want to do? (first time i look at Flank code, but I guess adding the parameter won't be too complex, naming may be)

I think it'll be faster to have the team do the changes. We'll discuss the best way to implement this.

[bootstraponline]

I'm closing this pull request for now. We discussed as a team and prioritized the ticket in the ranked column on ZenHub. There are a few higher priority tasks that'll be worked on first (such as fixing the rate limit issue).