Incremental runs are worse than full runs

[krausest]

There are two kind of runs:

• Full runs. For a new chrome version I'm running all frameworks on my machine. The first night only the keyed implementations and when that looks good the second night the non-keyed implementations. Before running the keyed implementations I'm rebooting and stopping all the helpers (BetterDisplay, OneDrive, Karabiner Elements), stopping the search index (sudo mdutil -i off /), and making sure that there are no active apps running on the machine (other browser instances, VSCode etc.)
• Incremental runs. For updated implementations I'm running vanillajs and the updated implementations. I'm stopping all active applications, but neither rebooting nor stopping the helper apps. vanillajs serves as a reference whether the achievable performance looks right. Sometimes it doesn't. It's often the case that results are worse than in a full run.

This issue tracks the impact of the measures for the full run. Maybe we can work out what's causing the effect.

[krausest]

For the first update since chrome 127 I decided to prepare the machine like for a full run. Results are pretty close and show how much noise is in the measurement.
Left incremental run, right last official run:

[krausest]

With indexing enabled and on battery, every else like like in a full run. A bit slower!

[krausest]

Once again, with indexing disabled and on AC:

Really hard to tell if that's better than the last one. Nevertheless the full run was slightly better...

[krausest]

Just running create 1k rows for vanillajs, but with 40 iterations for the following cases:

1. Firefox, Chrome, Mail, Notes, Onedrive, Betterdisplay, VSCode, Preview, Settings
2. Firefox, Chrome, Mail, Notes, Onedrive, Betterdisplay, Preview, Settings
3. Only: Onedrive, Betterdisplay
4. No Apps running, Onedrive stopped, Betterdisplay stopped
5. After Rebooting: No Apps running, everything stopped except indexing
6. After Reboot: No Apps running, everything stopped incl. indexing

The results aren't as clear as I hoped.

[krausest]

Huh. There was one case missing. Running on AC, all above was on battery

1. Battery: Firefox, Chrome, Mail, Notes, Onedrive, Betterdisplay, VSCode, Preview, Settings
2. Battery: Firefox, Chrome, Mail, Notes, Onedrive, Betterdisplay, Preview, Settings
3. Battery: Only: Onedrive, Betterdisplay
4. Battery: No Apps running, Onedrive stopped, Betterdisplay stopped
5. Battery: After Rebooting: No Apps running, everything stopped except indexing
6. On AC: After Rebooting: No Apps running, everything stopped except indexing
7. Battery: After Reboot: No Apps running, everything stopped incl. indexing
   

So, running on AC instead of on battery is the key. I really thought the mac was immune to that.
Full runs were always on battery, incrementals runs were often on battery.

[robbiespeed]

I wonder if there's any variation caused from some incremental runs executing on efficiency cores. I know I've seen it a fair amount on Linux with my intel cpus when running benchmarks, so I use something like taskset -c 0-7 program [args] to set affinity to only performance cores.

The closest ability I found for macos is taskpolicy -c utility program [args], from what I've read that should favour p-cores, but still has a chance to run on e-cores. From a clean boot with no other programs it'll likely run on p-cores anyway, so I'd expect full runs to be safe.

Or maybe no runs are happening on the e-cores anyway and mac being a holistic platform just has really good defaults for cpu affinity handling out of the box.