Progress doesn't work well at all

[ezyang]

I'm not sure why, but Shake's progress estimation seems to just not work very well at all for me (as in, continuously reporting 3s left perpetually through the build, or 37777m left, and not improving if I rerun the build system multiple times), with a Shake build system that is directly calling the GHC API.

Is there something I'm doing wrong? It sounds like this feature works OK for most people; is in-process causing Shake problems, or am I poisoning build times with bad data (because of recompilation avoidance), or something?

[ndmitchell]

The first time round times are usually a bit iffy. Subsequent times it's more reasonable, but recomp avoidance will make it worse. Can you --progress=record and attach the result, that dumps the raw info and can be used to produce graphs of accuracy etc.

[ezyang]

I'll upload some progress logs, but I think recomp avoidance is exactly the problem, as it means a file sometimes takes several seconds to build, and sometimes is instantaneous. Is there any way to let Shake know, "No, we didn't actually build this"? (Can Shake predict if something is going to be recomp avoided or not?)

[ezyang]

Here are a few progress files I got from running Shake, touching some intermediate source file, and then running it again.
progress1.txt
progress2.txt
progress3.txt

These are for compiling Cabal; I can also try compiling some simpler projects and see what happens.

[ndmitchell]

Thanks. I wouldn't worry about simpler projects - it's all averages and approximations, so the short builds tend to be less accurate.

[ndmitchell]

Looking at those through the profile viewer (--progress=replay=progress3.prog --report=progress.html) they look universally awful. Not just "recompilation avoidance messing with the numbers", but buggy in the extreme - the red line should ideally follow the gray line. I'll investigate properly at some point.

[ndmitchell]

So there are three entirely different stories here. The one common thread seems to be something that takes < 1e-4 seconds to build and builds immediately on starting. After that we see:

• For 3, we have 7 things to do, and we make zero progress throughout - presumably they all complete in the last second. Here there is no progress being made. We do know in advance the time taken for everything is ~100s (it was actually 66s), but I ignore that in the calculations, and don't count down unless things happen. Maybe that's a way to do better here.
• For 2, we have some things to do at the beginning and no real idea how long it's going to take, so get a silly number. After 10 seconds we actually do a very good job of tracking the progress onwards.
  
• For 1, for the first minute, only one thing has built in < 1e-4, and we know there are 7 things remaining, so we guess at the time - and the guess is consequently silly because it takes 72s for anything else to happen. For the remaining, it rarely has a good idea of how much to do, so is a bit difficult to predict.

So maybe there is something to be gained by continually decreasing the progress of running rules (makes 3 better), ignoring weirdness at the start (better say nothing for 2 than something silly), and not sure if anything can be done about 1.

[ezyang]

Hi, is there any experiments/etc you would like me to do? I'm regularly running this Shake build system and I can submit more data, vary the set of things being compiled, etc.