Improve Handling on multiple commits

[Daltz333]

Details

• Read the Docs project URL: https://frc-docs.readthedocs.io/en/latest/
• Build URL (if applicable): https://readthedocs.org/projects/frc-docs/builds/12136540/
• Read the Docs username (if applicable): daltz333

Expected Result

When merging multiple PRs in a short time-spam, ReadTheDocs often freaks out and gets stuck on the "triggered" page, refusing to build anything for a long period of time (at the time of writing, 35 minutes+)

Actual Result

Cleanly cancel the old builds, and build the new one without breaking.

[Daltz333]

Additionally force-pushes massively break PRs, and they are incredibly common due to rebase merges. I'd consider this a bug and not a feature request.

[humitos]

I'm not sure exactly what happened in this case. We should probably need more debugging on this to find out a solution. However, I can tell that the build you linked (https://readthedocs.org/projects/frc-docs/builds/12136540/) has been "killed" because we didn't get back an answer regarding its state (#3312). This usually happens when the build fails for some reason that we weren't able to catch when it happened.

Cleanly cancel the old builds, and build the new one without breaking.

I think the issue was marked as a feature request because of this. We already attempt to cancel a running build and we found out that it wasn't simple, #7031

Unfortunately, I don't have a good answer/workaround to give at the moment.

[humitos]

I think this problem was already solved when we implemented "duplicated build error" that are immediately canceled. Have you experimented this problem recently, @Daltz333 ?

[Daltz333]

Yes. Even just recently.

[Daltz333]

I pushed multiple commits to multiple PRs. What tends to happen, still, is that one PR will get started, and the rest will be "queued" saying there is 4 already building (even if there is not).

In the below screenshot, you can see one build failed (it says due to inactivity, but that's a lie) and two are currently queued. There is 2 possible outcomes that will happen, eventually RTD will figure it out and things will be built again, or these two builds will fail to "inactivity".

It seems like the check that happens every 5 minutes to determine concurrent builds is invalid or incorrect in some way.

Here is a link to the below failed build, https://readthedocs.org/projects/frc-docs/builds/15963261/. I would like to say this is a rare occurrence, but realistically it is uncommon at best. I do think this is something to do with the logic behind detecting the concurrent builds not be correct (and thus causing builds to infinitely think things are being built when they are not)

[humitos]

I plan to come back to this issue in the near future and try to fix it. Your description is pretty good, but I'd like to have a programmatic way to reproduce it, so I can create a test case and execute it locally over and over again until I find exactly where is our problem.

Do you know the exact git commands that I would need to execute starting from an empty/new repository to reproduce this behavior? I think I'm probably asking too much to you, but since you understand better than myself this issue at this point I didn't want to miss the opportunity of asking this to you 😄

[Daltz333]

I'll try to set up a minimal test repo and see if I can reliably trigger it.

[Daltz333]

To be fair on RTD, our documentation project is probably one of the most complex ones out there. We hit 25 minutes of build time building HTML and PDF. We have multiple custom extensions and are adding more, so we are incredibly grateful for the RTD platform to provide this to us. We are working on giving back to the Sphinx and RTD ecosystem by developing several extensions that enhance documentation that are already used by hundreds of projects:

• dynamic responsive images
• turning sphinx websites into offline PWA (this may have a bit more intensive impact on RTD servers, but we are investigating multiple caching strategies to minimize this)
• opengraph metadata
• dynamic translation metadata for search engines
• extensive rediraffe redirect extension
• and a lot more.

So again, thank everyone at RTD for making this possible!

[humitos]

In #8850 we implemented a way to cancel a build. Besides, in #8866 we plan to cancel old builds automatically when a new push is done to the same branch.

With the ability to manually cancel a build I think this issue is partially solved and it will be fully solved once automatic canceling is implemented. Please, let me know if the current solution is enough for you.

[Daltz333]

That sounds good to me!