improved ambiguity computation caused latency regression

[KristofferC]

Putting #36962 (comment) into it's own issue so it can go on the milestone.

Current master:

Base  ───────  20.674068 seconds
Stdlibs: ────  31.106509 seconds 60.0718%

julia> @time using Plots
[ Info: Precompiling Plots [91a5bcdd-55d7-5caf-9e0b-520d859cae80]
 94.738177 seconds (8.06 M allocations: 545.291 MiB, 0.12% gc time)

julia> @time display(plot(rand(10)))
  7.265292 seconds (10.40 M allocations: 601.608 MiB, 3.82% gc time)

julia> @time using Plots
  4.513663 seconds (7.97 M allocations: 539.723 MiB, 2.35% gc time)

with this reverted:

Base  ───────  20.185031 seconds
Stdlibs: ────  29.825570 seconds 59.6366%

julia> @time using Plots
[ Info: Precompiling Plots [91a5bcdd-55d7-5caf-9e0b-520d859cae80]
 85.283090 seconds (6.21 M allocations: 443.919 MiB, 0.12% gc time)

julia> @time display(plot(rand(10)))
  6.373428 seconds (8.95 M allocations: 514.353 MiB, 4.12% gc time)

julia> @time using Plots
  3.221717 seconds (6.12 M allocations: 438.344 MiB, 3.30% gc time)