APQ test fails occasionally

[BrynCooke]

Very occasionally the APQ test will fail for me.
The risk is that this is something that is not in test code and could hit folks in prod. Needs investigating.

test automated_persisted_queries ... FAILED


failures:

---- automated_persisted_queries stdout ----
thread 'automated_persisted_queries' panicked at 'attempt to add with overflow', /rustc/9d1b2106e23b1abd32fce1f17267604a5102f57a/library/core/src/num/mod.rs:830:5
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
thread 'automated_persisted_queries' panicked at 'called `Result::unwrap()` on an `Err` value: Closed', apollo-router/tests/integration_tests.rs:499:48

[garypen]

I can't reproduce this. This test has been impacted by the changes in the recent plugin utils updates, so maybe there was an intermittent problem which was fixed as part of this change?

I've been running:

while true; do RUST_BACKTRACE=1 cargo test automated_persisted_queries 2>/tmp/ERRORS | grep persisted_queries || break; done

all afternoon on my laptop and no fails.

One other possibility: my laptop is an M1 and it was reported on Linux (x86), so I've asked the reporter to run this loop on their laptop to see if they can trigger the fail.

[garypen]

Extensive exercising of this test has failed to trip it. We've exercised it on M1 and X86, OS X and Linux, with no occurrence. For now, I suggest we mark this wontfix/deferred and see if anyone sees this in the wild.
(NB: One environment does trip this fairly easily, but it could be a faulty environment...)

[BrynCooke]

Looks like this is is related to metrics-rs/quanta#61 via moka-rs/moka#113.
I can reproduce fairly easily on my AMD based laptop, and I have seen this once on CircleCI.

Let's see what the library maintainers think. For now I have added logging of panics, so it should be fairly easy to spot if users come across this.

Short term we may have to swap out Moka until this is resolved.