Add e2e tests for common plugins to test compatibility

[noahnu]

Is your feature request related to a problem? Please describe.

We had a regression with pytest-rerunfailures here: #879. This isn't the first tool compatibility bug we've had.

Describe the solution you'd like

A suite of e2e tests where we install various plugins to test for compatibility. This shouldn't be part of the standard test suite since some of these plugins might have conflicting dependencies.