[BUG] Audit and fix/test hasSideEffects

[revans2]

Describe the bug
we added in hasSideEffects a long time ago to be able to deal with conditional expressions where an exception might be thrown. Spark does lazy evaluation on the CPU but we tend to evaluation both sides greedily. We detect when we need to be lazy based on the hasSideEffects method. But this does not look like it was properly maintained. Things like GpuBinaryArithmetic expressions that just check for ANSI mode instead of checking to see if failOnError returns true.

This ends up being both a performance problem in some cases and a correctness problem in others.

Steps/Code to reproduce bug
Bitwise operations in ANSI mode says it could throw an exception, but it cannot.

GpuAdd in ANSI mode says it has side effects in cases it does not.

I just don't trust that all of our expressions are doing the right thing here and we need to check them all and add tests to verify that they are doing what we expect.