test_optimizer.test_signum has fixed seed that can mask flakiness

[szha]

The unit test in title have been using fixed seed to mask flakiness. Suggested action:

1. Evaluate whether the test is flaky without fixed seed. If not, remove seed. Else move to 2
2. If test is flaky, determine whether it's an actual uncaught edge case. If so, fix the operator. Else move to 3
3. If numerical instability is inevitable, adjust tolerance level appropriately.

[apeforest]

Thannks for filing this issue. We will investigate this Flaky test

[haojin2]

@szha Related PR is merged, should be good to close.

[haojin2]

@sandeep-krishnamurthy Related PR is merged, should be good to close.