[1.0-beta4 -> main] Watermark violation should create speculative block

[heifner]

Failing watermark validation would cause node to go into tight spin:

error 2024-08-01T19:05:54.914 nodeos    producer_plugin.cpp:1893      start_block          ] Not producing block because "defproducerd" signed a block at the next block time or later (2024-08-01T19:05:55.000) than the pending block time (2024-08-01T19:05:55.000)
debug 2024-08-01T19:05:54.914 nodeos    producer_plugin.cpp:2554      schedule_production_ ] Waiting till another block is received and scheduling Speculative/Production Change
debug 2024-08-01T19:05:54.914 nodeos    producer_plugin.cpp:2628      schedule_delayed_pro ] Scheduling Speculative/Production Change at 2024-08-01T19:05:54.887
... [snip over 2000 entries]
error 2024-08-01T19:05:54.999 nodeos    producer_plugin.cpp:1893      start_block          ] Not producing block because "defproducerd" signed a block at the next block time or later (2024-08-01T19:05:55.000) than the pending block time (2024-08-01T19:05:55.000)
debug 2024-08-01T19:05:54.999 nodeos    producer_plugin.cpp:2554      schedule_production_ ] Waiting till another block is received and scheduling Speculative/Production Change
debug 2024-08-01T19:05:55.000 nodeos    producer_plugin.cpp:2628      schedule_delayed_pro ] Scheduling Speculative/Production Change at 2024-08-01T19:05:55.350

Move block time check in speculative mode above watermark check in production mode.

Merges release/1.0-beta4 into main including #471

Reolves: AntelopeIO/leap#2380

[ericpassmore]

Note:start
group: STABILITY
category: INTERNALS
summary: Move block time check in speculative mode above watermark check in production mode.
Note:end