fix: working timeout with celery kill and logic fix

[dpgaspar]

SUMMARY

Fixes logic for working timeout, adds a Celery Soft time out and an Hard time out associated with the working timeout.

Note: Killing the task does not guarantee the query is canceled on the analytics DB

[2021-04-01 12:29:23,298: INFO/MainProcess] Received task: reports.scheduler[1c635a18-091d-4c9a-b15f-7445e5e28e2d]
[2021-04-01 12:29:43,715: WARNING/MainProcess] Soft time limit (101s) exceeded for reports.execute[6f8adb2d-0395-4009-b8bd-fcb219315960]
[2021-04-01 12:29:52,720: ERROR/MainProcess] Task handler raised error: TimeLimitExceeded(110)
Traceback (most recent call last):
  File "/Users/daniel/workarea/preset/preset_fork_superset/venv3/lib/python3.7/site-packages/billiard/pool.py", line 684, in on_hard_timeout
    raise TimeLimitExceeded(job._timeout)
billiard.exceptions.TimeLimitExceeded: TimeLimitExceeded(110,)
[2021-04-01 12:29:52,720: ERROR/MainProcess] Hard time limit (110s) exceeded for reports.execute[6f8adb2d-0395-4009-b8bd-fcb219315960]
[2021-04-01 12:29:52,905: INFO/MainProcess] Received task: reports.scheduler[cfc7d8fb-ca4e-42f4-b1db-821fa170e806]
[2021-04-01 12:29:55,241: ERROR/MainProcess] Process 'ForkPoolWorker-9' pid:8042 exited with 'signal 9 (SIGKILL)'

ADDITIONAL INFORMATION

• Has associated issue:
• Changes UI
• Includes DB Migration (follow approval process in SIP-59)
  • Migration is atomic, supports rollback & is backwards-compatible
  • Confirm DB migration upgrade and downgrade tested
  • Runtime estimates and downtime expectations provided
• Introduces new feature or API
• Removes existing feature or API

[codecov]

Codecov Report

Merging #13911 (fbbc053) into master (9156f08) will increase coverage by 1.35%.
The diff coverage is 90.67%.

❗ Current head fbbc053 differs from pull request most recent head cb8a29f. Consider uploading reports for the commit cb8a29f to get more accurate results

@@            Coverage Diff             @@
##           master   #13911      +/-   ##
==========================================
+ Coverage   77.83%   79.18%   +1.35%     
==========================================
  Files         934      936       +2     
  Lines       47320    47427     +107     
  Branches     5913     5940      +27     
==========================================
+ Hits        36831    37555     +724     
+ Misses      10346     9745     -601     
+ Partials      143      127      -16     

Flag	Coverage Δ
cypress	56.04% <73.83%> (+0.01%)	⬆️
hive	?
mysql	80.78% <85.25%> (+0.21%)	⬆️
postgres	80.82% <85.25%> (+0.21%)	⬆️
presto	80.51% <89.10%> (+0.24%)	⬆️
python	76.16% <84.54%> (-4.96%)	⬇️
sqlite	75.52% <81.36%> (-4.65%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
...tend/src/SqlLab/components/ScheduleQueryButton.tsx	28.12% <ø> (ø)
...et-frontend/src/SqlLab/components/TableElement.jsx	88.60% <ø> (+0.23%)	⬆️
...end/src/common/components/DropdownButton/index.tsx	24.00% <0.00%> (ø)
...rset-frontend/src/components/ErrorMessage/types.ts	100.00% <ø> (ø)
...rset-frontend/src/components/ProgressBar/index.tsx	100.00% <ø> (ø)
.../components/CrossFilterScopingModal/utils/index.ts	100.00% <ø> (ø)
...src/dashboard/components/PublishedStatus/index.jsx	100.00% <ø> (ø)
...dashboard/components/SliceHeaderControls/index.jsx	77.89% <ø> (ø)
...shboard/components/filterscope/FilterFieldItem.jsx	100.00% <ø> (+50.00%)	⬆️
...shboard/components/filterscope/FilterFieldTree.jsx	66.66% <ø> (+22.22%)	⬆️
... and 129 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9156f08...cb8a29f. Read the comment docs.

[dpgaspar]

Adding a log info with the time a query took to execute