fix: cache-warmup fails

[nsivarajan]

SUMMARY

This PR addresses #30900, which reports a "400 Bad Request: The CSRF session token is missing" error during cache warmup. The issue was due to missing header updates with the CSRF token and cookie (expecting session='session_cookie'). This PR ensures the token is correctly fetched and included in headers during the warmup process, resolving the error.

Fixes #30900
Fixes #31283

BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

Success:

...
...
[2024-11-26 19:00:00,054: INFO/ForkPoolWorker-62] cache-warmup[cad28d1b-df9c-4800-98bd-a09ec3af5d73]: Loading strategy
[2024-11-26 19:00:00,054: INFO/ForkPoolWorker-62] cache-warmup[cad28d1b-df9c-4800-98bd-a09ec3af5d73]: Loading DashboardTagsStrategy
[2024-11-26 19:00:00,055: INFO/ForkPoolWorker-62] cache-warmup[cad28d1b-df9c-4800-98bd-a09ec3af5d73]: Success!
[2024-11-26 19:00:00,131: INFO/ForkPoolWorker-62] fetch_url[90921f09-84a0-49ee-8b66-790864c72524]: Fetching https://sample.example.com/api/v1/security/csrf_token/
[2024-11-26 19:00:00,130: INFO/ForkPoolWorker-62] fetch_url[90921f09-84a0-49ee-8b66-790864c72524]: URL 'https://sample.example.com/api/v1/chart/warm_up_cache' is secure. Adding Referer header.
[2024-11-26 19:00:00,207: INFO/ForkPoolWorker-62] fetch_url[90921f09-84a0-49ee-8b66-790864c72524]: Fetching https://sample.example.com/api/v1/chart/warm_up_cache with payload {"chart_id": 1}
[2024-11-26 19:00:00,593: INFO/ForkPoolWorker-62] fetch_url[90921f09-84a0-49ee-8b66-790864c72524]: Fetched https://sample.example.com/api/v1/chart/warm_up_cache with payload {"chart_id": 1}, status code: 200
....
....
[2024-11-27 01:00:00,015: INFO/ForkPoolWorker-16] cache-warmup[35eba83f-4d8a-420c-9965-2a4a0411c891]: Loading strategy
[2024-11-27 01:00:00,016: INFO/ForkPoolWorker-16] cache-warmup[35eba83f-4d8a-420c-9965-2a4a0411c891]: Loading DashboardTagsStrategy
[2024-11-27 01:00:00,016: INFO/ForkPoolWorker-16] cache-warmup[35eba83f-4d8a-420c-9965-2a4a0411c891]: Success!
[2024-11-27 01:00:00,091: INFO/ForkPoolWorker-64] fetch_url[e9a5874d-0123-4d9b-8919-48b3bddd9ece]: Fetching https://sample.example.com/api/v1/security/csrf_token/
[2024-11-27 01:00:00,090: INFO/ForkPoolWorker-64] fetch_url[e9a5874d-0123-4d9b-8919-48b3bddd9ece]: URL 'https://sample.example.com/api/v1/chart/warm_up_cache' is secure. Adding Referer header.
[2024-11-27 01:00:00,163: INFO/ForkPoolWorker-64] fetch_url[e9a5874d-0123-4d9b-8919-48b3bddd9ece]: Fetching https://sample.example.com/api/v1/chart/warm_up_cache with payload {"chart_id": 1}
[2024-11-27 01:00:00,548: INFO/ForkPoolWorker-64] fetch_url[e9a5874d-0123-4d9b-8919-48b3bddd9ece]: Fetched https://sample.example.com/api/v1/chart/warm_up_cache with payload {"chart_id": 1}, status code: 200
...

TESTING INSTRUCTIONS

Since we are now explicitly fetching the CSRF token, we can enable WTF_CSRF_ENABLED = True or safely remove WTF_CSRF_EXEMPT_LIST for any cache warmup-related endpoints. This ensures that all requests, including those for cache warmup, are properly validated with CSRF protection, thereby enhancing application security.

If CSRF protection is disabled or the endpoint is exempted, no CSRF checks will be performed. However, the ChartRestApi.warm_up_cache endpoint still requires a session cookie (session=session_cookie) to determine the user's context and permissions. Without this, the request will fail with a 401 Unauthorized error.

This PR ensures the session cookie is correctly updated in the request headers under the Cookie field, enabling the user context to be accurately resolved for authorization.

ADDITIONAL INFORMATION

• Has associated issue:
• Required feature flags:
• Changes UI
• Includes DB Migration (follow approval process in SIP-59)
  • Migration is atomic, supports rollback & is backwards-compatible
  • Confirm DB migration upgrade and downgrade tested
  • Runtime estimates and downtime expectations provided
• Introduces new feature or API
• Removes existing feature or API

[codecov]

Codecov Report

Attention: Patch coverage is 80.00000% with 2 lines in your changes missing coverage. Please review.

Project coverage is 83.81%. Comparing base (76d897e) to head (f41571e).
Report is 1143 commits behind head on master.

Files with missing lines	Patch %	Lines
superset/utils/urls.py	50.00%	2 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##           master   #31173       +/-   ##
===========================================
+ Coverage   60.48%   83.81%   +23.32%     
===========================================
  Files        1931      537     -1394     
  Lines       76236    38998    -37238     
  Branches     8568        0     -8568     
===========================================
- Hits        46114    32685    -13429     
+ Misses      28017     6313    -21704     
+ Partials     2105        0     -2105     

Flag	Coverage Δ
hive	48.78% <30.00%> (-0.38%)	⬇️
javascript	?
mysql	76.54% <80.00%> (?)
postgres	76.60% <80.00%> (?)
presto	53.27% <30.00%> (-0.53%)	⬇️
python	83.81% <80.00%> (+20.32%)	⬆️
sqlite	76.06% <80.00%> (?)
unit	60.90% <20.00%> (+3.28%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[AnTapTanhCode]

@nsivarajan I have tried it. But when I access the chart, it doesn't use the warmed up cache, but creates a different cache.

[nsivarajan]

@AnTapTanhCode, thanks for checking this. I have verified the behavior in my test environment, and the cache-warmup process successfully caches, and the charts/dashboards retrieve data as expected.

It seems that your case, where a different cache is fetched and cached instead of the warm-up-cached data, might require further investigation. However, I believe this issue is unrelated to this PR. If needed, please open a new issue with reproducible steps, and we can debug it further later.

[rusackas]

Looks to me like this ticks all the boxes of test coverage, proper cookie formatting, and HTTPS validation, but since this is so close to security concerns, I'm hopeful that someone more qualified (perhaps @dpgaspar and/or @mistercrunch) can take a look.

[mistercrunch]

LGTM, thanks for the contribution!

[nsivarajan]

Thanks for your guidance and support!