API Issues when upgrading to 3.0.0

[Stefan781]

My use case for Superset is to be able to recreate a predefined environment from scratch using Docker.

After upgrading to Superset 3.0.0 a script which uploads dashboards, created following #2488 (comment), stopped working. The errors I get are "wtforms.validators.ValidationError: The CSRF session token is missing." and "json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)". I am using the API V1 to upload the dashboard, namely the dashboard/import method. The image with tag e6f7c73f543ea5c2e5ffdab8e9e01ec87fd94322 is the one I am using now. I had to specify it after the upload functionality broke using the latest image. Perhaps the issue lies somewhere here. Running it on 2.1.1 also works.

I did not notice any changes in UPDATING.MD that should be related to this issue. Are there any changes that I might have missed which break the approach in the linked comment? I tried delving deeper into the CSRF but didn't get any meaningful results. Using the security/csrf_token endpoint to retrieve it did not help either. At some point I also started getting errors that the user does not have the rights to create the database (which is specified in the dashboard file) which is strange since I am using the admin credentials to upload the dashboard.

Furthermore, when accessing /swagger/v1 I get an empty page.

How to reproduce the bug

1. Export your favorite dashboard.
2. Use the linked script to make a request to <superset_host>/api/v1/dashboard/import
3. Start a new instance of Superset
4. Run the script
5. The script should cause internal error that is logged
6. The swagger page is empty.

Expected results

The dashboard should be uploaded without an error. The swagger page should not be empty.

Actual results

You get the described problems.

Checklist

Make sure to follow these steps before submitting your issue - thank you!

• I have checked the superset logs for python stacktraces and included it here as text if there are any.
• I have reproduced the issue with at least the latest released version of superset.
• I have checked the issue tracker for the same issue and I haven't found one similar.

Additional context

Logs:

2023-09-20 15:08:03 2023-09-20 13:08:03,792:INFO:flask_wtf.csrf:The CSRF session token is missing.
2023-09-20 15:08:03 Refresh CSRF token error
2023-09-20 15:08:03 Traceback (most recent call last):
2023-09-20 15:08:03 File "/usr/local/lib/python3.9/site-packages/flask_wtf/csrf.py", line 261, in protect
2023-09-20 15:08:03 validate_csrf(self._get_csrf_token())
2023-09-20 15:08:03 File "/usr/local/lib/python3.9/site-packages/flask_wtf/csrf.py", line 103, in validate_csrf
2023-09-20 15:08:03 raise ValidationError("The CSRF session token is missing.")
2023-09-20 15:08:03 wtforms.validators.ValidationError: The CSRF session token is missing.
2023-09-20 15:08:03
2023-09-20 15:08:03 During handling of the above exception, another exception occurred:
2023-09-20 15:08:03
2023-09-20 15:08:03 Traceback (most recent call last):
2023-09-20 15:08:03 File "/usr/local/lib/python3.9/site-packages/flask/app.py", line 1821, in full_dispatch_request
2023-09-20 15:08:03 rv = self.preprocess_request()
2023-09-20 15:08:03 File "/usr/local/lib/python3.9/site-packages/flask/app.py", line 2313, in preprocess_request
2023-09-20 15:08:03 rv = self.ensure_sync(before_func)()
2023-09-20 15:08:03 File "/usr/local/lib/python3.9/site-packages/flask_wtf/csrf.py", line 229, in csrf_protect
2023-09-20 15:08:03 self.protect()
2023-09-20 15:08:03 File "/usr/local/lib/python3.9/site-packages/flask_wtf/csrf.py", line 264, in protect
2023-09-20 15:08:03 self._error_response(e.args[0])
2023-09-20 15:08:03 File "/usr/local/lib/python3.9/site-packages/flask_wtf/csrf.py", line 307, in _error_response
2023-09-20 15:08:03 raise CSRFError(reason)
2023-09-20 15:08:03 flask_wtf.csrf.CSRFError: 400 Bad Request: The CSRF session token is missing.
2023-09-20 15:08:03 2023-09-20 13:08:03,792:WARNING:superset.views.base:Refresh CSRF token error
2023-09-20 15:08:03 Traceback (most recent call last):
2023-09-20 15:08:03 File "/usr/local/lib/python3.9/site-packages/flask_wtf/csrf.py", line 261, in protect
2023-09-20 15:08:03 validate_csrf(self._get_csrf_token())
2023-09-20 15:08:03 File "/usr/local/lib/python3.9/site-packages/flask_wtf/csrf.py", line 103, in validate_csrf
2023-09-20 15:08:03 raise ValidationError("The CSRF session token is missing.")
2023-09-20 15:08:03 wtforms.validators.ValidationError: The CSRF session token is missing.
2023-09-20 15:08:03
2023-09-20 15:08:03 During handling of the above exception, another exception occurred:
2023-09-20 15:08:03
2023-09-20 15:08:03 Traceback (most recent call last):
2023-09-20 15:08:03 File "/usr/local/lib/python3.9/site-packages/flask/app.py", line 1821, in full_dispatch_request
2023-09-20 15:08:03 rv = self.preprocess_request()
2023-09-20 15:08:03 File "/usr/local/lib/python3.9/site-packages/flask/app.py", line 2313, in preprocess_request
2023-09-20 15:08:03 rv = self.ensure_sync(before_func)()
2023-09-20 15:08:03 File "/usr/local/lib/python3.9/site-packages/flask_wtf/csrf.py", line 229, in csrf_protect
2023-09-20 15:08:03 self.protect()
2023-09-20 15:08:03 File "/usr/local/lib/python3.9/site-packages/flask_wtf/csrf.py", line 264, in protect
2023-09-20 15:08:03 self._error_response(e.args[0])
2023-09-20 15:08:03 File "/usr/local/lib/python3.9/site-packages/flask_wtf/csrf.py", line 307, in _error_response
2023-09-20 15:08:03 raise CSRFError(reason)
2023-09-20 15:08:03 flask_wtf.csrf.CSRFError: 400 Bad Request: The CSRF session token is missing.
2023-09-20 15:08:03 127.0.0.1 - - [20/Sep/2023:13:08:03 +0000] "POST /login/ HTTP/1.1" 302 201 "-" "python-requests/2.30.0"
2023-09-20 15:08:03 127.0.0.1 - - [20/Sep/2023:13:08:03 +0000] "GET /login/ HTTP/1.1" 200 48800 "-" "python-requests/2.30.0"
2023-09-20 15:08:03 Invalid JSON file
2023-09-20 15:08:03 Traceback (most recent call last):
2023-09-20 15:08:03 File "/app/superset/dashboards/commands/importers/v0.py", line 343, in validate
2023-09-20 15:08:03 json.loads(content)
2023-09-20 15:08:03 File "/usr/local/lib/python3.9/json/init.py", line 346, in loads
2023-09-20 15:08:03 return _default_decoder.decode(s)
2023-09-20 15:08:03 File "/usr/local/lib/python3.9/json/decoder.py", line 337, in decode
2023-09-20 15:08:03 obj, end = self.raw_decode(s, idx=_w(s, 0).end())
2023-09-20 15:08:03 File "/usr/local/lib/python3.9/json/decoder.py", line 355, in raw_decode
2023-09-20 15:08:03 raise JSONDecodeError("Expecting value", s, err.value) from None
2023-09-20 15:08:03 json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
2023-09-20 15:08:03 2023-09-20 13:08:03,875:ERROR:superset.dashboards.commands.importers.v0:Invalid JSON file
2023-09-20 15:08:03 Traceback (most recent call last):
2023-09-20 15:08:03 File "/app/superset/dashboards/commands/importers/v0.py", line 343, in validate
2023-09-20 15:08:03 json.loads(content)
2023-09-20 15:08:03 File "/usr/local/lib/python3.9/json/init.py", line 346, in loads
2023-09-20 15:08:03 return _default_decoder.decode(s)
2023-09-20 15:08:03 File "/usr/local/lib/python3.9/json/decoder.py", line 337, in decode
2023-09-20 15:08:03 obj, end = self.raw_decode(s, idx=_w(s, 0).end())
2023-09-20 15:08:03 File "/usr/local/lib/python3.9/json/decoder.py", line 355, in raw_decode
2023-09-20 15:08:03 raise JSONDecodeError("Expecting value", s, err.value) from None
2023-09-20 15:08:03 json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
2023-09-20 15:08:03 Error running import command
2023-09-20 15:08:03 Traceback (most recent call last):
2023-09-20 15:08:03 File "/app/superset/dashboards/commands/importers/dispatcher.py", line 57, in run
2023-09-20 15:08:03 command.run()
2023-09-20 15:08:03 File "/app/superset/dashboards/commands/importers/v0.py", line 333, in run
2023-09-20 15:08:03 self.validate()
2023-09-20 15:08:03 File "/app/superset/dashboards/commands/importers/v0.py", line 343, in validate
2023-09-20 15:08:03 json.loads(content)
2023-09-20 15:08:03 File "/usr/local/lib/python3.9/json/init.py", line 346, in loads
2023-09-20 15:08:03 return _default_decoder.decode(s)
2023-09-20 15:08:03 File "/usr/local/lib/python3.9/json/decoder.py", line 337, in decode
2023-09-20 15:08:03 obj, end = self.raw_decode(s, idx=_w(s, 0).end())
2023-09-20 15:08:03 File "/usr/local/lib/python3.9/json/decoder.py", line 355, in raw_decode
2023-09-20 15:08:03 raise JSONDecodeError("Expecting value", s, err.value) from None
2023-09-20 15:08:03 json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
2023-09-20 15:08:03 2023-09-20 13:08:03,876:ERROR:superset.dashboards.commands.importers.dispatcher:Error running import command
2023-09-20 15:08:03 Traceback (most recent call last):
2023-09-20 15:08:03 File "/app/superset/dashboards/commands/importers/dispatcher.py", line 57, in run
2023-09-20 15:08:03 command.run()
2023-09-20 15:08:03 File "/app/superset/dashboards/commands/importers/v0.py", line 333, in run
2023-09-20 15:08:03 self.validate()
2023-09-20 15:08:03 File "/app/superset/dashboards/commands/importers/v0.py", line 343, in validate
2023-09-20 15:08:03 json.loads(content)
2023-09-20 15:08:03 File "/usr/local/lib/python3.9/json/init.py", line 346, in loads
2023-09-20 15:08:03 return _default_decoder.decode(s)
2023-09-20 15:08:03 File "/usr/local/lib/python3.9/json/decoder.py", line 337, in decode
2023-09-20 15:08:03 obj, end = self.raw_decode(s, idx=w(s, 0).end())
2023-09-20 15:08:03 File "/usr/local/lib/python3.9/json/decoder.py", line 355, in raw_decode
2023-09-20 15:08:03 raise JSONDecodeError("Expecting value", s, err.value) from None
2023-09-20 15:08:03 json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
2023-09-20 15:08:03 Expecting value: line 1 column 1 (char 0)
2023-09-20 15:08:03 Traceback (most recent call last):
2023-09-20 15:08:03 File "/usr/local/lib/python3.9/site-packages/flask/app.py", line 1823, in full_dispatch_request
2023-09-20 15:08:03 rv = self.dispatch_request()
2023-09-20 15:08:03 File "/usr/local/lib/python3.9/site-packages/flask/app.py", line 1799, in dispatch_request
2023-09-20 15:08:03 return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
2023-09-20 15:08:03 File "/usr/local/lib/python3.9/site-packages/flask_appbuilder/security/decorators.py", line 87, in wraps
2023-09-20 15:08:03 return f(self, *args, **kwargs)
2023-09-20 15:08:03 File "/app/superset/views/base_api.py", line 127, in wraps
2023-09-20 15:08:03 raise ex
2023-09-20 15:08:03 File "/app/superset/views/base_api.py", line 121, in wraps
2023-09-20 15:08:03 duration, response = time_function(f, self, *args, **kwargs)
2023-09-20 15:08:03 File "/app/superset/utils/core.py", line 1526, in time_function
2023-09-20 15:08:03 response = func(*args, **kwargs)
2023-09-20 15:08:03 File "/app/superset/utils/log.py", line 255, in wrapper
2023-09-20 15:08:03 value = f(*args, **kwargs)
2023-09-20 15:08:03 File "/app/superset/views/base_api.py", line 108, in wraps
2023-09-20 15:08:03 return f(self, *args, **kwargs)
2023-09-20 15:08:03 File "/app/superset/dashboards/api.py", line 1199, in import
2023-09-20 15:08:03 command.run()
2023-09-20 15:08:03 File "/app/superset/dashboards/commands/importers/dispatcher.py", line 68, in run
2023-09-20 15:08:03 raise exc
2023-09-20 15:08:03 File "/app/superset/dashboards/commands/importers/dispatcher.py", line 57, in run
2023-09-20 15:08:03 command.run()
2023-09-20 15:08:03 File "/app/superset/dashboards/commands/importers/v0.py", line 333, in run
2023-09-20 15:08:03 self.validate()
2023-09-20 15:08:03 File "/app/superset/dashboards/commands/importers/v0.py", line 343, in validate
2023-09-20 15:08:03 json.loads(content)
2023-09-20 15:08:03 File "/usr/local/lib/python3.9/json/init.py", line 346, in loads
2023-09-20 15:08:03 return _default_decoder.decode(s)
2023-09-20 15:08:03 File "/usr/local/lib/python3.9/json/decoder.py", line 337, in decode
2023-09-20 15:08:03 obj, end = self.raw_decode(s, idx=w(s, 0).end())
2023-09-20 15:08:03 File "/usr/local/lib/python3.9/json/decoder.py", line 355, in raw_decode
2023-09-20 15:08:03 raise JSONDecodeError("Expecting value", s, err.value) from None
2023-09-20 15:08:03 json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
2023-09-20 15:08:03 2023-09-20 13:08:03,876:ERROR:superset.views.base:Expecting value: line 1 column 1 (char 0)
2023-09-20 15:08:03 Traceback (most recent call last):
2023-09-20 15:08:03 File "/usr/local/lib/python3.9/site-packages/flask/app.py", line 1823, in full_dispatch_request
2023-09-20 15:08:03 rv = self.dispatch_request()
2023-09-20 15:08:03 File "/usr/local/lib/python3.9/site-packages/flask/app.py", line 1799, in dispatch_request
2023-09-20 15:08:03 return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
2023-09-20 15:08:03 File "/usr/local/lib/python3.9/site-packages/flask_appbuilder/security/decorators.py", line 87, in wraps
2023-09-20 15:08:03 return f(self, *args, **kwargs)
2023-09-20 15:08:03 File "/app/superset/views/base_api.py", line 127, in wraps
2023-09-20 15:08:03 raise ex
2023-09-20 15:08:03 File "/app/superset/views/base_api.py", line 121, in wraps
2023-09-20 15:08:03 duration, response = time_function(f, self, *args, **kwargs)
2023-09-20 15:08:03 File "/app/superset/utils/core.py", line 1526, in time_function
2023-09-20 15:08:03 response = func(*args, **kwargs)
2023-09-20 15:08:03 File "/app/superset/utils/log.py", line 255, in wrapper
2023-09-20 15:08:03 value = f(*args, **kwargs)
2023-09-20 15:08:03 File "/app/superset/views/base_api.py", line 108, in wraps
2023-09-20 15:08:03 return f(self, *args, **kwargs)
2023-09-20 15:08:03 File "/app/superset/dashboards/api.py", line 1199, in import
2023-09-20 15:08:03 command.run()
2023-09-20 15:08:03 File "/app/superset/dashboards/commands/importers/dispatcher.py", line 68, in run
2023-09-20 15:08:03 raise exc
2023-09-20 15:08:03 File "/app/superset/dashboards/commands/importers/dispatcher.py", line 57, in run
2023-09-20 15:08:03 command.run()
2023-09-20 15:08:03 File "/app/superset/dashboards/commands/importers/v0.py", line 333, in run
2023-09-20 15:08:03 self.validate()
2023-09-20 15:08:03 File "/app/superset/dashboards/commands/importers/v0.py", line 343, in validate
2023-09-20 15:08:03 json.loads(content)
2023-09-20 15:08:03 File "/usr/local/lib/python3.9/json/init.py", line 346, in loads
2023-09-20 15:08:03 return _default_decoder.decode(s)
2023-09-20 15:08:03 File "/usr/local/lib/python3.9/json/decoder.py", line 337, in decode
2023-09-20 15:08:03 obj, end = self.raw_decode(s, idx=_w(s, 0).end())
2023-09-20 15:08:03 File "/usr/local/lib/python3.9/json/decoder.py", line 355, in raw_decode
2023-09-20 15:08:03 raise JSONDecodeError("Expecting value", s, err.value) from None
2023-09-20 15:08:03 json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

[Stefan781]

I tried another approach from Stack Overflow and am getting "superset.commands.exceptions.ImportFailedError: Database doesn't exist and user doesn't have permission to create databases", along with the invalid JSON. If I upload the dashboard through the UI, logged in with the same credentials, there are no errors and the dashboards are uploaded.

[shafad1]

I could not login superset even after success full installation, getting CSRF token errors. this tool is not a stable solution for reporting , we cant confidently deploy this in production environment . for all releases is having some bugs . as per my analysis version 3.0 is not stable . better to migrate after the confirmation of whether the purticular version is stable or not

[betodealmeida]

It looks like you might be missing the X-CSRFToken header. Take a look a the Preset CLI to see how we authenticate against Superset:

https://github.com/preset-io/backend-sdk/blob/main/src/preset_cli/auth/superset.py

[Stefan781]

Hello @betodealmeida and thanks for the response!

The second approach I have linked was already adding the X-CSRFToken and it still did not work for me. I tried your approach with adding it directly to the headers of the session and still got the same errors.

I followed another comment and disabling Talisman worked, so it must be security related. I assume that you don't have such an issue in Preset CLI? How do you get it to work? I, more or less, do the same as you do in the linked code.

It would also be nice if there is an entry added to Superset's documentation about how to authenticate and use the API, since apparently it is not trivial.

[liberty-rising]

@Stefan781 Would you mind sharing how you import the dashboards using the api? I'm not able to post my dashboard zip file to api/v1/dashboard/import without getting an error

[Stefan781]

@liberty-rising I authenticate and then call the dashboard API with the zip file. It will be easier if you share your approach and the error, perhaps opening another issue.

[dpgaspar]

Hello @betodealmeida and thanks for the response!

The second approach I have linked was already adding the X-CSRFToken and it still did not work for me. I tried your approach with adding it directly to the headers of the session and still got the same errors.

I followed another comment and disabling Talisman worked, so it must be security related. I assume that you don't have such an issue in Preset CLI? How do you get it to work? I, more or less, do the same as you do in the linked code.

It would also be nice if there is an entry added to Superset's documentation about how to authenticate and use the API, since apparently it is not trivial.

If talisman is blocking you, your probably hitting a CSP restriction (CORS?), I assume your script runs on a web env.

I don't consider this to be an API issue, feel free to use our Slack channel to ask for further help (if needed)

[Stefan781]

Hello @dpgaspar. Thanks for taking your time to check this issue!

I have now tried using Talisman again and the problem is gone. Once again, this to me means that there were some changes that broke something. Apparently, a later version fixed it. As I have mentioned, my script was working before I updated Superset back then.

I don't find it helpful to close such issues without some meaningful resolution. This is (was) obviously a problem with changes to the codebase. My personal guess is that the cause is (was) related to #24579 and #24616.

In any case, since the issue is fixed in the current version of Superset, it does make sense to close the ticket.