Problem with PyJWT 2.10: {'msg': 'Subject must be a string'}

[amotl]

Bug description

When setting up a fresh sandbox environment, PyJWT 2.10 gets installed, released on Nov 17, i.e. three days ago. That breaks a little integration test suite we are running ¹. This is the exception being raised:

AssertionError: {'msg': 'Subject must be a string'}

When downgrading to use pyjwt<2.10, the test suite succeeds again.

You may want to accompany this by potentially adjusting dependencies or code in Apache Superset?

Superset version

3.x and 4.x

Additional context

The software test suite maintained here can be used to reproduce the problem.

• https://github.com/crate/cratedb-examples/tree/main/application/apache-superset

We added relevant details to this ticket, where we started to investigate this issue.

• Software tests start failing with Apache Superset 3.x/4.x and PyJWT 2.10 crate/cratedb-examples#741

Checklist

• I have searched Superset docs and Slack and didn't find a solution to my problem.
• I have searched the GitHub issue tracker and didn't find a similar bug report.
• I have checked Superset's logs for errors and if I found a relevant Python stacktrace, I included it here as text in the "additional context" section.

Footnotes

1. ... which orchestrates CLI invocations of the superset program and HTTP calls to the Superset API, in order to validate it works well together with CrateDB. ↩

[amotl]

We just found this is most likely the root cause.

• sub must be a string now? jpadilla/pyjwt#1017

The canonical recommendation is to downgrade to PyJWT in the meanwhile.

pip install 'pyjwt<2.10'

[amotl]

That patch submitted by Dependabot also demonstrates the problem.

• Update pyjwt requirement from <2.10 to <2.11 in /application/apache-superset crate/cratedb-examples#744

[amotl]

@jkogut: Do you have any idea why only we might be affected, but Superset's test suite seems to still succeed, and nobody else seems to be tripped? Is it related to the value of the SECRET_KEY maybe?

[jkogut]

@amotl yes indeed that was caused by the incorrect SECRET_KEY. Happened during test migration of superset instance. So indeed please double check SECRET_KEY. 🙏

[jkogut]

@amotl actually I was wrong, correct SECRET_KEY only allowed me to get access token but cannot proceed further with getting chart list for instance:

def get_chart_list():
    headers = {
        'Authorization': f"Bearer {get_bearer_token()}"
    }
    response = requests.get(f"{base_url}/chart/", headers=headers, verify=False)
    return response.json()

problem observed as reported on Superset 3.1.3, and fixed as recommended with installing PyJWT==2.9.0.

So still looks like new PyJWT 2.10.0 release can cause some problems with Superset API access.

[amotl]

We re-generated the value for SECRET_KEY using openssl rand -base64 42, as advertised in Superset's documentation, but it did not improve the situation.

[amotl]

Summary

• Relevant change: Validate sub and jti claims for the token jpadilla/pyjwt#1005
• Problem report: sub must be a string now? jpadilla/pyjwt#1017
• Solution: Incompatibility with PyJWT 2.10.0: Non-string identities unsupported in create_access_token vimalloc/flask-jwt-extended#556
• Downstream reports:
  • PyJWT 2.10.0 Enforces sub vimalloc/flask-jwt-extended#561
  • Update to pyjwt has added breaking changes to released version yunojuno/django-request-token#65
• Downstream patches:
  • Add controls for verify_sub option in PyJWT vimalloc/flask-jwt-extended#562

Evaluation

Because Apache Superset is based on Flask-AppBuilder, which in turn pulls in Flask-JWT-Extended, we guess the most recent release 4.7.1 resolves this problem already.

[amotl]

We can still observe the problem, even after upgrading to Flask-JWT-Extended 4.7.1.

$ uv pip install --upgrade "PyJWT>=2.10" "Flask-JWT-Extended>=4.7.1" "Flask<3"
 - flask-jwt-extended==4.7.0
 + flask-jwt-extended==4.7.1

pytest -k api

            response = requests.delete(resource_to_delete, headers=get_auth_headers())
>           assert response.status_code in [200, 404], response.json()
E           AssertionError: {'msg': 'Subject must be a string'}
E           assert 422 in [200, 404]
E            +  where 422 = <Response [422]>.status_code

conftest.py:74: AssertionError

/cc @jlucier, @vimalloc

[jlucier]

Have you configured flask-jwt-extended to not verify the sub claim? 'JWT_VERIFY_SUB=False'

[amotl]

Thank you so much for your quick response, @jlucier. Actually, we are just approaching this problem as users of Apache Superset, not having any knowledge about its code base and its dependencies at all, other than knowing a bit about Flask.

We found that Flask-JWT-Extended is being used by Flask-AppBuilder, so this is probably the right spot to address this configuration update needed for the most recent PyJWT library? In this spirit, we figure creating a corresponding issue over there makes sense?

• Problem in Apache Superset: PyJWT 2.10 changed semantics, Flask-JWT-Extended 4.7.1 needs a configuration change dpgaspar/Flask-AppBuilder#2287

Please correct me if I am wrong with my evaluations.

/cc @dpgaspar

[jlucier]

Since my patch, this isn't a problem with flask-jwt-extended anymore. If you don't want enforce that the sub claim is a string, you just need to configure 'JWT_VERIFY_SUB' to be False. That's your choice to make, not the library's.

[amotl]

As a user of Apache Superset, I am was not touching JWT concerns at all, so [I thought] I can't make any choices about how JWT is used by its ingredients.

In this case, I guess it will be the concern of Flask-AppBuilder to configure Flask-JWT-Extended correctly, that's why I created a ticket there, but I may be wrong. Please educate me where you see a flaw in my thinking, as I don't have any strong knowledge of the subsystems we are talking about.

[amotl]

All right, I wasn't aware that JWT_VERIFY_SUB = False could easily be configured within the Flask application configuration file, which is effectively the superset_config.py file in our case.

After getting it right, this update perfectly resolves the problem we have been observing.

• Apache Superset: Fix JWT problem by updating to flask-jwt-extended 4.7.1 crate/cratedb-examples#749

Thank you so much! 💯

[rusackas]

@amotl are we good to close this then, if your problem is solved? If you think there's a good way/place to capture this in the docs, let us know or open a PR, it'd be appreciated!

[amotl]

Hi Evan. First things first, thanks for and keep up your excellent work on Superset and Preset.

Thank you for your reminder. I guess it will be good to close. As it turned out it might have only been a niche thing tripping us, I think having it documented on behalf of this very ticket will be good enough to support others being tripped by the same thing.

Please re-open if you think it deserves being captured on the docs explicitly, so we can check if we can come up with a corresponding patch.