Unable to integrate keycloak provider with Airflow 2.7.3

[geoffo-dev]

Apache Airflow version

Other Airflow 2 version (please specify below)

If "Other Airflow 2 version" selected, which one?

2.7.3

What happened?

Hello,

I am trying to integrate keycloak with airflow which I have managed successfully in the past however I am currently struggling - possibly due to it being a newer version of airflow where I understand certain authentication mechanisms have changed.

Having followed the guide have the following webserver_config.py file:

from airflow.www.security import AirflowSecurityManager
from flask_appbuilder.security.manager import AUTH_OAUTH
import os

AUTH_TYPE = AUTH_OAUTH
AUTH_ROLES_SYNC_AT_LOGIN = True
AUTH_USER_REGISTRATION = True

AUTH_ROLE_ADMIN = os.environ['KEYCLOAK_ADMIN_ROLE_NAME']
AUTH_ROLE_OP = os.environ['KEYCLOAK_OP_ROLE_NAME']
AUTH_ROLE_USER = os.environ['KEYCLOAK_USER_ROLE_NAME']
AUTH_ROLE_VIEWER = os.environ['KEYCLOAK_VIEWER_ROLE_NAME']
AUTH_ROLE_PUBLIC = os.environ['KEYCLOAK_PUBLIC_ROLE_NAME']

# Will allow user self registration
AUTH_USER_REGISTRATION = True
AUTH_USER_REGISTRATION_ROLE = "Public"
AUTH_ROLES_SYNC_AT_LOGIN = True
AUTH_ROLES_MAPPING = {
    "Admin": [AUTH_ROLE_ADMIN],
    "Op": [AUTH_ROLE_OP],
    "User": [AUTH_ROLE_USER],
    "Viewer": [AUTH_ROLE_VIEWER],
    "Public": [AUTH_ROLE_PUBLIC],
}

PROVIDER_NAME = 'keycloak'
CLIENT_ID = os.environ['KEYCLOAK_CLIENT_ID']

# Get the OIDC SECRET
with open('/secret/oidc-secret/client-secret', 'r') as file:
    CLIENT_SECRET = file.read().rstrip()

OIDC_ISSUER = os.environ['OIDC_ISSUER_URL']
OIDC_BASE_URL = "{oidc_issuer}/protocol/openid-connect".format(oidc_issuer=OIDC_ISSUER)
OIDC_TOKEN_URL = "{oidc_base_url}/token".format(oidc_base_url=OIDC_BASE_URL)
OIDC_AUTH_URL = "{oidc_base_url}/auth".format(oidc_base_url=OIDC_BASE_URL)

OAUTH_PROVIDERS = [{
    'name': PROVIDER_NAME,
    'token_key':'access_token',
    'icon':'fa-globe',
    'remote_app': {
        'client_id': CLIENT_ID,
        'client_secret': CLIENT_SECRET,
        'api_base_url': OIDC_BASE_URL,
        'client_kwargs': {
            'scope': 'email profile'
        },
        'access_token_url': OIDC_TOKEN_URL,
        'authorize_url': OIDC_AUTH_URL,
        'request_token_url': None,              
    }
}]


  # Make sure to replace this with your own implementation of AirflowSecurityManager class
  SECURITY_MANAGER_CLASS = KeycloakAuthorizer

and the following keycloakAuthorizer.py which should be in the same path:

from airflow.www.security import AirflowSecurityManager
import logging
from typing import Any, List, Union
import os

# Create some logging
log = logging.getLogger(__name__)
log.setLevel(os.getenv("AIRFLOW__LOGGING__FAB_LOGGING_LEVEL", "INFO"))

class KeycloakAuthorizer(AirflowSecurityManager):
CLIENT_ID = os.environ['KEYCLOAK_CLIENT_ID']
def get_oauth_user_info(self, provider: str, resp: Any) -> dict[str, Union[str, list[str]]]:
    remote_app = self.appbuilder.sm.oauth_remotes[provider]
    me = remote_app.get("user")
    roles = me["resource_access"][CLIENT_ID]["roles"]
    if len(roles) < 1:
        roles = ["public"]

    userinfo = {
        "username": me.get("preferred_username"),
        "email": me.get("email"),
        "first_name": me.get("given_name"),
        "last_name": me.get("family_name"),
        "role_keys": roles,
    }
    log.debug(f"User info from Keycloak: {userinfo}\Role info from Keycloak: {roles}")
    return userinfo

However when the webserver launches, there is no option to login via keycloak - only standard username and password.

The only error I can see that might be relevant when the webserver loads which relates to the google package (for what I believe is required for keycloak to work):

/home/airflow/.local/lib/python3.11/site-packages/flask_limiter/extension.py:336 UserWarning: Using the in-memory storage for tracking rate limits as no storage was explicitly specified. This is not recommended for production use. See: https://flask-limiter.readthedocs.io#configuring-a-storage-backend for documentation about configuring the storage backend.
[2024-01-15 15:53:42 +0000] [13] [INFO] Starting gunicorn 21.2.0
[2024-01-15T15:53:49.840+0000] {providers_manager.py:271} INFO - Optional provider feature disabled when importing 'airflow.providers.google.leveldb.hooks.leveldb.LevelDBHook' from 'apache-airflow-providers-google' package
/home/airflow/.local/lib/python3.11/site-packages/snowflake/connector/options.py:103 UserWarning: You have an incompatible version of 'pyarrow' installed (11.0.0), please install a version that adheres to: 'pyarrow<10.1.0,>=10.0.1; extra == "pandas"'
[2024-01-15 15:53:53 +0000] [13] [INFO] Listening at: http://0.0.0.0:8080 (13)
[2024-01-15 15:53:53 +0000] [13] [INFO] Using worker: sync
[2024-01-15 15:53:53 +0000] [25] [INFO] Booting worker with pid: 25
[2024-01-15 15:53:53 +0000] [26] [INFO] Booting worker with pid: 26
[2024-01-15 15:53:53 +0000] [27] [INFO] Booting worker with pid: 27
[2024-01-15 15:53:53 +0000] [28] [INFO] Booting worker with pid: 28
10.38.0.0 - - [15/Jan/2024:15:53:59 +0000] "GET /health HTTP/1.1" 200 318 "-" "kube-probe/1.28+"
10.38.0.0 - - [15/Jan/2024:15:53:59 +0000] "GET /health HTTP/1.1" 200 318 "-" "kube-probe/1.28+"

I have tried version 2.8.0 as well, but due to #36702 - this does not work.

I am at a bit of a loss as to how to resolve, I have tried a number of images, but not certain if this is supported anymore or if there is another way to resolve.

What you think should happen instead?

User should be presented with a login option with keycloak

How to reproduce

Have described above

Operating System

Tags 2.7.3-python3.11 and 2.7.3 in docker

Versions of Apache Airflow Providers

apache-airflow-providers-amazon==8.10.0
apache-airflow-providers-celery==3.4.1
apache-airflow-providers-cncf-kubernetes==7.8.0
apache-airflow-providers-common-sql==1.8.0
apache-airflow-providers-daskexecutor==1.1.0
apache-airflow-providers-docker==3.8.0
apache-airflow-providers-elasticsearch==5.1.0
apache-airflow-providers-ftp==3.6.0
apache-airflow-providers-google==10.11.0
apache-airflow-providers-grpc==3.3.0
apache-airflow-providers-hashicorp==3.5.0
apache-airflow-providers-http==4.6.0
apache-airflow-providers-imap==3.4.0
apache-airflow-providers-microsoft-azure==8.1.0
apache-airflow-providers-mysql==5.4.0
apache-airflow-providers-odbc==4.1.0
apache-airflow-providers-openlineage==1.2.0
apache-airflow-providers-postgres==5.7.1
apache-airflow-providers-redis==3.4.0
apache-airflow-providers-sendgrid==3.3.0
apache-airflow-providers-sftp==4.7.0
apache-airflow-providers-slack==8.3.0
apache-airflow-providers-snowflake==5.1.0
apache-airflow-providers-sqlite==3.5.0
apache-airflow-providers-ssh==3.8.1

Deployment

Official Apache Airflow Helm Chart

Deployment details

Deploying on EKS v1.28.3
Helm Version v3.13.1
Helm Chart Version 1.11.0

Anything else?

Would be happy to support updating a PR, but not really sure where to start!

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[eladkal]

Hi @geoffo-dev can you please clarify what is the bug?
Your report seems more like a support request.

[geoffo-dev]

Hi @eladkal - So sorry it might be support or it might be a bug. I think I put it here as I have followed the configuration steps and the files are visible in the pod - however nothing is happening. The only thing I can see is the info in the log above.

Is this a bug... I dont know..! I havent seen many examples specific to keycloak since Airflow 2.5.0.

[eladkal]

We use Issues for bug reports since for this moment your problem is more of community support I'm converting this thread to GitHub Discussion.

I hope someone from the community might be able to help you.