Airflow API creating roles #43658

D2CIT · 2024-11-04T09:27:49Z

D2CIT
Nov 4, 2024

Apache Airflow version

2.10.2

If "Other Airflow 2 version" selected, which one?

No response

What happened?

I'm trying to create a role via the airflow api but keep getting an error message.

I'm running airflow in a container . Local test env in podman and in AWS ECS contiainer
Reading out roles via the API goes fine.
tested via curl ( see picture below)
tested via python ( see script at "how to reproduce")
error on version 2.5.1 but also on 2.10
same error on version 2.10.2

AIRFLOW_API_URL: http://localhost:8080/api/v1
Failed to create role: 500


<!DOCTYPE html>
<html lang="en">
  <head>
    <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.5/css/bootstrap.min.css">
  </head>
  <body>
    <div class="container">
      <h1> Ooops! </h1>
      <div>
          <pre>
Something bad has happened. For security reasons detailed information about the error is not logged.

  * You should check your webserver logs and retrieve details of this error.

  * When you get the logs, it might explain the reasons, you should also Look for similar issues using:

     * <b><a href="https://github.com/apache/airflow/discussions">GitHub Discussions</a></b>
     * <b><a href="https://github.com/apache/airflow/issues">GitHub Issues</a></b>
     * <b><a href="https://stackoverflow.com/questions/tagged/airflow">Stack Overflow</a></b>
     * the usual search engine you use on a daily basis

    All those resources might help you to find a solution to your problem.

  * if you run Airflow on a Managed Service, consider opening an issue using the service support channels

  * only after you tried it all, and have difficulty with diagnosing and fixing the problem yourself,
    get the logs with errors, describe results of your investigation so far, and consider creating a
    <b><a href="https://github.com/apache/airflow/issues/new/choose">bug report</a></b> including this information.

Python version: 3.9.20
Airflow version: 2.10.2
Node: redacted
-------------------------------------------------------------------------------
Error! Please contact server admin.</pre>
      </div>
    </div>
  </body>
</html>

What you think should happen instead?

API bug

How to reproduce

Python script to create roles

import requests
import json

# Configuratie
AIRFLOW_API_URL = "http://localhost:8080/api/v1"
USERNAME = 'admin'
PASSWORD = 'admin'

auth = (USERNAME, PASSWORD)

print(f"AIRFLOW_API_URL: {AIRFLOW_API_URL}")

# Functie voor het ophalen van rollen in Airflow via API
def get_airflow_roles(uri, auth):
    url = f"{uri}/roles"
    response = requests.get(url, auth=auth)

    if response.status_code == 200:
        print("Get roles:")
        print(response.json())
    else:
        print(f"Failed to get roles: {response.status_code}")
        print(response.text)

# Functie voor het aanmaken van een rol in Airflow via API
def create_airflow_role(uri, auth, role_name):
    url = f"{uri}/roles"
    headers = {'Content-Type': 'application/json'}
    data = json.dumps({
        "name": role_name
    })

    response = requests.post(url, headers=headers, auth=auth, data=data)

    if response.status_code == 201:
        print(f"Role '{role_name}' created.")
    else:
        print(f"Failed to create role: {response.status_code}")
        print(response.text)

# Hoofdfunctie
if __name__ == "__main__":
    # Voorbeeld: Ophalen van bestaande rollen
    #get_airflow_roles(AIRFLOW_API_URL, auth)

    # Voorbeeld: Aanmaken van een nieuwe rol zonder permissies
    nieuwe_rol_naam = "admin_config"
    create_airflow_role(AIRFLOW_API_URL, auth, nieuwe_rol_naam)

Operating System

MacOS 15.1

Versions of Apache Airflow Providers

No extra providers. Just default airflow installation

Deployment

Docker-Compose

Deployment details

pod man-compose.yml local test environment;

version: '3.7'
services:
  postgres:
    image: postgres:14-alpine
    environment:
      POSTGRES_USER: airflow
      POSTGRES_PASSWORD: airflow
      POSTGRES_DB: airflow
      POSTGRES_HOST_AUTH_METHOD: trust
    ports:
      - "5432:5432"
    volumes:
      - postgres_db:/var/lib/postgresql/data

  airflow-init:
    # image: apache/airflow:2.5.1-python3.9
    image: apache/airflow:2.10.2-python3.9
    environment:
      AIRFLOW__API__AUTH_BACKENDS: "airflow.api.auth.backend.basic_auth,airflow.api.auth.backend.session"
      AIRFLOW__CORE__EXECUTOR: LocalExecutor
      AIRFLOW__DATABASE__SQL_ALCHEMY_CONN: postgresql+psycopg2://airflow:airflow@postgres/airflow
      AIRFLOW__CORE__FERNET_KEY: "EKrgNDRARUhv3jOhbixG6EwZGKpUxwVa6KLj0nuTA4A="
      AIRFLOW__CORE__LOAD_EXAMPLES: 'True'
      SQLALCHEMY_WARN_20: '1'
      SQLALCHEMY_SILENCE_UBER_WARNING: '1'
    volumes:
      - ./dags:/opt/airflow/dags
    entrypoint: ["airflow", "db", "init"]
    depends_on:
      - postgres

  airflow-create-user:
    image: apache/airflow:2.10.2-python3.9
    environment:
      AIRFLOW__API__AUTH_BACKENDS: "airflow.api.auth.backend.basic_auth,airflow.api.auth.backend.session"  # Geïdentificeerde typfout gecorrigeerd
      AIRFLOW__CORE__EXECUTOR: LocalExecutor
      AIRFLOW__DATABASE__SQL_ALCHEMY_CONN: postgresql+psycopg2://airflow:airflow@postgres/airflow
      AIRFLOW__CORE__FERNET_KEY: "EKrgNDRARUhv3jOhbixG6EwZGKpUxwVa6KLj0nuTA4A="
      AIRFLOW__CORE__LOAD_EXAMPLES: 'True'
    volumes:
      - ./dags:/opt/airflow/dags
    entrypoint: ["airflow", "users", "create", "--username", "admin", "--password", "admin", "--firstname", "Admin", "--lastname", "User", "--role", "Admin", "--email", "admin@example.com"]
    depends_on:
      - postgres
      - airflow-init

  airflow-webserver:
    image: apache/airflow:2.10.2-python3.9
    environment:
      AIRFLOW__API__AUTH_BACKENDS: "airflow.api.auth.backend.basic_auth,airflow.api.auth.backend.session"
      AIRFLOW__CORE__EXECUTOR: LocalExecutor
      AIRFLOW__DATABASE__SQL_ALCHEMY_CONN: postgresql+psycopg2://airflow:airflow@postgres/airflow
      AIRFLOW__CORE__FERNET_KEY: "EKrgNDRARUhv3jOhbixG6EwZGKpUxwVa6KLj0nuTA4A="
      AIRFLOW__CORE__LOAD_EXAMPLES: 'True'
      AIRFLOW__WEBSERVER__RBAC: 'True'
      SQLALCHEMY_WARN_20: '1'
      SQLALCHEMY_SILENCE_UBER_WARNING: '1'
      AIRFLOW__WEBSERVER__EXPOSE_CONFIG: 'True'
    volumes:
      - ./dags:/opt/airflow/dags
      # - ./config/webserver_config.py:/opt/airflow/webserver_config.py
    command: webserver
    ports:
      - "8080:8080"
    depends_on:
      - postgres
      - airflow-init

  airflow-scheduler:
    image: apache/airflow:2.10.2-python3.9
    environment:
      AIRFLOW__API__AUTH_BACKENDS: "airflow.api.auth.backend.basic_auth,airflow.api.auth.backend.session"
      AIRFLOW__CORE__EXECUTOR: LocalExecutor
      AIRFLOW__DATABASE__SQL_ALCHEMY_CONN: postgresql+psycopg2://airflow:airflow@postgres/airflow
      AIRFLOW__CORE__FERNET_KEY: "EKrgNDRARUhv3jOhbixG6EwZGKpUxwVa6KLj0nuTA4A="
      AIRFLOW__CORE__LOAD_EXAMPLES: 'True'
      SQLALCHEMY_WARN_20: '1'
      SQLALCHEMY_SILENCE_UBER_WARNING: '1'
    volumes:
      - ./dags:/opt/airflow/dags
    depends_on:
      - postgres
      - airflow-init

volumes:
  postgres_db:

Anything else?

Same error in

different versions (2.5.1 and 2.10.2)
tested in podman, docker and AWS ECS container

Are you willing to submit PR?

Yes I am willing to submit a PR!

Code of Conduct

I agree to follow this project's Code of Conduct

2024-11-04T09:27:53Z

boring-cyborg[bot]
bot Nov 4, 2024

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

0 replies

potiuk · 2024-11-04T20:38:15Z

potiuk
Nov 4, 2024
Collaborator

As explained in the error message - you need to look for a reason in logs. Without finding the error in logs, there is no way to help you - and likely when you see the logs, you will find what's wrong.

2 replies

D2CIT Nov 5, 2024
Author

thank you for your response. I have created a GitHub page with my setup and logfiles of each step.
See --> https://github.com/D2CIT/Airflow

command.sh : contains all the commands step by step
podman-compose.yaml : is the setup for the local airflow enviroment.
Config folder : contains the airflow.cfg and the webserver_config.py
log folder : contains the logfiles from the container for each step

Is this enough?
thanks

D2CIT Nov 5, 2024
Author

The log shows that a KeyError occurs because of the missing key 'permissions ’ in the JSON message I've send via the API. This means probably a missing required key in the API request.

I tried this suggestion

curl -X POST "http://localhost:8080/api/v1/roles" \
-H "Content-Type: application/json" \
--data '{
  "name": "example_role",
  "permissions": [
    {
      "action": {
        "name": "can_read"
      },
      "resource": {
        "name": "DagRuns"
      }
    }
  ]
}

but then I get the following error

{
 "detail": null,
 "status": 403,
 "title": "Forbidden",
 "type": "https://airflow.apache.org/docs/apache-airflow/2.10.2/stable-rest-api-ref.html#section/Errors/PermissionDenied"
}

This error message indicates that the permissions field is treated as unknown by the Airflow API, despite being expected according to the code.it seems there may be a nesting or incompatibility problem.

potiuk · 2024-11-04T20:38:55Z

potiuk
Nov 4, 2024
Collaborator

Converting to a discussion - this is not an actionable issue without log details and analysis of those.

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Airflow API creating roles #43658

{{title}}

Replies: 3 comments 2 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

Select a reply

Airflow API creating roles #43658

D2CIT Nov 4, 2024

Apache Airflow version

If "Other Airflow 2 version" selected, which one?

What happened?

What you think should happen instead?

How to reproduce

Operating System

Versions of Apache Airflow Providers

Deployment

Deployment details

Anything else?

Are you willing to submit PR?

Code of Conduct

Replies: 3 comments · 2 replies

boring-cyborg[bot] bot Nov 4, 2024

potiuk Nov 4, 2024 Collaborator

D2CIT Nov 5, 2024 Author

D2CIT Nov 5, 2024 Author

potiuk Nov 4, 2024 Collaborator

D2CIT
Nov 4, 2024

Replies: 3 comments 2 replies

boring-cyborg[bot]
bot Nov 4, 2024

potiuk
Nov 4, 2024
Collaborator

D2CIT Nov 5, 2024
Author

D2CIT Nov 5, 2024
Author

potiuk
Nov 4, 2024
Collaborator