How to enable RBAC and add users? #225
Comments
|
Just spent an hour trying to solve this. The problem here was, it wasn't reading my webserver_config.py file, and that was failing silently, in app.config.from_pyfile(webserver_config_path, silent=True)Without reading that file, sqlalchemy is defaulting to sqlite, which is why no user is being inserted into the postgres DB. So I suggest disabling |
|
Awesome, thank you! The problem was Airflow was creating the webserver_config.py in So, to get it running, one has to...
Default webserver_config.py: # -*- coding: utf-8 -*-
#
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
import os
from airflow import configuration as conf
from flask_appbuilder.security.manager import AUTH_DB
# from flask_appbuilder.security.manager import AUTH_LDAP
# from flask_appbuilder.security.manager import AUTH_OAUTH
# from flask_appbuilder.security.manager import AUTH_OID
# from flask_appbuilder.security.manager import AUTH_REMOTE_USER
basedir = os.path.abspath(os.path.dirname(__file__))
# The SQLAlchemy connection string.
SQLALCHEMY_DATABASE_URI = conf.get('core', 'SQL_ALCHEMY_CONN')
# Flask-WTF flag for CSRF
CSRF_ENABLED = True
# ----------------------------------------------------
# AUTHENTICATION CONFIG
# ----------------------------------------------------
# For details on how to set up each of the following authentication, see
# http://flask-appbuilder.readthedocs.io/en/latest/security.html# authentication-methods
# for details.
# The authentication type
# AUTH_OID : Is for OpenID
# AUTH_DB : Is for database
# AUTH_LDAP : Is for LDAP
# AUTH_REMOTE_USER : Is for using REMOTE_USER from web server
# AUTH_OAUTH : Is for OAuth
AUTH_TYPE = AUTH_DB
# Uncomment to setup Full admin role name
# AUTH_ROLE_ADMIN = 'Admin'
# Uncomment to setup Public role name, no authentication needed
# AUTH_ROLE_PUBLIC = 'Public'
# Will allow user self registration
# AUTH_USER_REGISTRATION = True
# The default user self registration role
# AUTH_USER_REGISTRATION_ROLE = "Public"
# When using OAuth Auth, uncomment to setup provider(s) info
# Google OAuth example:
# OAUTH_PROVIDERS = [{
# 'name':'google',
# 'whitelist': ['@YOU_COMPANY_DOMAIN'], # optional
# 'token_key':'access_token',
# 'icon':'fa-google',
# 'remote_app': {
# 'base_url':'https://www.googleapis.com/oauth2/v2/',
# 'request_token_params':{
# 'scope': 'email profile'
# },
# 'access_token_url':'https://accounts.google.com/o/oauth2/token',
# 'authorize_url':'https://accounts.google.com/o/oauth2/auth',
# 'request_token_url': None,
# 'consumer_key': CONSUMER_KEY,
# 'consumer_secret': SECRET_KEY,
# }
# }]
# When using LDAP Auth, setup the ldap server
# AUTH_LDAP_SERVER = "ldap://ldapserver.new"
# When using OpenID Auth, uncomment to setup OpenID providers.
# example for OpenID authentication
# OPENID_PROVIDERS = [
# { 'name': 'Yahoo', 'url': 'https://me.yahoo.com' },
# { 'name': 'AOL', 'url': 'http://openid.aol.com/<username>' },
# { 'name': 'Flickr', 'url': 'http://www.flickr.com/<username>' },
# { 'name': 'MyOpenID', 'url': 'https://www.myopenid.com' }] |
|
@bcb do you have issues with gunicorn workers starting up after enabling RBAC? Airflow starts the workers, but they fail to start up in time. They are then killed and the process repeats itself until it finally succeeds. That usually takes 10-20min. I disabled all volumes to make sure it is nothing with my config, but that had no effect. The odd thing is that it works on my local machine, but fails on my digital ocean server... Does anyone have an idea how to even start debugging this? |
|
I was able to solve this issue... For some reason the RBAC interface uses a lot of CPU on startup. If you are running on a low powered server, this can cause a very slow webserver startup and permanently high CPU usage. I have documented this bug as AIRFLOW-3037. To solve it you can adjust the config: |
|
Thanks @KimchaC for your inputs. By the way how did you do to locate the problème with webserver_config.py and stuff ? Anyone trid to filter dogs by owner using RBAC ? |
|
@cmourouvin it was @bcb's comment. I made the I haven't tried filtering dag. |
|
By the way the bad file location is reported to airflow bug ? It should be to avoir some workaround :\ |
|
I haven't reported it, because I'm not sure if it is a bug in all airflow installs or just in this docker file. If it was a bug in airflow itself, I am sure someone would have noticed. So I suspect it is a bug in this image. |
|
I followed these steps:
Log when I created the user: but I get this error when I hit webserver:
|
|
I do see apache/airflow#3937 got merged 20 hours ago, not sure if this is related or if I am missing something. |
|
Ok, it worked when I manually decremented the version of flask-appbuilder in my docker file - && pip install 'flask-appbuilder==1.11.1' \ |
|
Looking at the Airflow source code, this issue can be resolved by explicitly setting the AIRFLOW_HOME environment variable in your Docker. ARG AIRFLOW_HOME=/usr/local/airflow If this environment variable is not set, AIRFLOW_HOME will default to /usr/local/airflow/airflow. This "bug"/feature only seems to effect the RBAC code. |
|
Awesome, looks like that would solve it. @puckel would you consider adding it to your Dockerfile so we don't have to do it in our's? |
@puckel Adding the environment variable AIRFLOW_HOME, would also solve a few other issues with the current Docker build. In your default build I can't for example run |
Further, it gave me the error |
|
@KimchaC thanks for taking the time to write all of this up, you saved me! |
|
when I run create_user, LocalExecutor is changed to Sequential airflow@ac85221fed9f:~$ airflow create_user -r Admin -u test -e admin@example.com -f test -l test -p test UPDATE Use entrypoint.sh instead. docker exec -it <container_id> /entrypoint.sh bash |
|
I had to change a few things before I managed to make this work; the gist of it was that the database connection was not set properly:
At this point you can continue with Note: I'm not sure if step 3 is always necessary, as the default webserver config file will be picked up anyways [or should be at least] and the SQL connection for FAB is set there from what you set in step 4 |
|
This comment chain was legendary. I got everything working and eventually created a new docker image that is very much influenced by puckel.docker-airflow and incorporates all the comments here. |
|
@KimchaC Have you encountered this problem https://issues.apache.org/jira/browse/AIRFLOW-3442 when working with google OAuth ? |
|
@OmerJog Can you tell me how Google OAuth works with the RBAC? When someone logs in for the first time, his username/e-mail will be written into the database? How does it work rolewise? Will their roles default to Viewer until an Admin changes it? |
|
@holypriest If you leave the AUTH_USER_REGISTRATION = True option the user can create their own access and their role will be defined by: AUTH_USER_REGISTRATION_ROLE = "Viewer" - I had some problems in redirecting the login with Public permission, so I switched to Viewer. The user will be inserted in the database after completing his / her registration, with name, surname and email. |
|
I was able to get everything working by following this chain. Running docker exec -it <container_id> /entrypoint.sh bash helped a lot.. I was so confused for hours (days) before this. I added a create user script in my entrypoint file, and I also added a method to read Docker Secrets rather than hard coding the SQLAlchemy URI or any passwords into a file on my gitlab repo. |
|
I'm trying to integrate Google OAuth and RBAC. Ideally I want users to not be able to see anything when they first go to the webserver, but only have an option to sign in with Google. Then I want them to only have the 'View' role. This is going well so far, but I don't know how I can modify a user's role (i.e make them an Admin) when they have signed in/registered with Google. I can see the users in the database with Anyone know how to change a user's role? I assume I'm gonna have to use fabmanager. |
|
I found a solution... Have Google authorized users start with the View role by setting If you want to elevate a user's privileges you can connect to your airflow db and manually update the would set user with id 1 to have the 'Admin' role (role_id = 1). You can get the role ids by doing |
|
;tldr - summary of how to use rbac+google_oauth+optional env vars for all configs For future reference for anyone looking at this issue, and has trouble compiling the pieces together, webserver_config.py:
airflow.cfg [webserver]:
in order to change the first registered user into an admin user, follow 'eightlimbed' comment, set the 'role_id' of your user to 1 on the database of you running for airflow, which by default should be the 'Admin' role also, on google's api credentials console:
I am not sure both are required [one of them for sure is], but from my testings and errors I've seen, both getting requests from either the browser or google.. caveats:
hopefully this also helps someone who is looking to fork this docker image, and use custom made airflow branch\version.. Credits on this guide comment is to everyone who already commented on this issue, Thank you all! edits: formatting |
|
@asaf400 Have you faced https://issues.apache.org/jira/browse/AIRFLOW-5462? I have enabled rbac+google_oauth and now when I click on login button it gets redirected to https://your-airflow.domain.com/oauth-authorized/login, I get this error
|
|
@ayush-san |
|
I am using |
|
@ayush-san Ahh, I see, I missed you were using 3.7 (as do I currently) I'll verify I didn't change any code that is related to that.. edit: typo |
|
@ayush-san Check out this issue: specifically these comments: Also, try to completely DROP all the tables, I think I have managed to solve my issue with the help of those comments and DROPing the db Hope that helps you, Report back if you succeed.. |
|
There was an issue with my webserver_config.py which was causing this issue. We only need to whitelist Since my webserver_config has whitelisted a wrong domain I was getting the flash message of |
|
@ayush-san Yeah, the docs really are awful about some stuff, that may well be what I encountered and I fixed it same way you did, without being able to recollect that it was what helped me.. |
|
@ayush-san @asaf400 |
|
@anpjai I cant help with with LDAP since I didn't do it myself |
|
After using Run Anyone could help and explain this? @KimchaC @bcb |
|
@benbendemo, at least for me the create_user didn't know about the postres connection db, user and password. I'm using this code in a re-packed image import airflow
from airflow import models, settings
from airflow.contrib.auth.backends.password_auth import PasswordUser
from sqlalchemy import create_engine
from os import getenv
POSTGRES_USER = getenv('POSTGRES_USER')
POSTGRES_PASSWORD = getenv('POSTGRES_PASSWORD')
POSTGRES_DB = getenv('POSTGRES_DB')
user = PasswordUser(models.User())
user.username = 'admin'
user.email = 'admin@test.com'
user.password = 'test'
user.firstname = 'admin'
user.lastname = 'test'
engine = create_engine(f'postgresql://{POSTGRES_USER}:{POSTGRES_PASSWORD}@postgres:5432/{POSTGRES_DB}')
session = settings.Session(bind=engine)
session.add(user)
session.commit()
session.close()Apparently the create user script has lost the context and doesn't know how to connect to the DB |
Thank for your advice, i've tried your way but still in vain. From my test result, i could connect Postgres and query results from the db with the method you provided. However creating new user didn't take into effect, i even tried the root user in airflow container. |
|
Thanks for your comments!
|
|
@benbendemo, use command |
|
I have to say that, ever since I updated the image to the official iv'e been using some sort of what @harry-cai says. You have a ready installation of airflow within the docker container. So you run a shell from within and run this command. You can also do this form local installation (outside the container) if you have airflow installed and declared the environment variable Theres an open Issue about this in the official airflow repo. They are working on providing an easy way of bootstrapping a container |
|
Hi, I enabled RBAC for airlfow 1.10.10, I am able to create user with 'airflow create-user' on webserver container bash. When I login to http://localhost:8080 with username and pwd, Recent Tasks, Last Run and DAG Runs keep spinning, is this bug in UI? How do I fix it?
|
|
well @aki1977 how did you enable it? |
|
I had issues enabling RBAC. After reading this forum, I discovered how to fix it. For those who might have similar issues:
See. I have a working example here: advance_scraping My issue was generated by mounting |
|
Create a user by using following command : |
|
tried following this but I still get the "no user yet created" error. Anyone have up to date solutions? |
|
This repo is not maintained animore. My personal recommendation is using either a managed solution in prod (Astronomer, GCP composer...) And use 'astro' CLI to manage local environments |
and others
Hi,
Thanks for the quick update to 1.10.0!
One of the most exciting features is RBAC support that allows to add users with different roles and permissions.
I tried following the instructions on...
https://wecode.wepay.com/posts/improving-airflow-ui-security
https://github.com/apache/incubator-airflow/blob/master/UPDATING.md
But I wasn't able to add users to the database and get it working.
I added
Which successfully enabled RBAC and showed the login screen on the webserver.
Next I opened a bash shell and ran the webserver to generate the
/usr/local/airflow/airflow/webserver_config.pyIt was created successfully and I didn't modify it as want to use the default
AUTH_TYPE = AUTH_DB.Then I ran
I get the output:
But the user is not created. I checked the postgres database and I can see there are several ab_ tables there, but they are all empty.
And when I re-run the user_add command I also get the same success message, whereas if the user existed it should say
admin already exist in the dbaccording to the cli.py airflow code.Airflow also constantly logs to the console:
So I've tried running fabmanager, but it always quits with the error
Was unable to import app Error: No module named 'app'and even when I tried running it from the app's dir (/usr/local/lib/python3.6/site-packages/airflow/www_rbac) it failed with a different exception.I have also tried preserving webserver_config.py by adding it to my volumes:
But this didn't make a difference either.
Another thing I noticed is that I tried enabling the option
AUTH_USER_REGISTRATION = True. It's supposed to allow self-registration of users. I am not sure how this feature works, but I expected to see an option on the login screen... yet nothing showed up.So perhaps something is broken and the config isn't used at all?
I can't think of anything else to try for now. Hopefully one of you guys has an idea.
The text was updated successfully, but these errors were encountered: