Added Shiro security

[hayssams]

Added shiro security. HTTP calls and websockets communications are both protected.

[hayssams]

Support for authentication only.
Notes are not shared among users.
To share notes, you'll have to share the username/password among different users.

[Leemoonsoo]

Hi @hayssams, Thanks for contribution. Let me test this branch.

[hayssams]

Hi @Leemoonsoo, did you get a chance to test this PR ?

[Leemoonsoo]

Sorry for late responding and thanks again for contribution.

In my understanding, this branch trying to provides 1) security of rest/websocket api. 2) multi-tenant environment for notebook save/load.

Because of this PR brings wide range of changes, we have couple of things to discuss.

• providing login window
• displaying current user
• how to merge changes https://issues.apache.org/jira/browse/ZEPPELIN-26
• migration script for notebook directory structure change.
• notebook multi tenant model (per user / shares all, due to any user actually can read any notebook from filesystem api on Spark/Pyspark interpreter)

I'm feeling like if it is possible to separate 1) security of rest/websocket 2) multi-tenancy of notebook into different issue, it'll be much faster to review and test.

What do you think?

[hayssams]

You're right, this PR addresses 1) & 2).

Regarding the points you raised

• The login window is useful if we want to support form auth in addition to the basic auth this PR provides. This may be added in another PR easily with almost no impact to the current code.
• Right, we need to display the current user. Do you want me to add it at the top of the note with a logout button ?
• Zeppelin-26 : Did not look at it yet.
• Migration script : that should be easy. to just move existing notes to the anonymous subdirectory with no impact on the existing code
• Multitenant : Would starting the "interpreter.sh" script with a different identity solve the problem ?

Actually feature 2) involves a very small amount of code (load/save notes in user directory).

I am personally interested by feature 2). let me know if you want me to complete the other points you raised first or if you prefer to stick to feature 1). I'll respect your choice.

[anthonycorbacho]

Hi,

For me I am really confuse about the change on the rest api, why do we need principal and ticket? and why this is part of the rest api url?
Why dont we create cookies for that?
Why do you refer stormpath in the shiro ini file?
Why the login page should be handled on an other PR?
since in the ini file the urls are anonimus /** = anon how do you authenticate users?
how the sessions are handled?

[hayssams]

Hi Anthony

1. You're right using cookies for the HTTP API would be a much better choice.
2. Regarding storm path, The comments you find in the shiro.ini file come from the default shiro.ini setup; Wen can safely ignore/remove them
3. The login page is not required for shiro to work. We can use basic auth by uncommenting line 17 in the shiro.ini file (/** = authcBasic). The browser will popup a login dialog without any coding required.
4. By default auth is not activated. To activate it, simply comment line 16 and uncomment line 17 in shiro.ini. The file shiro.ini need to be accessible through the classpath.

[hayssams]

I missed your last question about session handling.
I do not use sessions since the credentials are passed with each call.

[elbamos]

I think we should have an organized plan to review and address this PR.

I propose that the PR be merged into a new branch of Zeppelin, where the community as a whole will have an easier time reviewing and making PRs on the branch.

Then I propose that we review the aspects of this separately. By reviewing/approving/voting-on each aspect in sequence, we will have a clear idea of where the PR is in process, and that will make it easier to move PR forward to final merge (if that's what we want to do). The aspects I see are:

• Feature set -- is this the kind of security interface we want to put in Zeppelin?
• Architecture -- Is this (shiro) how we want to implement security?
• Does it work?
• Do we have adequate tests to be able to support and maintain it?
• Is it adequately documented that people will be able to use it?

The reason I propose a more formal process is that this PR has been sitting here for 8 months now. With that, and with some 82 (!!!) commits, a more structured approach seems like the best way to resolve this PR (one way or the other).

(In full disclosure -- My only view about this is that since its security-related, it needs to have mounds of tests to make sure the security doesn't get compromised by later commits.)

[jongyoul]

In addition to @elbamos's comments, I hope that the community would understand that it's too hard to review a kind of this PR which has changed widely because committers may not know features well. I, however, also know some features take extensive changes. In this case, I propose that a contributor makes a feature Jira issue that includes several sub-tasks, and makes a - relatively - small PR for every sub-task. It would help review, test and merge the PR easily.

[rconline]

@jongyoul @elbamos @hayssams this change is an important one, and we should try to get this in. My two cents on how we could divide the whole pull request into:

• inclusion of libraries and dependencies
• Addition of security filter, changes in Angular Object, configurable - turned off by default. With a basic set of test-cases, which ensure the changes are non-breaking.
• Front-end changes, which include -display of username, accepting header parameters of principal, ticket.
• Subsequently, addition of more test cases, before we call it done. Till such time we keep the feature as default off.
  (Each of the above steps will have user guides/documentation).

If the committers can provide some bandwidth for review, I'm willing to make the above changes + documentation.

As side notes:

• We will need to include the notion of proxy-users at sometime, which enable command execution of secure spark/hdfs clusters
• Subsequently, there will have to be a mechanism of how to provide default access to the cluster users, either by way of import of AD/LDAP users, through some auth_to_local kind of rules.

I'm of the opinion that Shiro is a good strategy, larger projects such as https://knox.apache.org/ use Shiro, for HDFS Api gateway security.

[hayssams]

@elbamos @jongyoul @rconline
Let me write down the whole set of changes I've made and how they may be split in my view in smaller commits.
There are also different strategies in using shiro. I would like to submit them and take into account the one you advise to use.

[jongyoul]

@hayssams Sure! You can split this into several commits because this PR is your contribution and we will thank you so much of that. If you want to do this, would you mind creating Jira issue or linking a related issue that is already created?

[jeffsteinmetz]

Is there an example of how shiro would be pluggable with other forms of
authentication and authorization?
Such as Java Web tokens (JWT), which require storing a secret. And the JWT
contains authorization roles.
On Dec 29, 2015 6:11 AM, "rconline" notifications@github.com wrote:

@jongyoul https://github.com/jongyoul @elbamos
https://github.com/elbamos @hayssams https://github.com/hayssams this
change is an important one, and we should try to get this in. My two cents
on how we could divide the whole pull request into:

• inclusion of libraries and dependencies
• Addition of security filter, changes in Angular Object, configurable
• turned off by default. With a basic set of test-cases, which ensure the
  changes are non-breaking.
• Front-end changes, which include -display of username, accepting
  header parameters of principal, ticket.
• Subsequently, addition of more test cases, before we call it done.
  Till such time we keep the feature as default off. (Each of the above steps
  will have user guides/documentation).

If the committers can provide some bandwidth for review, I'm willing to
make the above changes + documentation.

As side notes:

• We will need to include the notion of proxy-users at sometime, which
  enable command execution of secure spark/hdfs clusters
• Subsequently, there will have to be a mechanism of how to provide
  default access to the cluster users, either by way of import of AD/LDAP
  users, through some auth_to_local kind of rules.

I'm of the opinion that Shiro is a good strategy, larger projects such as
https://knox.apache.org/ use Shiro, for HDFS Api gateway security.

—
Reply to this email directly or view it on GitHub
#53 (comment)
.

[rconline]

@hayssams please do that, it will be great. I can help with Junits, Docs and separation, should you need it, will commit into your branches, which can then be submitted as PRs.

[hayssams]

@jongyoul @elbamos @rconline
I have just added a short security readme that may allow us to choose the best strategy in adding Shiro. Basically the two questions are :

1. Is the way we handle web socket security a good pattern
2. Should we rely on the ThreadLocal pattern or explicitly pass the username in function calls or should we use a different approach ?

Once we decide how to move on, I think that we will be able to focus on the implementation details and how this PR should be split.

Waiting to hear from you.

[elbamos]

@hayssams Can you provide a very step-by-step intro readme? Default username/pass, what the authentication backend is, how users/groups are added, how to make notebooks belong to one user or another... That very basic stuff?

[hayssams]

@elbamos @jongyoul @rconline
I created a new PR (#586) which implements HTTP et Websocket security but does not implement note ownership. This means that a user must authenticate to access Zeppelin but once authenticated, he will have access to all notes. That's what @Leemoonsoo wanted to do first.

For easier review, this new PR (#586) contains only two commits. The first one implements HTTP Auth and the second one Websocket authentication .

Once PR # 586 is merged / validated, I suggest that we move forward with note ownership and permissions.

[jeffsteinmetz]

One comment: why is Stormpath (a commercial auth service unrelated to Zeppelin) mentioned in the comments for this PR?
See:
https://github.com/apache/incubator-zeppelin/pull/586/files#diff-9b7b90bbe0069ba81385d11637dcd4e4R27

From: Hayssam Saleh
Reply-To: apache/incubator-zeppelin
Date: Thursday, December 31, 2015 at 10:06 AM
To: apache/incubator-zeppelin
Cc: Jeff Steinmetz
Subject: Re: [incubator-zeppelin] Added Shiro security (#53)

@elbamos @jongyoul @rconline
I created a new PR (#586) which implements HTTP et Websocket security but does not implement note ownership. This means that a user must authenticate to access Zeppelin but once authenticated, he will have access to all notes. That's what @Leemoonsoo wanted to do first.

For easier review, this new PR (#586) contains only two commits. The first one implements HTTP Auth and the second one Websocket authentication .

Once PR # 586 is merged / validated, I suggest that we move forward with note ownership and permissions.

—
Reply to this email directly or view it on GitHub.

[hayssams]

You are right that should be removed.
I took it from the shiro sample.
I am removing the whole section right away.

On 31 Dec 2015, at 20:27, Jeff Steinmetz notifications@github.com wrote:

One comment: why is Stormpath (a commercial auth service unrelated to Zeppelin) mentioned in the comments for this PR?
See:
https://github.com/apache/incubator-zeppelin/pull/586/files#diff-9b7b90bbe0069ba81385d11637dcd4e4R27

From: Hayssam Saleh
Reply-To: apache/incubator-zeppelin
Date: Thursday, December 31, 2015 at 10:06 AM
To: apache/incubator-zeppelin
Cc: Jeff Steinmetz
Subject: Re: [incubator-zeppelin] Added Shiro security (#53)

@elbamos @jongyoul @rconline
I created a new PR (#586) which implements HTTP et Websocket security but does not implement note ownership. This means that a user must authenticate to access Zeppelin but once authenticated, he will have access to all notes. That's what @Leemoonsoo wanted to do first.

For easier review, this new PR (#586) contains only two commits. The first one implements HTTP Auth and the second one Websocket authentication .

Once PR # 586 is merged / validated, I suggest that we move forward with note ownership and permissions.

—
Reply to this email directly or view it on GitHub.

—
Reply to this email directly or view it on GitHub #53 (comment).

EBIZNEXT http://www.ebiznext.com Java / .NET */ DevOps / BigData / UX
/ Agile*
23 rue Vernet 75008 Paris
https://maps.google.com/?q=23%20rue%20Vernet%2075008%20Paris

[hayssams]

@jongyoul
I just created 2 JIRA issues.
ZEPPELIN-548 for Zeppelin Authentication
ZEPPELIN-549 for Zeppelin Authorisation

[hayssams]

I am closing this PR and suggest that we move the discussion to #586

[guptabandhu25]

Hi Hayssams , the authentication letter is applied but in terms of exception as i am not getting any window for putting my username and password . I am in the customer network so it is not asking for any ausername and password .

Showing the message

ERROR [2016-06-02 14:27:22,265]({qtp513169028-34} NotebookServer.java[onMessage]:179) - Can't handle message
java.lang.Exception: Anonymous access not allowed
at org.apache.zeppelin.socket.NotebookServer.onMessage(NotebookServer.java:110)
at org.apache.zeppelin.socket.NotebookSocket.onMessage(NotebookSocket.java:56)
at org.eclipse.jetty.websocket.WebSocketConnectionRFC6455$WSFrameHandler.onFrame(WebSocketConnectionRFC6455.java:835)
at org.eclipse.jetty.websocket.WebSocketParserRFC6455.parseNext(WebSocketParserRFC6455.java:349)
at org.eclipse.jetty.websocket.WebSocketConnectionRFC6455.handle(WebSocketConnectionRFC6455.java:225)
at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:667)
at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
at java.lang.Thread.run(Thread.java:745)