Security

Introduction

JEM implements security to face 2 main topics, like: Authentication, from both cluster connection and user interfaces (web and Eclipse) perspective Authorization, from both job execution and user interface perspective.

Job and User Interface Security

Using Apache Shiro, JEM has a powerful and easy-to-use security framework that performs authentication, authorization, and session management. Part of following documentation has been taken by Apache Shiro documentation.

Authentication

Authentication is the process of identity verification (you are trying to prove a user is who they say they are). To do so, a user needs to provide some sort of proof of identity that your system understands and trusts. Here are some entities currently used to authenticate users:

Subject: security specific user 'view' of JEM user. Basically, it is anything or anyone communicating with JEM.
Principals: a subjects identifying attributes (for instance: First name, last name, organizational unit, username).
Credentials: secret data that are used to verify identities (for instance: passwords, biometric data, x509 certificates).
Realms: security specific DAO (Data Access Object), software component used to communicate with a backend data source. If you have usernames and password in LDAP, for instance, then you would have an LDAP Realm that would communicate with LDAP.

Authorization

Authorization, also known as access control, is the process of managing access to resources. In other words, controlling "who has access to what" in an application. Authorization has three core elements that we reference quite a bit: permissions, roles, and users.

Permissions

Permissions represent the most atomic element of a security policy. They are fundamentally statements about behaviour and represent explicitly what can be done in an application. A well-formed permission statement essentially describes resources and what actions are possible when a subject interacts with those resources.

Some examples of permission statements:

Read a file by a job
View the running queue page
Cancel job documents
Delete permission

JEM resources will support the typical CRUD (create, read, update, delete) actions, but any action that makes sense for a particular resource type is ok. The basic idea is that permission statements at a minimum are based on resources and actions.

When looking at permissions, probably the most important thing to realize is that permission statements have no representation of who can perform the represented behaviour. They are only statements of what can be done in an application. Permission statements reflect behavior (actions associated with resource types) only. They do not reflect who is able to perform such behavior.

Defining who (users or org-units) is allowed to do what (permissions) is an exercise of assigning permissions to users in some way.

Permissions must be grouped in a role and roles can be associated with one or more users or org-units. In JEM, you assign roles to users directly or per org-units: a role assigned to an org-unit is implicitly assigned to all its users.

Here are the list of JEM defined permissions:

Views permissions

Label	Scope	Description
`view:input`	User interface	authorization to view the INPUT queue panel
`view:running`	User interface	authorization to view the RUNNING queue panel
`view:output`	User interface	authorization to view the OUTPUT queue panel
`view:routing`	User interface	authorization to view the ROUTING queue panel
`view:nodes`	User interface	authorization to view the NODES panel
`view:admin`	User interface	authorization to view the ADMINISTRATION panel
`view:roles`	User interface	authorization to view the ROLES panel
`view:status`	User interface	authorization to view the STATUS panel
`view:gfsexplorer`	User interface	authorization to view for GFS panel
`view:swarm-nodes`	User interface	authorization to view for SWARM panel
`view:*`	User interface	authorization to view all panels

Jobs actions permissions

Label	Scope	Description
`jobs:purge`	User interface	authorization to purge jobs in input, output and routing queues
`jobs:hold`	User interface	authorization to block (hold) jobs in input, output and routing queues
`jobs:release`	User interface	authorization to release (unhold )jobs in input, output and routing queues
`jobs:cancel`	User interface	authorization to cancel jobs while executing
`jobs:kill`	User interface	authorization to kill jobs while executing
`jobs:submit`	User interface	authorization to submit job into JEM
`jobs:update`	User interface	authorization to update fields of job (domain, affinity, priority, memory) in input queue
`jobs:*`	User interface	authorization to all jobs actions

Nodes actions permissions

Label	Scope	Description
`nodes:start`	User interface	authorization to start node, i.e to change its status from DRAINED to STARTING status
`nodes:drain`	User interface	authorization to drain nodes, i.e to change its status from INACTIVE to DRAINED
`nodes:update`	User interface	authorization to update some attributes of node (domain, static affinities, memory and parallel jobs)
`nodes:*`	User interface	authorization to all nodes actions

Roles actions permissions

Label	Scope	Description
`roles:create`	User interface	authorization to create new roles, assigning permissions and subjects (and/or org-units)
`roles:delete`	User interface	authorization to remove existing roles. Some roles are not removeable, because mandatory to JEM (see Roles section)
`roles:update`	User interface	authorization to change definitions of roles
`roles:read`	User interface	authorization to see definitions about roles
`roles:*`	User interface	authorization to all roles actions

Resources permissions

Label	Scope	Description
`resources:create`	User interface	authorization to create new common resources
`resources:delete`	User interface	authorization to remove already existing common resources
`resources:update`	User interface	authorization to update already existing common resources
`resources:read`	User interface	authorization to get all information and definition about already existing common resources
`resources:*`	User interface	authorization to fully manage all common resources

Certificates permissions

Label	Scope	Description
`certificates:create`	User interface	authorization to upload a new certificate for a user
`certificates:delete`	User interface	authorization to remove already existing certificates
`certificates:read`	User interface	authorization to read existing certificates
`certificates:*`	User interface	authorization to fully manage all certificates actions

Swarm actions permissions

Label	Scope	Description
`swarm-nodes:start`	Node management	authorization to start swarm inside a JEM cluster
`swarm-nodes:drain`	Node management	authorization to drain swarm of specific JEM cluster
`swarm-nodes:viewconfig`	Node management	authorization to see swarm configuration
`swarm-nodes:editconfig`	Node management	authorization to edit and change swarm configuration
`swarm-nodes:*`	Node management	authorization to all swarm actions

Global-File-System actions permissions

Label	Scope	Description
`gfs:data`	User interface	authorization to see DATA folder of JEM global file system
`gfs:library`	User interface	authorization to see LIBRARY folder of JEM global file system and to write files on LIBRARY folder
`gfs:sources`	User interface	authorization to see SOURCES folder of JEM global file system and to write files on SOURCES folder
`gfs:classpath`	User interface	authorization to see CLASSPATH folder of JEM global file system and to write files on CLASSPATH folder
`gfs:binary`	User interface	authorization to see BINARY folder of JEM global file system and to write files on BINARY folder
`gfs:*`	User interface	authorization to see all folders of JEM global file system

Administrations actions permissions

Label	Scope	Description
`administration:cluster:folder`	User interface	authorization to see all actions on JEM cluster
`administration:cluster:configuration`	User interface	authorization to edit configuration files of JEM cluster
`administration:cluster:workload`	User interface	authorization to see workload statistics on JEM cluster
`administration:cluster:grs`	User interface	authorization to see resources contention inside JEM cluster
`administration:cluster:redo`	User interface	authorization to see all redo pending statements, caused by database failure
`administration:cluster:gfs-usage`	User interface	authorization to see utilization of GFS
`administration:cluster:*`	User interface	authorization to have all actions on JEM cluster
`administration:nodes:folder`	User interface	authorization to see all actions on JEM nodes
`administration:nodes:configuration`	User interface	authorization to edit configuration files of JEM nodes
`administration:nodes:commands`	User interface	authorization to perform commands on JEM nodes
`administration:nodes:system`	User interface	authorization to see system resources consumption of JEM nodes
`administration:nodes:queues`	User interface	authorization to see distribution of JOB queues on JEM nodes
`administration:nodes:*`	User interface	authorization to have all actions on JEM nodes
`administration:queues:folder`	User interface	authorization to see all actions on JEM queues
`administration:queues:current`	User interface	authorization to see current size of JOB queues in JEM cluster
`administration:queues:statistics`	User interface	authorization to see short history and current size of JOB queues in JEM cluster
`administration:queues:internalMaps`	User interface	authorization to see internal maps size of JEM cluster
`administration:queues:*`	User interface	authorization to have all actions on JEM queues
`administration:security:folder`	User interface	authorization to see all actions on JEM security utilities
`administration:security:secret`	User interface	authorization to have the web utility to encrypt values using JEM cluster key
`administration:security:certificate`	User interface	authorization to have the web utility to manage certificates related to users
`administration:security:*`	User interface	authorization to have all actions on JEM security utilities

Search permissions

Label	Scope	Description
`search:jobs:[regex]`	User interface	authorization to search in the job queues by job names filter. A filter could be a regular expression
`search:nodes:[regex]`	User interface	authorization to search in the nodes queue by hostname or ip address filter. A filter could be a regular expression

File permissions

Label	Scope	Description
`file:read:[regex]`	JCL	authorization to read files whose name matches `[regex]`
`file:write:[regex]`	JCL	authorization to write files whose name matches `[regex]`
`file:execute:[regex]`	JCL	authorization to execute programs whose name matches `[regex]`
`file:*:[regex]`	JCL	authorization to read and write files whose name matches `[regex]`

Data Source permissions

Label	Scope	Description
`datasource:[type]:[regex]`	JCL	authorization to use data sources of type `[type]` whose name matches `[regex]`.

Surrogate permissions

Is the control about the possibility for a Subject to impersonate a different Subject (and get all the latter's permissions). It's used mainly on JCL, to submit a job with different (and more powerful) grants than the submitter.

Label	Scope	Description
`surrogate:[pattern]`	JCL	authorization to impersonate a subject whose ID matches `[pattern]`

Roles

A role is a named entity that typically represents a set of behaviors or responsibilities. These behaviors translate to things you can or can't do with JEM. Roles are assigned to user accounts (or org-unit), so by association, users can 'do' the things attributed to various roles.

A role is essentially a named collection of actual permission statements. In this form, JEM knows exactly what it means to have a particular role or not. Because it is known the exact behavior that can be performed or not, there is no guessing or implying what a particular role can or can not do (Resource-Based Access Control).

JEM provides a set of out-of-the-box roles with some permissions: they are already assigned and can't be changed nor removed. To implement specific policies, you can (actually, you're suggested to) create new roles.

Here is the list of JEM pre-defined roles:

Label	Description	Permissions assigned
administrator	can manage JEM platform, has all permissions and can do everything inside JEM	`*` all permissions
grantor	can manage all permissions	`view:roles` has roles view panel to manage roles
		`roles:*` all actions on roles
operator	it can manage jobs and nodes, checking how they are working inside JEM	view:input to manage input queue
		`view:running` to manage running queue
		`view:output` to manage output queue
		`view:routing` to manage routing queue
		`jobs:*` has all permissions about jobs actions
		`nodes:*` has all permissions about nodes actions
		`search:jobs:*` has all permissions to search all jobs
		`search:nodes:*` has all permissions to search all nodes
developer	can see its own jobs and how they are working	`view:input` input queue panel to check its own jobs
		`view:running` running queue panel to check its own jobs
		`view:output` output queue panel to check its own jobs
		`view:routing` routing queue panel to check its own jobs

Users

A user essentially is the 'who' of JEM. As we've covered previously however, the Subject is really JEM 'User' concept. Every user has an Organizational Unit (org-unit) related to him (this is the usual way to map organizations, for example in LDAP).

Users or org-units (via Subjects) are allowed to perform certain actions in JEM through their association with roles and permissions. JEM defines exactly how a Subject is allowed to do something or not and defines exactly how authorization will work, by Realm concept. JEM Realm implements a custom authorization data model, managed by grantors via web user interface.

For instance, if your Authentication/Authorization model is LDAP based, you'll need an LDAP Realm.

Cluster Security

Cluster security, in JEM, means that only authorized nodes/clients should be able to connect to cluster. With the default installation, where the login protocol is disabled, this issue is accomplished with the use of the environment password either for clients (jem_submit and jem_localhost_submit) and nodes. That is, to be able to connect to the cluster, the clients and the nodes must present a valid password (the one define during installation). This level of security, could not be enough for some users and that's why JEM the BEE give the possibility to raise Cluster Security with the use of a login protocol (protected by encryption and One Time Password). The login process expect for a node to be able to access the shared symmetric key from the environment keystore present in the persistence path of the Global File System while for a client to present a private pem key relative to the X509 certificate store in the user certificates keystore present in the persistence path of the Global File System (see keystores for more informations). As you will see in the next paragraphs, this feature will be as easy to install as setting a property but once you installed it you must be aware that to be able to submit jobs you will need to provide to the JEM the X509 certificate of the authorized users.

Enable Login Protocol

In the create node paragraph are explained all the installation properties in detail. Let remark here those properties needed to enable the login protocol with an example:

#------------------------------------------------------
# Node login security module properties
#
# this section can enable a login process protected
# by a symmetric key. For more detail see documentation 
#-------------------------------------------------------
# optional, valid value are true or false, default is false
# if set to true and no one of the other properties are set, by default the environment will create and use
# a symmetric key store in the file: jem.persistence/jemencryption_[jem.environment.name].key
jem.login.protocol.enable=true

# is the prefix of the name of the file for the keystore containing the symmetric 
# key for encryption. The complete name will be [prefix]_[jem.environment.name].keystore
# the file will be placed in the persistence path (see property jem.persistence)
jem.keystore.name=

# mandatory, is the password for both keystores present in the persistence path. 
# One keystore relative to the cluster, contains the symmetric used during login phase.
# The other keystore is used to contains public authorized user certificate 
jem.keystore.pwd=jem_password

# mandatory, is the password of the symmetric key used for encryption and decryption 
jem.crypt.key.pwd=jem_password

Setting properties as the above example, will configure JEM the BEE in a high security manner.

Keystores

When you install a JEM instance, either you enable or not the login protocol, 2 keystores will be generated and put in the persistence path of the GFS (Global File System):

The environment keystore. A JCEKS keystore contains a random auto-ssssssgenerated TRIPLE DES symmetric key that will be used during the login phase if the login protocol is enabled. The alias of the symmetric key will be the name of the environment. The password for the keystore and the password for the key are chosen by the user during installation phase with the properties: jem.keystore.pwd, jem.crypt.key.pwd. The name of this keystore will be [jem.keystore.name]_[jem.environment.name].keystore.

To investigate or alter the content of the keystore using keytool here a couple of examples:_

Insert symmetric TripleDES key

keytool -genseckey -alias ENV1 -keyalg TripleDES -keysize 168 -storetype JCEKS -keystore "c:\tmp\jem.keystore"

Check if key is in the keystore accessing it by alias

keytool -list -storetype JCEKS -keystore "c:\tmp\jem.keystore" -alias jemkey

The user's certificates keystore. A JKS keystore, used only if the login protocol is anabled, and it will eventually contain the X509 certificates provided by the user that want to submit jobs. The X509 certificate can be add or remove via web-interface. The name of this keystore will be user_certificates_[jem.environment.name].keystore_. The password is the same of the environment keystore. For both keystores JEM provides a backup.

To generate a self signed X509 certificate with the relative private key (that can be protected by password) you can use the following openssl command:

openssl req -new -x509 -newkey rsa:1024 -days 365 -keyout private.key -out userCertificate.crt

where

private.key will be the private key. Note that a password will be asked for it (Enter PEM pass phrase:)
userCertificate.crt will be the X509 user certificate that contain the public key
365 is the validity of the certificate in term of days

Login Protocol

The login protocol is a set of xml requests made by a client (a node that wants to join the cluster or a client asking the cluster for some operations), and a set of xml responses made by a server (a node that decides if the client is authorized to join the cluster). In JEM, this has been implemented with the use of a feature offered by the Hazelcast framework: the Socket Interceptor. As explained in the Hazelcast documentation, it is possible to implement MemberSocketInterceptor and SocketInterceptor interface to create a custom login protocol. For this purpose we have implemented the

org.pepstock.jem.node.security.socketinterceptor.NodeInterceptor that describes the request process made by a node joining the cluster and the response process made by the cluster when a client or a node try to connect.
org.pepstock.jem.node.security.socketinterceptor.SubmitInterceptor that describes the request process made by a client that try to connect to the cluster. You can see if login protocol is enabled looking in the jem-env-hazelcast configuration:

<network>
    <socket-interceptor enabled="true">
        <class-name>org.pepstock.jem.node.security.socketinterceptor.NodeInterceptor</class-name>
        <properties>
			... keystores informations
        </properties>
    </socket-interceptor>
</network>

Security, during the login phase is garanteed by the use of encryption and OTP (One Time Password). While the protocol remains the same for node and client, encryption policy is slightly different. For nodes we will use a symmetric key alghoritm (and thus a symmetric key) while for client we will use an asymmetric key algorithm (and thus a public-private key pair).

Let see the protocol in action for a node that want to join the cluster:

The connecting node (the client) sends a BASE64-encoded request telling who he is and asking for a one time password to login:

<request>
    <address>197.12.1.1:58824</address>
    <subjectID>JEM_NODE</subjectID>
    <operation>
        <name>getPassword</name>
    </operation>
</request>

2 The Server:

Decodes and the request
Checks if the request is valid (veryfing also if the address is the same that come from the socket connection), otherwise sends back a "bad request" response and closes the connection:

<response>
    <exception>bad request</exception>
</response>

stores the address and the one-time password generated for it
sends the response to the client encoded in B64:

<response>
   <address>197.12.1.1:58824</address>
   <subjectID>JEM_NODE</subjectID>
   <operation>
       <name>getPassword</name>
       <result>randomGeneratedPassword</result>
   </operation>
</response>

3 The Client decode the response and produces a new encoded request asking to log in. But this time the password will be crypted with the environment symmetric key that can be access only by authorized nodes:

<request>
   <address>197.12.1.1:58824</address>
   <subjectID>JEM_NODE</subjectID>
    <password>encrypted randomGeneratedPassword received from server</password>
    <operation>
        <name>login</name>
    </operation>
</request>

4 The Server:

Decodes the request
Checks if the request is valid: otherwise sends back a "bad request" response and closes the connection:

<response>
    <exception>bad request</exception>
</response>

Decrypts the password with the symmetric key
Checks if the password is correct. If it's correct, the Client can proceed and become member of the cluster or retrieve the informations asked to the cluster. If it's not correct, the Server sends back the following "login denied" response:

<response>
   <address>197.12.1.1:58824</address>
   <subjectID>JEM_NODE</subjectID>
    <operation>
        <name>login</name>
        <result>denied</result>
    </operation>
</response>

Every time a login phase will not succeed, the log of the Client and the Server will explain what was the problem.

Let see the protocol in action for a client (job submit) that want to join the cluster:

The client sends a BASE64-encoded request telling who he is and asking for a one time password to login:

<request>
    <address>197.12.1.1:58824</address>
    <subjectID>userID</subjectID>
    <operation>
        <name>getPassword</name>
    </operation>
</request>