JCLReference

What's JCL?

Job Control Language (JCL) is a scripting language to instruct the system on how to run a batch job. The modern JCL are XML slang, as ANT or SpringBatch that we'll analyze on next chapters. In JCL the unit of work is the job. A job consists of one or several steps, each of which is a request to run one specific program. JCL is the way how a job can be configured and can declare all needed resources to use. JCL declares some run-time attributes as well, necessary to execute correctly the job.

JCL JEM Properties

JEM is able to manage different JCL by a JEM factory but in all JCL JEM will try to read some important (and in some cases mandatory) information for job execution.

These information are usually defined by properties in the JCL. There is not any length restriction and as following it's allowed definition of characters which could be used:

'"' ([^<&"] | Reference)* '"' | "'" ([^<&'] | Reference)* "'"

So, you can't have:

• the same character that opens/closes the attribute value (either ' or " )
• a naked ampersand (& must be &)
• a left angle bracket (< must be <)

You should also not being using any characters that are outright not legal anywhere in an XML document. All information, that JEM reads, are following:

• Job Name is mandatory string property which is used to assign the name to the job.
• Environment is mandatory string property with the name of cluster which the job must be executed in.
• Domain is optional string property that represents a subset of nodes (by a tag) which job must be executed in. Default is ***. When a job has got the default value can be executed in whatever inactive node.
• Affinity is optional string property that represents a subset of nodes of a domain (by one or more tags, comma separated) which job must be executed in. Default is ***. When a job has got the default value can be executed in whatever inactive node.
• User is optional string property that represents the user to use to execute the job. Default is null. Usually the job must use the user who submitted it during the execution. This is used for authorization phases to access to resources. Indicating a different user in JCL, you can override the submitter's user and to have different permissions. To use this feature you must be authorize to the surrogate permission.
• Locking Scope is optional string property what is used to decide at which level the job must request the locks for defined resources (data description and/or lock). There are 3 possible scopes:
  • job: (default)JEM must collect all the declared resources definitions and try to lock all them. The lock is maintained till the end of job. This kind of locking scope blocks concurrent jobs, reducing the parallel executions when jobs ask for same resources.
  • step: JEM locks the resources at the beginning of step and releases them at the end of each step. This scope increases jobs concurrency but be careful with the job business logic, because not always is acceptable that another job uses the resources of another during either executions.
  • task: JEM locks the resources at the beginning of each task (if JCL implements task concept) and releases them at the end of task. This scope increases jobs concurrency but be careful with the job business logic, because not always is acceptable that another job uses the resources of another during either executions.
• Hold is optional boolean property which is used to maintain the job in the input queue and wait a release command. Default is false. This is helpful when you don't want to execute the job immediately but you want to decided when this job can be manually executed.
• Priority is optional integer property which is used to assign a priority in input queue to job. Default is 10. This property is used by nodes to decide which job must be executed. The logic is check the job with highest priority. With same priorities, node chooses the oldest one, using the submitted timestamp.
• Memory is optional integer property which is used to assign the right amount of memory, in megabytes (MB). Default is 128. Every node of JEM knows how much memory can use and decides to execute the job if the amount of memory indicated is less than available one for node.
• Classpath is optional string property which is used to assign a classpath before starting job. All different paths must be semicolon (;) separated. Default is null. All paths will be added at the end of default classpath, created by JEM. This property becomes mandatory when the job must execute business logic, contained in jars not in JEM classpath. It's helpful for JCL extensions (by java implementation) as well.
• Prior Classpath is optional string property which is used to assign the right classpath before starting job. All different paths must be semicolon (;) separated. Default is null. All paths will be added at the beginning of default classpath, created by JEM. This property becomes mandatory when the job must execute business logic, contained in jars not in JEM classpath. It's helpful for JCL extensions (by java implementation) as well. PAY ATTENTION to use it because if some libraries are added at the beginning some features of JEM used at runtime in the job couldn't work.
• Emails Notification is optional string property which is used to assign a list of email addresses to notify when a job ends. All different addresses must be semicolon (;) separated. Default is null. A Job life-cycle listener is implemented out-of-the-box and when the ended job contains this property, a formatted mail is sent to defined mail addresses.
• Java is optional string property which is used to indicate which java runtime (the tag name( must be used to execute the job. See JEM node configuration) to understand how to configure it. If missing, JEM will use the JRE used by JEM node.

Datasets and Data descriptions

One of most important goal of a job is create, read and write files, usually named datasets. A dataset represents a file that a task must use during the job execution to implement its business logic. There is not any length name restriction and as following it's allowed definition of characters which could be used:

'"' ([^<&"] | Reference)* '"' | "'" ([^<&'] | Reference)* "'"

So, you can't have:

• the same character that opens/closes the attribute value (either ' or " )
• a naked ampersand (& must be &)
• a left angle bracket (< must be <)

You should also not being using any characters that are outright not legal anywhere in an XML document.

There are 4 kinds of datasets, as following:

• File is a normal file, can be indicated by complete absolute or relative path (even using variables to compose it). The name can not start with *. or @@
  • Example: My/data/info.txt
• GDG is version file which is addressable by a relative position starting from last version, can be indicated by complete absolute or relative path (even using variables to compose it). This kind of dataset is helpful when you create and use the same relative version every periodical execution. The name can not start with *. or @@
  • Example: My/data/info(0) or My/data/info(+1) o My/data/info(-1)
• Reference is a link to dataset previously defined in other tasks. With this kind of dataset you don't have to define the same dataset twice or more. Using GDGs and different locking scope, you are sure to use the previous file, at the same version. The name must start with *..
  • Example : *.step1.dd1
• Temporary is a file created in temporary directory and deleted on ended of job. The temporary files are used to park some data for further operations in next steps of the same job. The name must start with @@ and will be use as prefix of temporary file name.
  • Example: @@mytemp
• Inline are data put directly inside JCL, usually to configure the program executed in task or step.
• Data Source is a link to data source previously defined in same task or step. Currently the unique data sources, usable as dataset, are FTP resources. The file name can not start with *. or @@

The data description is a container of datasets (one or more), addressable by name, both a java code(JNDI) and executable command (by environment variables). It could be defined without datasets when job wants to writes data on a system output stream (called sysout). There is not any length name restriction and as following it's allowed definition of characters which could be used:

'"' ([^<&"] | Reference)* '"' | "'" ([^<&'] | Reference)* "'"

So, you can't have:

• the same character that opens/closes the attribute value (either ' or " )
• a naked ampersand (& must be &)
• a left angle bracket (< must be <)

You should also not being using any characters that are outright not legal anywhere in an XML document.

There are 3 kinds of data description.

• Sysout is a system output stream, used by job to write information without creating any common dataset. A sysout is physically saved in output folder. This is very useful during test phase to check the result of business logic, seeing what the program writes , before creating any dataset.
• Single dataset is a data description which contains only a single file, used by a job. The type of access (read or write) is defined by disposition attribute
• Multiple datasets are a data description which contains more than one dataset , used by a job. The type of access must be only in read because is not possible to read more than one file on the same output stream.

Inside the data description, you must define the disposition that means how you want to access to data (read or write). This attribute is really important to lock the resources in right way, compliant with business logic.

The possible dispositions are the following:

• NEW: declares that the job is asking for a new dataset. If exists, an exception will be throw. This is also mandatory for sysout data description.
• OLD: declares that the job is asking for an existing dataset in write mode, overriding the previous content. If dataset doesn't exists, an exception will be throw.
• MOD: declares that the job is asking for an existing dataset in write mode, in append mode to previous content. If dataset doesn't exists, an exception will be throw.
• SHR: declares that the job is asking for an existing dataset in read mode. If dataset doesn't exists, an exception will be throw.

Data Sources

Data sources are common resources defined centrally and addressable by JCL from any job (if the user of the job is authorized to use them). All resources must have a name, the key used by job to have it. There is not any length name restriction and as following it's allowed definition of characters which could be used:

'"' ([^<&"] | Reference)* '"' | "'" ([^<&'] | Reference)* "'"

So, you can't have:

• the same character that opens/closes the attribute value (either ' or " )
• a naked ampersand (& must be &)
• a left angle bracket (< must be <)

You should also not being using any characters that are outright not legal anywhere in an XML document.

Currently this implementation works only for JAVA main programs, because data sources are provided by JNDI (Java Naming and Directory Interface). The great benefit of data sources is you could access to a database or ftp server using logical name, without caring about authentication stuff (as user-id or password) because JEM will prepare a connection already ready to use.

All data sources are stored inside a Hazelcast in a specific map, called common-resources. The map is configured to be persistent. All resources can be defined by a specific component of web application (if authorized) or by batch Apache Ant utilities (only for Enterprise Edition).

Locks

Locks are virtual resources that you could use when you need a traffic light inside of whole JEM cluster. With a lock, you can serialize the executions of jobs on several nodes by a name. There is not any length name restriction and as following it's allowed definition of characters which could be used:

'"' ([^<&"] | Reference)* '"' | "'" ([^<&'] | Reference)* "'"

So, you can't have:

• the same character that opens/closes the attribute value (either ' or " )
• a naked ampersand (& must be &)
• a left angle bracket (< must be <)

You should also not being using any characters that are outright not legal anywhere in an XML document.

A lock generates a global lock by name in exclusive mode, so when more jobs use the same lock will be serialized, respecting the locking scope.