RBatch: Batch Script Framework

About RBatch

RBatch is a ruby-based framework for batch scripts. RBatch help to make batch scripts such as "data backup" or "proccess controll ".

There are following functions:

Auto Logging
Auto Library Loading
Auto Mail Sending
Auto Config Reading
External Command Wrapper
Double Run Check

Note: RBatch works on Ruby 1.9 above.

Note: This software is released under the MIT License, see LICENSE.txt.

Quick Start

$ sudo gem install rbatch
$ rbatch-init    # => make directories and sample scripts
$ ruby bin/hello_world.rb
$ cat log/YYYYMMDD_HHMMSS_hello_world.log

Overview

RBatch home directory

If you set the ${RB_HOME} as environment variable, RBatch home directory is fixed at ${RB_HOME}. Otherwise, ${RB_HOME} is the parent directory of the directory where the script is located. In other words, default of ${RB_HOME} is (script path)/../ .

Directory Structure and File Naming Convention

RBatch is made based on the rule of "convention over configuration". So there are conventions of file naming and directory structure. If you follow these rules, you will gain plenty development environment without any configurations.

The rules are below:

Script should be ${RB_HOME}/bin/hoge.rb
Config file should be ${RB_HOME}/conf/hoge.yaml
Common config file should be ${RB_HOME}/conf/common.yaml
Libraries should be ${RB_HOME}/lib/*.rb
Log files should be ${RB_HOME}/log/YYYYMMDD_HHMMSS_hoge.log

For example

${RB_HOME}         <--- RBatch home
 |
 |- .rbatchrc      <--- RBatch Run-Conf
 |
 |- bin            <--- Scripts
 |   |-  A.rb
 |   |-  B.rb
 |
 |- conf           <--- Config files
 |   |-  A.yaml    <--- Individual config file
 |   |-  B.yaml
 |   |-  common.yaml <--- Common config file
 |
 |- log            <--- Log files
 |   |-  YYYYMMDD_HHMMSS_A.log
 |   |-  YYYYMMDD_HHMMSS_B.log
 |
 |- lib            <--- Libraries
     |-  lib_X.rb
     |-  lib_Y.rb

Auto Logging

By using the auto logging block ,RBatch::Log, RBatch automatically output logfiles. The default location of log files are ${RB_HOME}/log/YYYYMMDD_HHMMSS_(script base).log. If exceptions are raised in auto logging block, RBatch will rescue it and output its stack trace to the logfile.

Sample

Make script ${RB_HOME}/bin/sample1.rb

require 'rbatch'
RBatch::Log.new(){ |log|  # Logging block
  log.info "info string"
  log.error "error string"
  raise "exception"
}

Run script. Log file is ${RB_HOME}/log/20121020_005953_sample1.log

# Logfile created on 2012-10-20 00:59:53 +0900 by logger.rb/25413
[2012-10-20 00:59:53 +900] [INFO ] info string
[2012-10-20 00:59:53 +900] [ERROR] error string
[2012-10-20 00:59:53 +900] [FATAL] Caught exception; existing 1
[2012-10-20 00:59:53 +900] [FATAL] exception (RuntimeError)
    [backtrace] test.rb:6:in `block in <main>'
    [backtrace]

/usr/local/lib/ruby192/lib/ruby/gems/1.9.1/gems/rbatch-1.0.0/lib/rbatch/auto_logger.rb:37:in initialize' [backtrace] test.rb:3:in new' [backtrace] test.rb:3:in `

'

Auto Library Loading

If you make libraries at ${RB_HOME}/lib/*.rb, these are loaded(required) before scripts run.

Auto Mail Sending

By using log_send_mail option, RBatch sends error messages by e-mail when ERROR or FATAL level logs are raised.

Auto Config Reading

By putting the configuration file as ${RB_HOME}/conf/"(script base).yaml", the file is read and parsed automatically.

Sample

Make the config file ${RB_HOME}/conf/sample2.yaml.

key: value
array:
 - item1
 - item2
 - item3

In script named ${RB_HOME}/bin/sample2.rb, this config values can be used.

require 'rbatch'

# You can read config value without loading file.
p RBatch.config["key"]   # => "value"
p RBatch.config["array"] # => ["item1", "item2", "item3"]

# If key does not exist, raise exception
p RBatch.config["not_exist"] # => Raise RBatch::ConfigException

If you want to describe yaml by using erb, rename config file from xxx.yaml into xxx.yaml.erb. RBatch evaluates config file as erb after that read it as yaml. When both xxx.yaml and xxx.yaml.erb exist, xxx.yaml is read.

Common Config

By putting shard config file at ${RB_HOME}/conf/common.yaml, the values in the file are shared by all scripts. If you want to change the name of the config file, you cau use the common_conf_name option. It accepts erb format like above normal config.

External Command Wrapper

RBatch have useful functions of external command execution. RBatch.cmd is wrapper of Kernel.#Spawn.

This function return a result object which contains command's string, STDOUT, STDERR and exit status.

Sample

require 'rbatch'

r = RBatch.cmd("ls")
p r.stdout # => "fileA\nfileB\n"
p r.stderr # => ""
p r.status # => 0

If you want to set a timeout of external command, you can use cmd_timeout option.

By using cmd_raise option, if exit status is not 0, RBatch raise an Exception. This function help to handle the errors of external command.

Double Run Check

By using forbid_double_run option, two scripts both has same name cannot run at the same time.

Manual

Manual -> Document (YardDoc)

Sample

AWS EC2 Volume Backup Script

First you make the configuration file.

Config File : ${RB_HOME}/conf/ec2_create_snapshot.yaml

access_key : AKIAITHEXXXXXXXXX
secret_key : JoqJSdP8+tpdFYWljVbG0+XXXXXXXXXXXXXXX

Next, you write the script.

Script : ${RB_HOME}/bin/ec2_create_snapshot.rb

require 'rbatch'  # <= require rbatch
require 'aws-sdk'
require 'net/http'

RBatch::Log.new do |log| # <= Start Log block. And write main logic in
this block.
  # get ec2 region
  @ec2_region = "ec2." +
    Net::HTTP.get("169.254.169.254",
"/latest/meta-data/placement/availability-zone").chop +
    ".amazonaws.com"
  log.info("ec2 region : #{@ec2_region}")  # <= Output Log

  #create ec2 instance
  @ec2 = AWS::EC2.new(:access_key_id     =>
RBatch.config["access_key"],  # <= Read Config
                      :secret_access_key => RBatch.config["secret_key"],
                      :ec2_endpoint      => @ec2_region)


  # create instance
  @instance_id = Net::HTTP.get("169.254.169.254",
"/latest/meta-data/instance-id")
  @instance = @ec2.instances[@instance_id]
  log.info("instance_id : #{@instance_id}")

  # create snapshots
  @instance.block_devices.each do | dev |
    desc = @instance_id + " " + dev[:device_name] + " " +
      dev[:ebs][:volume_id] + " " +Time.now.strftime("%Y/%m/%d %H:%M").to_s
    log.info("create snapshot : #{desc}")
    @ec2.volumes[dev[:ebs][:volume_id]].create_snapshot(desc)
    log.info("sucess")
  end
end

Finally, you run the script, then following logfile is output.

Log file : ${RB_HOME}/log/20140121_123124_ec2_create_snapshot.log

[2014-01-21 12:31:24 -0500] [INFO ] === START RBatch === (PID=10095)
[2014-01-21 12:31:24 -0500] [INFO ] RB_HOME : "/opt/MyProject"
[2014-01-21 12:31:24 -0500] [INFO ] Load Run-Conf: "/opt/MyProject/.rbatchrc"
[2014-01-21 12:31:24 -0500] [INFO ] Load Config  :
"/opt/MyProject/conf/ec2_create_snapshot.yaml"
[2014-01-21 12:31:24 -0500] [INFO ] Start Script :
"/opt/MyProject/bin/ec2_create_snapshot.rb"
[2014-01-21 12:31:24 -0500] [INFO ] Logging Start:
"/opt/MyProject/log/20140121_123124_ec2_create_snapshot.log"
[2014-01-21 12:31:24 -0500] [INFO ] ec2 region :
ec2.ap-northeast-1.amazonaws.com
[2014-01-21 12:31:25 -0500] [INFO ] instance_id : i-cc25f1c9
[2014-01-21 12:31:25 -0500] [INFO ] create snapshot : i-cc25f1c9
/dev/sda1 vol-82483ea7 2014/01/21 12:31
[2014-01-21 12:31:25 -0500] [INFO ] sucess

Only you write a short code, this batch script has logging and reading config function.

Customize

If you want to customize RBatch, you have two methods.

(1) Write Run-Conf(${RB_HOME}/.rbatchrc).
(2) Pass an option object to constructor of RBatch classes in a script.

When an option is set in both (1) and (2), (2) is prior to (1).

Customize by writing Run-Conf (.rbatchrc)

Sample of RBatch Run-Conf ${RB_HOME}/.rbatchrc.

# RBatch Run-Conf (.rbatchrc)
#
#   This format is YAML.
#

# -------------------
# Global setting
# -------------------

# Conf Directory
#
#   Default is "<home>/conf"
#   <home> is replaced to ${RB_HOME}
#
#conf_dir : <home>/config/
#conf_dir : /etc/rbatch/

# Common Config file name
#
#   Default is "common.yaml"
#
#common_conf_name : share.yaml

# Library Directory
#
#   Default is "<home>/lib"
#   <home> is replaced to ${RB_HOME}
#
#lib_dir: /usr/local/lib/rbatch/

# Auto Library Load
#
#   Default is true
#   If true, require "(library directory)/*.rb" before script run.
#
#auto_lib_load : true
#auto_lib_load : false

# Forbit Script Running Doubly
#
#   Default is false.
#   If true, two same name scripts cannot run at the same time.
#
#forbid_double_run: true
#forbid_double_run: false

# RBatch Journal Level
#
#   Default is 1
#   RBatch Journal is message of RBatch and is output to STDOUT.
#   If 2, put much more information.
#   If 0, put nothing.
#
#   Example of RBatch Journal are follows.
#     [RBatch] === START RBatch === (PID=5795)
#     [RBatch] RB_HOME : "/path/to/"
#     [RBatch] Load Run-Conf: "/path/to/.rbatchrc"
#     [RBatch] Start Script : "/path/to/bin/hello.rb"
#     ....
#
#rbatch_journal_level : 2
#rbatch_journal_level : 0

# Mix RBatch Journal to Logs
#
#   Default is true.
#   If true, RBatch Journal is output not only STDOUT
#   but also log file(s) which is(are) opened at time.
#
#mix_rbatch_journal_to_logs : true
#mix_rbatch_journal_to_logs : false

# -------------------
# Log setting
# -------------------

# Log Directory
#
#   Default is "<home>/log"
#   <home> is replaced to ${RB_HOME}
#
#log_dir : <home>/rb_log
#log_dir : /var/log/rbatch/

# Log File Name
#
#   Default is "<date>_<time>_<prog>.log".
#   <data> is replaced to YYYYMMDD date string
#   <time> is replaced to HHMMSS time string
#   <prog> is replaced to Program file base name (except extention).
#   <host> is replaced to Hostname.
#
#log_name : "<date>_<time>_<prog>.log"
#log_name : "<date>_<prog>.log"

# Append Log
#
#   Default is true.
#
#log_append : true
#log_append : false

# Log Level
#
#   Default is "info".
#   Effective values are "debug","info","wran","error",and "fatal".
#
#log_level : "debug"
#log_level : "warn"

# Print log string both log file and STDOUT
#
#   Default is false.
#
#log_stdout : true
#log_stdout : false

# Delete old log files
#
#   Default is false.
#   If this is true, delete old log files when RBatch::Log.new is called.
#   If log filename format does not include "<date>", do nothing.
#
#log_delete_old_log : true
#log_delete_old_log : false

# Expire Days of Log Files
#
#   Default is 7.
#
#log_delete_old_log_date : 14

# Log buffering
#
#   Default is false.
#   If true, log output is bufferd.
#
#log_bufferd : true
#log_bufferd : false

# Output Exit Status
#
#   Default is true.
#   When you use the "exit" method in a log block,
#   output exit status into the log file.
#
#log_output_exit_status : true
#log_output_exit_status : false

# Send Mail
#
#   Default is false.
#   When log.error is called, log.fatal is called,
#   or rescue an Exception, send e-mail.
#
#log_send_mail : true

# Mail Parameters
#
#log_mail_to   : "xxx@sample.com"
#log_mail_from : "xxx@sample.com"
#log_mail_server_host : "localhost"
#log_mail_server_port : 25
#
# if you want to send multiple recipients, use array.
#
#log_mail_to :
#  - "AAA@sample.com"
#  - "BBB@sample.com"

# -------------------
# Cmd setting
# -------------------

# Raise Exception
#
#   Default is false.
#   If true, when command exit status is not 0, raise exception.
#
#cmd_raise : true
#cmd_raise : false

# Command Timeout
#
#   Default is 0 [sec] (=no timeout).
#
#cmd_timeout : 5

Customize by passing option object to constructor

If you want to change options in a script, you pass an options object to the constructor of RBatch::Log or RBatch::Cmd.

option of RBatch::Log

Sumple

opt = {
      :name      => "<date>_<time>_<prog>.log",
      :dir       => "/var/log",
      :append    => true,
      :level     => "info",
      :stdout    => false,
      :delete_old_log => false,
      :delete_old_log_date => 7,
      :bufferd => false,
      :output_exit_status => true,
      :send_mail => false,
      :mail_to   => nil,
      :mail_from => "rbatch.localhost",
      :mail_server_host => "localhost",
      :mail_server_port => 25
      }
RBatch::Log.new(opt)

option of RBatch::Cmd

Sample

opt = {
      :raise     => false,
      :timeout   => 0
      }
RBatch::Cmd.new("ls -l", opt).run

Migration from version 1 to version 2

Move ${RB_HOME}/conf/rbatch.yaml to ${RB_HOME}/.rbatchrc .

Name		Name	Last commit message	Last commit date
Latest commit History 282 Commits
bin		bin
lib		lib
sample		sample
spec		spec
.gitignore		.gitignore
.travis.yml		.travis.yml
CHANGELOG		CHANGELOG
Gemfile		Gemfile
LICENSE.txt		LICENSE.txt
README.ja.md		README.ja.md
README.md		README.md
Rakefile		Rakefile
rbatch.gemspec		rbatch.gemspec

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Repository files navigation

RBatch: Batch Script Framework

About RBatch

Quick Start

Overview

RBatch home directory

Directory Structure and File Naming Convention

Auto Logging

Auto Library Loading

Auto Mail Sending

Auto Config Reading

Common Config

External Command Wrapper

Double Run Check

Manual

Sample

AWS EC2 Volume Backup Script

Customize

Customize by writing Run-Conf (.rbatchrc)

Customize by passing option object to constructor

option of RBatch::Log

option of RBatch::Cmd

Migration from version 1 to version 2

About

Releases

Packages

Languages

License

naokikurosawa/rbatch

Folders and files

Latest commit

History

Repository files navigation

RBatch: Batch Script Framework

About RBatch

Quick Start

Overview

RBatch home directory

Directory Structure and File Naming Convention

Auto Logging

Auto Library Loading

Auto Mail Sending

Auto Config Reading

Common Config

External Command Wrapper

Double Run Check

Manual

Sample

AWS EC2 Volume Backup Script

Customize

Customize by writing Run-Conf (.rbatchrc)

Customize by passing option object to constructor

option of RBatch::Log

option of RBatch::Cmd

Migration from version 1 to version 2

About

Resources

License

Stars

Watchers

Forks

Releases

Packages 0

Languages

Packages