Magpie 1.66

Support Spark 1.6.0, update default Spark to 1.6.0.
Fix bug in Phoenix launch due to mis-setting of ssh.

Magpie 1.65

Support Spark 1.5.2, update default Spark to 1.5.2.
Support Zookeeper 3.4.7, update default Zookeeper to 3.4.7.

Magpie 1.64

Support Phoenix 4.6.0. Update default to 4.6.0.
On initial HDFS setup, create user home directory if it does not exist (/user/$USER).
Support basic support for Mahout (BETA), initial version supported 0.11.0, 0.11.1.

Magpie 1.63

Support easy mechanism to kill jobs running in interactive mode.
Add MAGPIE_REMOTE_CMD_OPTS to run output as needed.
Support Phoenix 4.5.1, 4.5.2
Support Spark 1.5.1, update default Spark to 1.5.1.
Support new convenience environment variables ZOOKEEPER_NODES and ZOOKEEPER_NODES_FIRST.

Magpie 1.62

Support Spark 1.5.0, update default Spark to 1.5.0-bin-hadoop2.6
Support No Local Dir in Hadoop 2.4.0
Support No Local Dir in Spark 1.2.0
Support No Local Dir in Hbase 0.98.3
Support various older Spark versions, 0.9.2, 1.2.1, 1.2.2, 1.3.1, 1.4.0.
Support various older Pig versions, 0.13.0.
Support various older Hadoop versions, 2.3.0, 2.4.1, 2.5.0, 2.5.1, 2.5.2, 2.6.1, 2.7.0.
Support various older Zookeeper versions, 3.4.0, 3.4.1, 3.4.2, 3.4.3, 3.4.4.
Support various older Hbase versions, 0.99.0, 0.99.1, 1.0.0, 1.0.1, 1.0.1.1, 1.0.2, 1.1.0, 1.1.0.1.
Fix corner case in spark word count w/ rawnetworkfs
Fix corner case in in No Local Dir with history server teardown in Hadoop.

Magpie 1.61

Default Java is now 1.7.
Support Hadoop 2.7.1, update default Hadoop to 2.7.1.
Support Spark 1.4.1, update default Spark to 1.4.1-bin-hadoop2.6.
Support Storm 0.9.5, update default Storm to 0.9.5.
Support Pig 0.15.0, update default Pig to 0.15.0.
Support Hbase 0.99.2, Hbase 1.1.1.
Support Hbase 1.1.2, update default Hbase to 1.1.2.
Support ZOOKEEPER_DATA_DIR_CLEAR.
Fix corner case in Storm wordcount test.
Support ZOOKEEPER_DATA_DIR_TYPE for timeout optimization.
Increase Zookeeper timeouts on Hbase & Storm if Zookeeper is on a network file system.
Wait for Hbase Master to come up w/ status checks before running job.
Fix corner case in Hadoop & Hbase shutdown path.

Magpie 1.60

Add support for job submission via LSF+Mpirun.

Support HADOOP_PER_JOB_HDFS_PATH and ZOOKEEPER_PER_JOB_DATA_DIR.

Support HADOOP_HDFS_PATH_CLEAR

Adjust MAGPIE_SUBMISSION_TYPE names given new script directories in 1.59

• slurmsbatch -> sbatchsrun
• msubslurm -> msubslurmsrun
• msubtorque -> msubtorquepdsh
• Backwards compatibility with older names is still supported.

Support the ability to run Magpie in environments without /tmp or a
very small amount of space in /tmp.

• Support MAGPIE_NO_LOCAL_DIR option. See README.no-local-dir for details.
• Remove ZOOKEEPER_FILESYSTEM_MODE setting.
  • Add node rank into all ZOOKEEPER_DATA_DIR paths, ensuring it is unique
    regardless if it is a network or local fs path.
  • This may potentially break backwards compatability with Zookeeper data that may already exist on network file systems
• Remove SPARK_LOCAL_SCRATCH_DIR_TYPE setting.
  • Add node rank into all scratch directory paths, ensuring it is
    unique regardless if it is a network or local fs path.
• By default, set java.io.tmpdir to a local scratch space in all projects
• By default, set java.io.tmpdir in PIG_OPTS when running Pig.
• Disable Java hsperfdata if MAGPIE_NO_LOCAL_DIR set.

Magpie 1.59

Rename submission script directories to include launching mechansim

• script-msub-slurm/ -> script-msub-slurm-srun/
• script-msub-torque/ -> script-msub-torque-pdsh/
• script-sbatch/ -> script-sbatch-srun/

Various documentation fixes
Few minor bug fixes

Magpie 1.58

Default teragen to use reasonable number of map tasks given blocksize.
Support HADOOP_TERAGEN_MAP_COUNT terasort configuration.
Support enabling thrift server with Hbase.
Support the ability to add localized configuration, see README for details.

Magpie 1.57

• Use --time instead of SBATCH_TIMELIMIT in sbatch scripts
• Fix corner case in MAGPIE_TIMELIMIT_MINUTES calculation if job < 1 hour
• When resource manager is slurm, calculate time left in job dynamically instead of using fixed MAGPIE_STARTUP_TIME
• In Hadoop terasort, check for old directories before removing them
• Add various error checks for HDFS over Lustre/NetworkFS, including varying node counts and mounting with different Hadoop versions.
• Fix up setup/cleanup/teardown code to fix various corner cases
• Clean up setup code into a new script, reducing number of calls to srun, etc.
• Fix msub-torque submission scripts to be able to use Tachyon.
• Various README/documentation updates/upgrades.
• Various other minor fixes