Magpie 1.76

• Add new "spark-with-zeppelin" submission scripts.
• Add support for Zeppelin 0.6.0.
  • Update default Zeppelin to 0.6.0.
  • Make default Zeppelin work with Spark 1.6.2-bin-hadoop2.6.
• Add support for Phoenix 4.8.0-Hbase-1.0, 4.8.0-Hbase-1.1, 4.8.0-Hbase-1.2.
  • Update default Phoenix to 4.8.0-Hbase-1.2.
• Add support for Storm 1.0.2.
  • Update default Storm to 1.0.2.
• Add older support for
  • Hbase 1.0.3 and 1.1.5.
  • Phoenix 4.5.0-Hbase-1.0, 4.5.0-Hbase-1.1, 4.5.1-Hbase-1.0, 4.5.2-Hbase-1.0, 4.6.0-Hbase-1.0, 4.7.0-Hbase-1.0
  • Spark 1.1.0-bin-hadoop2.3, 1.1.0-bin-hadoop2.4, 1.1.1-bin-hadoop2.3, 1.1.1-bin-hadoop2.4, 1.2.0-bin-hadoop2.3, 1.2.1-bin-hadoop2.3, 1.2.2-bin-hadoop2.3, 1.3.0-bin-hadoop2.3, 1.3.1-bin-hadoop2.3, 1.3.1-bin-hadoop2.6, 1.4.0-bin-hadoop2.3, 1.4.0-bin-hadoop2.4, 1.4.1-bin-hadoop2.3, 1.4.1-bin-hadoop2.4.
  • Hbase 0.98.0-hadoop2 0.98.1-hadoop2 0.98.2-hadoop2 0.98.4-hadoop2, 0.98.5-hadoop2 0.98.6-hadoop2 0.98.6.1-hadoop2 0.98.7-hadoop2, 0.98.8-hadoop2 0.98.10-hadoop2 0.98.10.1-hadoop2 0.98.11-hadoop2, 0.98.12-hadoop2 0.98.12.1-hadoop2 0.98.13-hadoop2 0.98.14-hadoop2, 0.98.15-hadoop2 0.98.16-hadoop2 0.98.16.1-hadoop2 0.98.17-hadoop2, 0.98.18-hadoop2 0.98.19-hadoop2 0.98.20-hadoop2, 0.98.5-hadoop2 0.98.6-hadoop2 0.98.6.1-hadoop2 0.98.7-hadoop2, 0.98.8-hadoop2 0.98.10-hadoop2 0.98.10.1-hadoop2 0.98.11-hadoop2, 0.98.12-hadoop2 0.98.12.1-hadoop2 0.98.13-hadoop2 0.98.14-hadoop2, 0.98.15-hadoop2 0.98.16-hadoop2 0.98.16.1-hadoop2 0.98.17-hadoop2, 0.98.18-hadoop2 0.98.19-hadoop2 0.98.20-hadoop2

Magpie 1.75

• Remove 'classifywikipedia' example with Mahout.
• Support Spark 2.0.0.
• Update default Spark to 2.0.0 w/ Hadoop 2.7.
• Support new SPARK_YARN_STAGING_DIR option.
• Support Pig 0.16.0.
• Update default Pig to 0.16.0
• Support Hbase 1.2.2.
• Update default Hbase to 1.2.2
• Support Mahout 0.12.2.
• Update default Mahout to 0.12.2

Magpie 1.74

Major refactoring and code cleanup within Magpie. See NEWS file for
details. Changes that may affect users.

• Support 'setuponly' option for PHOENIX_MODE.
• Export ZOOKEEPER_CLIENT_PORT for jobs.
• Export KAFKA_PORT for jobs.
• Remove a number of errant variable exports.
• Remove scripts/job-scripts/zookeeper-ruok-script.sh script
• Hadoop terasort may no longer do the "right thing" under Hadoop 1.0. Users should set HADOOP_TERAGEN_MAP_COUNT and HADOOP_TERASORT_REDUCER_COUNT appropriately.

Magpie 1.73

• Support "args" options to Magpie, Hadoop, Hbase, Spark, Storm script options.
• Support Spark 1.6.2-bin-hadoop2.6.
• Update default Spark to 1.6.2-bin-hadoop2.6.
• Support Storm 0.10.1, 1.0.0, and 1.0.1.
• Update default Storm to 1.0.1.
• When users want to use alternate config files, instead of copying all files from ${MAGPIE_SCRIPTS_HOME}/conf, into a project's alternate CONF_FILES director (e.g. HADOOP_CONF_FILES), users can copy only the files they wish to modify.
• Fix magpie-gather-config-files-and-logs-script.sh to gather Pig, Zeppelin, Phoenix, and Kafka conf/log files.
• Support HADOOP_LOCALSTORE_CLEAR option.
• Support SPARK_LOCAL_SCRATCH_DIR_CLEAR option.
• Support multiple paths for SPARK_LOCAL_SCRATCH_DIR.
• Various minor bug fixes regarding Spark w/ Yarn use.

Magpie 1.72

Support SPARK_USE_YARN option for Spark to default to using Yarn.

Magpie 1.71

• Adjust Yarn yarn.scheduler.maximum-allocation-mb to be same as yarn.nodemanager.resource.memory-mb.
• Support Mahout 0.12.0, 0.12.1
• Update default Mahout to 0.12.1
• Support Phoenix 4.7.0
• Update default Phoenix to 4.7.0.
• Support Hbase 1.2.1
• Update default Hbase to 1.2.1.

Magpie 1.70

• Handle corner cases where job name contains dollar sign or whitespace. Can lead to problems because job name is used internally within Magpie for file paths, etc.
• Support setting MAGPIE_PYTHON, to configure Python paths in projects appropriately. Currently used to set PYSPARK_PYTHON in Spark.
• Support ability to kill scripts early and all Yarn jobs cleanly.
• Exit job if an error occurs during magpie-setup phase.
• Adjust defaults of some projects based on dependency requirements.
  • default of Spark w/ HDFS to Hadoop 2.6.4
  • default of Phoenix w/ Hbase to Hbase 1.1.4

Magpie 1.69

• Output warning as time for interactive and setuponly jobs is running down.
• Support ability to kill scripts and other jobs early and begin teardown early. Similar to ability within interactive mode.
• Support new 'decommissionhdfsnodes' option to HADOOP_MODE.
• Remove convenience job-scripts/hadoop-hdfs-over-lustre-or-hdfs-over-networkfs-nodes-decommission-script.sh and
  hbase-hdfs-over-lustre-or-hdfs-over-networkfs-nodes-decommission-script.sh script which is now obsoleted by 'decommissionhdfsnodes' option.
• Increase default Kafka Zookeeper timeout from 6000 to 30000
• Support Hadoop 2.6.2, 2.6.3, 2.6.4, 2.7.2.
• Update default Hadoop to 2.7.2.
• Support Spark 1.6.1.
• Update default Spark to 1.6.1.
• Support Hbase 1.1.4, 1.2.0.
• Update default Hbase to 1.2.0.
• Support Storm 0.9.6, 0.10.0.
• Update default Storm to 0.10.0.
• Support Mahout 0.11.2
• Update default Mahout to 0.11.2.
• Support Zookeeper 3.4.8.
• Update default Zookeeper to 3.4.8.

Magpie 1.68

Fix use of SPARK_LOCAL_SCRATCH_DIR.
Support Hbase 1.1.3, update default Hbase to 1.1.3.
Support Zeppelin 0.5.6.

Magpie 1.67

Support Kafka, initially with version 2_11-0.9.0.0.
Support HADOOP_TERASORT_OUTPUT_REPLICATION option.
Support HADOOP_TERASORT_RUN_TERAVALIDATE option.
Support HADOOP_TERASORT_RUN_TERACHECKSUM option.
Move most documentation to doc/ subdirectory.