Merge 3.x (Cassandra 2.x) branch to master

.travis.yml

@@ -1,6 +1,6 @@
 language: java
 jdk:
-  - oraclejdk8
+- oraclejdk8
 sudo: false
 install: ./installViaTravis.sh
 script: ./buildViaTravis.sh

CHANGELOG.md

@@ -1,3 +1,108 @@
+# Changelog
+## 2018/08/20 3.1.60
+***WARNING*** THIS IS A BREAKING RELEASE 
+### New Feature
+* (#706) Move to spectator to collect Metrics. This is a breaking change, if you were depending on MetricPublisher to collect metrics. 
+
+## 2018/08/13 3.1.59 
+### New Feature
+* (#671) Compactions on CRON: Added ability to schedule compactions for columnfamilies on CRON. 
+* (#691) Snapshot Meta Service: This will form the basis of backup version 2. This service will run on CRON and will ensure we upload a meta file containing the list of SSTables on disk. 
+* (#703) Making post restore hook heartbeat timeout and heartbeat check frequency configurable
+
+## 2018/07/28: 3.1.58
+* (#695) Moving restore-finished status update, to after PostRestoreHook execution.
+
+## 2018/07/11: 3.1.57
+* (#689) Adding partitioner endpoint to cassadmin resource to get C* partitioner name.
+
+## 2018/06/28: 3.1.54
+### Improvements
+* (#683) PostRestoreHook logging improvements.
+
+
+## 2018/06/27: 3.1.53
+### New Feature
+* (#681) PostRestoreHook changes. Adding ability to call a post restore hook once files get downloaded as part of the restore process.
+
+Following are relevant configurations around post restore hook
+- priam.postrestorehook.enabled: indicates if postrestorehook is enabled.
+- priam.postrestorehook: contains the command with arguments to be executed as part of postrestorehook. Priam would wait for completion of this hook before proceeding to starting C*.
+- priam.postrestorehook.heartbeat.filename: heartbeat file that postrestorehook emits. Priam keeps a tab on this file to make sure postrestorehook is making progress. Otherwise, a new process of postrestorehook would be spawned (upon killing existing process if still exists)
+- priam.postrestorehook.done.filename:'done' file that postrestorehook creates upon completion of execution.
+- priam.postrestorehook.timeout.in.days:maximum time that Priam should wait before killing the postrestorehook process (if not already complete)
+
+## 2018/06/12: 3.1.52
+### Bug Fixes
+* (#679) Mark snapshot as a failure if there is an issue with uploading a file. This is to ensure we fail-fast. This is in contrast to previous behavior where snapshot would "ignore" any failures in the upload of a file and mark snapshot as "success". 
+         Since it was not truly a "success" marking that as "failure" is the right thing to do. Also, meta.json should really be uploaded in case of "success" and not in case of "failure" as the presence of "meta.json" marks the backup as successful.
+         The case for fail-fast: In a scenario where we had an issue say at the start of the backup, it makes more sense to fail-fast then to keep uploading other files (and waste bandwidth and use backup resources). The remediation step for backup failure is anyways to take a full snapshot again.
+
+## 2018/06/07: 3.1.51
+### Bug Fixes
+* (#678) Change the default location of backup status and downloaded meta.json as part of backup verification
+
+## 2018/03/07: 3.1.50
+
+### New Features
+* (#664) Cassandra Process Manager can be configured to gracefully stop using the new 
+ `gracefulDrainHealthWaitSeconds` option. If this option set to a positive integer (>=0) then before calling 
+ the shutdown script, Priam will fail healthchecks (`InstanceState.isHealthy`) for the configured number of seconds and  then will issue a `nodetool drain` with 30s timeout (since drain can hang), and finally call the provided stop script. By default this is set to `-1` to disable this feature for backwards compatibility. This is useful if you want to gracefully drain cassandra clients off a node before running `drain` (which kills the Native/Thrift server and resets and tcp connections that were established; in flight requests can get   dropped), then running drain to safely stop Cassandra, and then call your stop script. If your service discovery system does not integrate with Priam's health system or your stop script already does all these things then leave this functionality disabled.
+* (#664) `/v1/cassadmin/stop` http API call now takes an optional `force` parameter (e.g. `/v1/cassadmin/stop?force=true` which will skip the graceful path for that particular stop; default value is `false`.
+* (#650) Enable auth on the jmx port via `jmxUsername` and `jmxPassword` options. By default these are null and not provided.
+
+### Bug Fixes
+* (#659) Fix to `Snapshotstatus` to actually contain `bkupMetadata`
+* (#661) Update `commons-io`, `aws-java-sdk`, `snakeyaml`
+
+### Breaking changes
+* (#664) If you previously implemented `ICassandraProcess` internally the `start` method has been refactored to take a `boolean force` parameter. If you implement this interface you can supply `false` to preserve previous behavior. 
+
+## 2018/02/01: 3.1.48
+
+### Bugs
+* Autostart functionality now uses timers instead of ratelimiters so that
+  the first autostart does not start until an interval after the first start.
+
+## 2018/01/11: 3.1.47
+
+### New Features
+* Cassandra Process Manager and Monitor now record metrics when C* is stopped, started or auto-started with recent autorestart functionality. 
+* Location of backup status file is now configurable via configuration `priam.backup.status.location`.
+* SDBInstance for token management with default binding to us-east-1 but configurable via `priam.sdb.instanceIdentity.region`.
+
+### Bugs
+* Exclude duplicate sl4j module binding.
+* Shut down quartz at application stop
+
+
+## 2017/12/14: 3.1.46
+
+### Bugs
+* Gradle 4.4 Support
+* Autostart functionality now only sets shouldCassandraBeAlive flag from
+  the start api to prevent a race against the stop API in the monitoring
+  thread.
+
+## 2017/12/06: 3.1.45
+
+### Bugs
+* None
+
+### New Features
+* Priam will now automatically restart Cassandra if it fails. If you use
+  Priam to stop Cassandra (via the API) it will not automatically restart
+  Cassandra until a subsequent start via the API. You can control this
+  via the ``priam.remediate.dead.cassandra.rate`` configuration option. If
+  negative it disables auto-remediation, if zero it immediately auto-remediates
+  on any failure, and if a positive integer the auto-remediation waits for
+  that number of seconds between restarts. The default is 360 seconds
+  (one hour).
+
+### Breaking Changes
+* None
+
+## Previous changelog
 1.1
 - Support for cassandra 1.1
 - Support to publish cassandra metrics (TODO)
@@ -8,4 +113,4 @@
 - Multiple bug fixes
 
 0.0.5
-- Initial external release
+- Initial external release

README.md

@@ -15,16 +15,16 @@ Priam is a process/tool that runs alongside Apache Cassandra to automate the fol
 
 Apache Cassandra is a highly available, column oriented database: http://cassandra.apache.org.
 
-The name 'Priam' refers to the King of Troy in Greek mythology, who was the father of Cassandra. 
+The name 'Priam' refers to the King of Troy in Greek mythology, who was the father of Cassandra.
 
-Priam is actively developed and used at Netflix. 
+Priam is actively developed and used at Netflix.
 
 Features:
 - Token management using SimpleDB
 - Support multi-region Cassandra deployment in AWS via public IP.
 - Automated security group update in multi-region environment.
 - Backup SSTables from local ephemeral disks to S3.
-- Uses Snappy compression to compress backup data on the fly. 
+- Uses Snappy compression to compress backup data on the fly.
 - Backup throttling
 - Pluggable modules for future enhancements (support for multiple data storage).
 - APIs to list and restore backup data.
@@ -37,9 +37,7 @@ Priam versions and their compatibility with Cassandra versions in explained in m
 
 |Priam Branch (Release) |Cassandra Version |Description |
 | :------- | :---- | :--- |
-|3.11| C* 3.11 | Currently it supports Apache Cassandra 3.x |
-|3.x| C* 2.1.x | Any minor version of C* 2.1.x|
-|2.x| C* 2.0.x| Any minor version of C* 2.0.x is supported. No longer under active development|
+|master |C* 2.x | Any version of Apache Cassandra 2.x |
+|3.11| C* 3.11 | Currently it supports Apache Cassandra 3.x. |
 |1.2.x | C* 1.2 only | Any minor version of C* 1.2.x is supported. No longer under active development |
 |1.1| C* 1.1 only| No longer under active devleopment|
-|master |C* 1.2 - 2.0| Currently it supports C* v1.2 and 2.0. No longer under active development|

build.gradle

@@ -1,14 +1,14 @@
 plugins {
-    id 'nebula.netflixoss' version '5.0.0'
+    id 'nebula.netflixoss' version '5.1.1'
 }
 
-// Establish version and status
 ext.githubProjectName = 'Priam'
 
 subprojects {
     apply plugin: 'nebula.netflixoss'
     apply plugin: 'java'
 
+
     group = 'com.netflix.priam'
 
     repositories {
@@ -24,7 +24,7 @@ subprojects {
         compile 'org.apache.commons:commons-lang3:3.5'
         compile 'commons-logging:commons-logging:1.2'
         compile 'org.apache.commons:commons-collections4:4.1'
-        compile 'commons-io:commons-io:2.5'
+        compile 'commons-io:commons-io:2.6'
         compile 'commons-cli:commons-cli:1.4'
         compile 'commons-httpclient:commons-httpclient:3.1'
         compile 'com.sun.jersey.contribs:jersey-multipart:1.19.4'
@@ -33,14 +33,14 @@ subprojects {
         compile 'com.sun.jersey.contribs:jersey-guice:1.19.4'
         compile 'com.google.guava:guava:21.0'
         compile 'com.google.code.findbugs:jsr305:3.0.2'
-        compile 'com.amazonaws:aws-java-sdk:1.11.211'
+        compile 'com.amazonaws:aws-java-sdk:1.11.386'
         compile 'com.google.inject:guice:4.1.0'
         compile 'com.google.inject.extensions:guice-servlet:4.1.0'
-        compile 'org.quartz-scheduler:quartz:2.2.3'
+        compile 'org.quartz-scheduler:quartz:2.3.0'
         compile 'com.googlecode.json-simple:json-simple:1.1.1'
         compile 'org.xerial.snappy:snappy-java:1.1.2.6'
-        compile 'org.yaml:snakeyaml:1.18'
-        compile 'org.apache.cassandra:cassandra-all:[1.2.0,1.2.8['
+        compile 'org.yaml:snakeyaml:1.19'
+        compile 'org.apache.cassandra:cassandra-all:2.1.17'
         compile 'javax.ws.rs:jsr311-api:1.1.1'
         compile 'joda-time:joda-time:2.9.9'
         compile 'org.apache.commons:commons-configuration2:2.1.1'
@@ -59,6 +59,7 @@ subprojects {
         }
         compile 'com.google.apis:google-api-services-storage:v1-rev100-1.22.0'
         compile 'com.google.http-client:google-http-client-jackson2:1.22.0'
+        compile 'com.netflix.spectator:spectator-api:0.74.2'
         compileOnly 'javax.servlet:javax.servlet-api:3.1.0'
         testCompile 'org.jmockit:jmockit:1.31'
         testCompile "org.spockframework:spock-core:1.1-groovy-2.4"
@@ -68,5 +69,4 @@ subprojects {
         sourceCompatibility = JavaVersion.VERSION_1_8
         targetCompatibility = JavaVersion.VERSION_1_8
     }
-}
-
+}

buildViaTravis.sh

@@ -3,21 +3,21 @@
 
 if [ "$TRAVIS_PULL_REQUEST" != "false" ]; then
   echo -e "Build Pull Request #$TRAVIS_PULL_REQUEST => Branch [$TRAVIS_BRANCH]"
-  ./gradlew build
+  ./gradlew build --stacktrace
 elif [ "$TRAVIS_PULL_REQUEST" == "false" ] && [ "$TRAVIS_TAG" == "" ]; then
   echo -e 'Build Branch with Snapshot => Branch ['$TRAVIS_BRANCH']'
-  ./gradlew -Prelease.travisci=true -PbintrayUser="${bintrayUser}" -PbintrayKey="${bintrayKey}" -PsonatypeUsername="${sonatypeUsername}" -PsonatypePassword="${sonatypePassword}" build snapshot --info --stacktrace
+  ./gradlew -Prelease.travisci=true -PbintrayUser="${bintrayUser}" -PbintrayKey="${bintrayKey}" -PsonatypeUsername="${sonatypeUsername}" -PsonatypePassword="${sonatypePassword}" build snapshot  --stacktrace
 elif [ "$TRAVIS_PULL_REQUEST" == "false" ] && [ "$TRAVIS_TAG" != "" ]; then
   echo -e 'Build Branch for Release => Branch ['$TRAVIS_BRANCH']  Tag ['$TRAVIS_TAG']'
   case "$TRAVIS_TAG" in
   *-rc\.*)
-    ./gradlew -Prelease.travisci=true -Prelease.useLastTag=true -PbintrayUser="${bintrayUser}" -PbintrayKey="${bintrayKey}" -PsonatypeUsername="${sonatypeUsername}" -PsonatypePassword="${sonatypePassword}" -PnetflixossAltCandidateRepo=true candidate --info --stacktrace
+    ./gradlew -Prelease.travisci=true -Prelease.useLastTag=true -PbintrayUser="${bintrayUser}" -PbintrayKey="${bintrayKey}" -PsonatypeUsername="${sonatypeUsername}" -PsonatypePassword="${sonatypePassword}" candidate --stacktrace
     ;;
   *)
-    ./gradlew -Prelease.travisci=true -Prelease.useLastTag=true -PbintrayUser="${bintrayUser}" -PbintrayKey="${bintrayKey}" -PsonatypeUsername="${sonatypeUsername}" -PsonatypePassword="${sonatypePassword}" final --info --stacktrace
+    ./gradlew -Prelease.travisci=true -Prelease.useLastTag=true -PbintrayUser="${bintrayUser}" -PbintrayKey="${bintrayKey}" -PsonatypeUsername="${sonatypeUsername}" -PsonatypePassword="${sonatypePassword}" final --stacktrace
     ;;
   esac
 else
   echo -e 'WARN: Should not be here => Branch ['$TRAVIS_BRANCH']  Tag ['$TRAVIS_TAG']  Pull Request ['$TRAVIS_PULL_REQUEST']'
-  ./gradlew build
-fi
+  ./gradlew build --stacktrace
+fi

gradle/convention.gradle

@@ -0,0 +1,98 @@
+// GRADLE-2087 workaround, perform after java plugin
+status = project.hasProperty('preferredStatus')?project.preferredStatus:(version.contains('SNAPSHOT')?'snapshot':'release')
+
+subprojects { project ->
+    apply plugin: 'java' // Plugin as major conventions
+
+    sourceCompatibility = 1.8
+
+    // Restore status after Java plugin
+    status = rootProject.status
+
+    task sourcesJar(type: Jar, dependsOn:classes) {
+        from sourceSets.main.allSource
+        classifier 'sources'
+        extension 'jar'
+    }
+
+    task javadocJar(type: Jar, dependsOn:javadoc) {
+        from javadoc.destinationDir
+        classifier 'javadoc'
+        extension 'jar'
+    }
+
+    configurations.create('sources')
+    configurations.create('javadoc')
+    configurations.archives {
+        extendsFrom configurations.sources
+        extendsFrom configurations.javadoc
+    }
+
+    // When outputing to an Ivy repo, we want to use the proper type field
+    gradle.taskGraph.whenReady {
+        def artifacts = project.configurations.sources.artifacts
+        def sourceArtifact = artifacts.iterator().next()
+        sourceArtifact.type = 'sources'
+    }
+
+    artifacts {
+        sources(sourcesJar) {
+            // Weird Gradle quirk where type will be used for the extension, but only for sources
+            type 'jar'
+        }
+        javadoc(javadocJar) {
+            type 'javadoc'
+        }
+    }
+
+    configurations {
+        provided {
+            description = 'much like compile, but indicates you expect the JDK or a container to provide it. It is only available on the compilation classpath, and is not transitive.'
+            transitive = true
+            visible = true
+        }
+    }
+
+    project.sourceSets {
+        main.compileClasspath += project.configurations.provided
+        main.runtimeClasspath -= project.configurations.provided
+        test.compileClasspath += project.configurations.provided
+        test.runtimeClasspath += project.configurations.provided
+    }
+}
+
+apply plugin: 'github-pages' // Used to create publishGhPages task
+
+def docTasks = [:]
+[Javadoc,ScalaDoc,Groovydoc].each{ Class docClass ->
+    def allSources = allprojects.tasks*.withType(docClass).flatten()*.source
+    if (allSources) {
+        def shortName = docClass.simpleName.toLowerCase()
+        def docTask = task "aggregate${shortName.capitalize()}"(type: docClass, description: "Aggregate subproject ${shortName}s") {
+            source = allSources
+            destinationDir = file("${project.buildDir}/docs/${shortName}")
+            doFirst {
+                def classpaths = allprojects.findAll { it.plugins.hasPlugin(JavaPlugin) }.collect { it.sourceSets.main.compileClasspath }
+                classpath = files(classpaths)
+            }
+        }
+        docTasks[shortName] = docTask
+        processGhPages.dependsOn(docTask)
+    }
+}
+
+githubPages {
+    repoUri = "git@github.com:Netflix/${rootProject.githubProjectName}.git"
+    pages {
+        docTasks.each { shortName, docTask ->
+            from(docTask.outputs.files) {
+                into "docs/${shortName}"
+            }
+        }
+    }
+}
+
+// Generate wrapper, which is distributed as part of source to alleviate the need of installing gradle
+task createWrapper(type: Wrapper) {
+    gradleVersion = '1.5'
+}