Exemple of a sparkling job on spark-jobserver

[francoisroyer]

Any luck or advice on running a sparkling job on Ooyala's spark-jobserver? (https://github.com/spark-jobserver/spark-jobserver)
I am gen-classing the job after extending spark.jobserver.SparkJob but may be running into class version issues ("java.lang.IncompatibleClassChangeError").
Spark job server seems to handle Spark versions 1.1.0 1.2.0 and 1.3.0.

[francoisroyer]

This seems like a dependency issue with jarred scala classes... But help from a scala pro would be appreciated

"status": "JOB LOADING FAILED",
"result": {
"message": "class SampleJob has interface spark.jobserver.SparkJob as super class",
"errorClass": "java.lang.IncompatibleClassChangeError",
"stack": ["java.lang.ClassLoader.defineClass1(Native Method)", ...

[chrisbetz]

Hi,

it seems as you're extending (instead of implementing) the spark.jobserver.SparkJob interface in your SampleJob class.

Cheers,

Chris

Am 28.04.2015 um 21:28 schrieb francoisroyer notifications@github.com:

This seems like a dependency issue with jarred scala classes... But help from a scala pro would be appreciated

"status": "JOB LOADING FAILED",
"result": {
"message": "class SampleJob has interface spark.jobserver.SparkJob as super class",
"errorClass": "java.lang.IncompatibleClassChangeError",
"stack": ["java.lang.ClassLoader.defineClass1(Native Method)", ...

—
Reply to this email directly or view it on GitHub.

[francoisroyer]

Indeed! Thank you. Making progress. Now just have to sort out a few more kinks - job is bombing on clojure issues now:
job-server[ERROR] Caused by: java.io.FileNotFoundException: Could not locate clojure/core__init.class or clojure/core.clj

[chrisbetz]

Good luck finding that one. My tip on this: check the uberjar for the file and also check the lein dependencies using „lein deps :tree".

Cheers,

Chris

Am 28.04.2015 um 22:23 schrieb francoisroyer notifications@github.com:

Indeed! Thank you. Making progress. Now just have to sort out a few more kinks - job is bombing on clojure issues now:
job-server[ERROR] Caused by: java.io.FileNotFoundException: Could not locate clojure/core__init.class or clojure/core.clj

—
Reply to this email directly or view it on GitHub #16 (comment).

[francoisroyer]

Feeling the pain already - this seems to lie on the Class loader side.

[francoisroyer]

(clojure/core__init.class is indeed in the uberjar deployed to the job server...)

[francoisroyer]

Update from the investigation..
Since the loader is choking at clojure.lang.RT.load, and that all classes are definitely present in the jar, this may be a sign that the classloader responsible for loading clj code is not correctly set in some thread.

[chrisbetz]

Hi,

did you get any further on that issue? Maybe you could share your findings to the community, so others can also use the Jobserver.

Thanks!

Cheers,

Chris

[francoisroyer]

Hi
I haven't solved it yet in a satisfactory way. The workaround has been to create a SparkJob from Clojure and exposing it via a Scala wrapper, e.g.

(:gen-class :name spark.jobserver.CljJob
:methods [#^{:static true} [validate [org.apache.spark.SparkContext com.typesafe.config.Config] String]
#^{:static true} [runJob [org.apache.spark.SparkContext com.typesafe.config.Config] Object]]
)

It works, but it calls for three steps: compiling clj code, then uberjarring the scala wrapper via sbt, then submitting the jar. I'll keep you guys posted if I find a better / leiningen-only workflow.

[chrisbetz]

Hi,

you might want to try https://github.com/technomancy/lein-scalac

Never tried it myself, but it sounds like an improvement to that awkward process.

Cheers,
Chris