-
Notifications
You must be signed in to change notification settings - Fork 68
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Exemple of a sparkling job on spark-jobserver #16
Comments
This seems like a dependency issue with jarred scala classes... But help from a scala pro would be appreciated "status": "JOB LOADING FAILED", |
Hi, it seems as you're extending (instead of implementing) the spark.jobserver.SparkJob interface in your SampleJob class. Cheers, Chris
|
Indeed! Thank you. Making progress. Now just have to sort out a few more kinks - job is bombing on clojure issues now: |
Good luck finding that one. My tip on this: check the uberjar for the file and also check the lein dependencies using „lein deps :tree". Cheers, Chris
|
Feeling the pain already - this seems to lie on the Class loader side. |
(clojure/core__init.class is indeed in the uberjar deployed to the job server...) |
Update from the investigation.. |
Hi, did you get any further on that issue? Maybe you could share your findings to the community, so others can also use the Jobserver. Thanks! Cheers, Chris |
Hi (:gen-class :name spark.jobserver.CljJob It works, but it calls for three steps: compiling clj code, then uberjarring the scala wrapper via sbt, then submitting the jar. I'll keep you guys posted if I find a better / leiningen-only workflow. |
Hi, you might want to try https://github.com/technomancy/lein-scalac Never tried it myself, but it sounds like an improvement to that awkward process. Cheers, |
Any luck or advice on running a sparkling job on Ooyala's spark-jobserver? (https://github.com/spark-jobserver/spark-jobserver)
I am gen-classing the job after extending spark.jobserver.SparkJob but may be running into class version issues ("java.lang.IncompatibleClassChangeError").
Spark job server seems to handle Spark versions 1.1.0 1.2.0 and 1.3.0.
The text was updated successfully, but these errors were encountered: