It makes sense to me that you need to explicitly specify jar deps from examples. I'm more confused by the cases where its working without those.

@erikerlandson can you elaborate on why this makes sense? I just ran into this and don't understand why the jar needs to be provided when I only intend to execute a Python app.

I don't think this is correct. @ifilonenko can you take another look at this?

@sahilprasad in general python jobs may execute jvm code, and in particular they may need additional jar deps that have to be supplied using --jars

@mccheah do you mean the spark-examples jar shouldn't be needed to run pi.py ?

I think in this particular example the error message is either incorrect or we're not passing arguments along properly: Error: Could not find or load main class .opt.spark.jars.activation-1.1.1.jar

If it was just a classpath failure then I would expect a ClassNotFoundException or something similar. Was there a stack trace @ifilonenko?

@mccheah when I replicated this problem, the error that @ifilonenko provided is the only thing that I see.

What is quite interesting is how I didn't have the missing jar exception when I ran this with #364. But as @erikerlandson mentioned, this seems to be attributed to the spark-examples jar being needed when running pyspark examples. I would assume that a better test would be to run PySpark tasks outside of the spark-examples and see if the error persists.

My hypothesis is that the Python Dockerfile is running an ill-formatted Java command. @sahilprasad @ifilonenko - if either of you can track down how the command isn't being formed properly then that would be helpful. We had to fix a similar problem with #444 for example.

@mccheah I can take a look. I also think that it's an ill-formatted Java command that's at the root of the issue, but I'll update this issue with what I find.

w.r.t. making it easier to observe command formatting, we might tweak entrypoint.sh to print out the command it's going to execute
#462

I was able to get @ifilonenko's first example working without --jars. When the java classpath is just /opt/spark/jars/* and submitted via the -cp flag within the Dockerfile, the jars are somehow not recognized. Adding a colon to the beginning of that classpath got it working. Should I submit a PR for this, or is there a better solution?

See changes here: branch-2.2-kubernetes...sahilprasad:python-jars

@sahilprasad you should submit that as PR and we can discuss - are we over-writing existing entries on SPARK_CLASSPATH ? If it's empty I'd not expect prepending a : would change the result.