Skip to content

Commit

Permalink
[SPARK-45497][K8S] Add a symbolic link file spark-examples.jar in K…
Browse files Browse the repository at this point in the history
…8s Docker images

### What changes were proposed in this pull request?

This PR aims to add a symbolic link file, `spark-examples.jar`, in the example jar directory.

```
$ docker run -it --rm spark:latest ls -al /opt/spark/examples/jars  | tail -n6
total 1620
drwxr-xr-x 1 root root    4096 Oct 11 04:37 .
drwxr-xr-x 1 root root    4096 Sep  9 02:08 ..
-rw-r--r-- 1 root root   78803 Sep  9 02:08 scopt_2.12-3.7.1.jar
-rw-r--r-- 1 root root 1564255 Sep  9 02:08 spark-examples_2.12-3.5.0.jar
lrwxrwxrwx 1 root root      29 Oct 11 04:37 spark-examples.jar -> spark-examples_2.12-3.5.0.jar
```

### Why are the changes needed?

Like PySpark example (`pi.py`), we can submit the examples without considering the version numbers which was painful before.
```
bin/spark-submit \
--master k8s://$K8S_MASTER \
--deploy-mode cluster \
...
--class org.apache.spark.examples.SparkPi \
local:///opt/spark/examples/jars/spark-examples.jar 10000
```

The following is the driver pod log.
```
+ exec /usr/bin/tini -s -- /opt/spark/bin/spark-submit ...
--deploy-mode client
--properties-file /opt/spark/conf/spark.properties
--class org.apache.spark.examples.SparkPi
local:///opt/spark/examples/jars/spark-examples.jar 10000
Files  local:///opt/spark/examples/jars/spark-examples.jar from /opt/spark/examples/jars/spark-examples.jar to /opt/spark/work-dir/./spark-examples.jar
```

### Does this PR introduce _any_ user-facing change?

No, this is an additional file.

### How was this patch tested?

Manually build the docker image and do `ls`.

### Was this patch authored or co-authored using generative AI tooling?

No.

Closes apache#43324 from dongjoon-hyun/SPARK-45497.

Authored-by: Dongjoon Hyun <dhyun@apple.com>
Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
  • Loading branch information
dongjoon-hyun committed Oct 11, 2023
1 parent 2784349 commit 64ac59d
Showing 1 changed file with 1 addition and 0 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,7 @@ COPY sbin /opt/spark/sbin
COPY kubernetes/dockerfiles/spark/entrypoint.sh /opt/
COPY kubernetes/dockerfiles/spark/decom.sh /opt/
COPY examples /opt/spark/examples
RUN ln -s $(basename $(ls /opt/spark/examples/jars/spark-examples_*.jar)) /opt/spark/examples/jars/spark-examples.jar
COPY kubernetes/tests /opt/spark/tests
COPY data /opt/spark/data

Expand Down

0 comments on commit 64ac59d

Please sign in to comment.