Skip to content

Latest commit

 

History

History
24 lines (15 loc) · 864 Bytes

README.md

File metadata and controls

24 lines (15 loc) · 864 Bytes
Raw

Master Projekt: SIGSPATIAL

plan: build/assemble locally, copy to cluster, run on cluster

build locally

  • install sbt
  • install dependencies?
  • specify scala version in build.sbt file
  • assemble: sbt assembly

copy to cluster

  • use scp: scp target/scala-2.11/masterprojekt_sigspatial-assembly-1.0.jar sXXXXX@clusterIP:/data/home/sXXXXX/mp.jar
  • protip: insert correct username and use correct artifact version

run on cluster

basic invocation scheme:

spark-submit [spark properties] --class [submission class] [submission jar] [path to input] [path to output] [cell size in degrees] [time step size in millis]
  • ssh into cluster and in homedir run: spark-submit --class Main mp.jar yellow_tripdata_2015-01.csv ./ 0.001 7200000
  • after a succesful run you'll find your results printed to stdout and written to ~/hdfs_home/mp_out.csv