Cluster

Jump to bottom Edit New page

Ondřej Moravčík edited this page May 3, 2015 · 2 revisions

Mesos

First make sure that Spark works on Mesos by this guide http://spark.apache.org/docs/latest/running-on-mesos.html.

Configuration can look like this. Make sure that spark.executor.uri contain spark version that is compatible.

Spark.config do
  set 'spark.master', 'mesos://localhost:5050'
  set 'spark.executor.uri', 'http://1.2.3.4/spark.tgz'
end

Also take a look on configuration page to see all executor configuration. For example if you are using rbenv or rvm:

Spark.config.set(
  'spark.ruby.executor.command',
  %{bash --norc -i -c "export HOME=/home/user; cd; source .bashrc; %s"}
)