Having job run on only one node in cluster

[krisleech]

I noticed that a job is seems to be run either zero, once or more than once on each node in a cluster on my local machine.

The configuration is:

config :export, Export.Scheduler,
  jobs: [
    [
      schedule: "* * * * *", 
      task: fn -> IO.puts("hello #{inspect(DateTime.utc_now())}") end
    ]
  ]

If I start three nodes then I get the following output:

alice node

[debug] file=lib/quantum/execution_broadcaster.ex  Scheduling job for execution
hello ~U[2024-11-05 14:41:00.663749Z]
[debug] file=lib/quantum/executor.ex  Task for job started on node
[debug] file=lib/quantum/executor.ex  Execute started for job
[debug] file=lib/quantum/executor.ex  Execution ended for job

Runs onces.

bob node

[debug] file=lib/quantum/execution_broadcaster.ex  Scheduling job for execution
[debug] file=lib/quantum/executor.ex  Task for job started on node
hello ~U[2024-11-05 14:41:00.049222Z]
[debug] file=lib/quantum/executor.ex  Execute started for job
[debug] file=lib/quantum/executor.ex  Execution ended for job
hello ~U[2024-11-05 14:41:00.567387Z]
[debug] file=lib/quantum/executor.ex  Execute started for job
[debug] file=lib/quantum/executor.ex  Execution ended for job

Runs twice.

kris node

[debug] file=lib/quantum/execution_broadcaster.ex  Scheduling job for execution
[debug] file=lib/quantum/executor.ex  Task for job started on node

Runs zero times.

I tried adding the Quantum.RunStrategy.Random as the run strategy (which I thought was the default anyway) but got the following error:

config :export, Export.Scheduler,
  run_strategy: Quantum.RunStrategy.Random
  jobs: [
    [
      schedule: "* * * * *", 
      task: fn -> IO.puts("hello #{inspect(DateTime.utc_now())}") end
    ]
  ]

# (FunctionClauseError) no function clause matching in Quantum.RunStrategy.Random.normalize_config!/1
            (quantum 3.5.3) lib/quantum/run_strategy/random.ex:33: Quantum.RunStrategy.Random.normalize_config!(nil)
            (elixir 1.17.3) lib/keyword.ex:1144: Keyword.update!/4
            (elixir 1.17.3) lib/keyword.ex:1148: Keyword.update!/4
            (export 0.1.0) /Users/kris/dev/nimbus/apps/abacus/deps/quantum/lib/quantum.ex:416: Export.Scheduler.__new_job__/2

Any help would be much appreciated!

I am using libcluster to enrol the nodes in a cluster and can see in Node.list() that this is the case.

[maennchen]

@krisleech Quantum automatically distributes tasks across the cluster. (Random Strategy is default)

It however does not ensure, that it only runs once in the cluster. This is why it is expected to see jobs being executed random times in your cluster.

You can do one of the following:

• If you want each node to execute the task on itself, switch to the local strategy.
• If you want jobs to be executed once across the cluster, make sure to only start it once. For example by starting it though highlander.

[krisleech]

Thanks for replying 👍 ❤️

Looking at the source for the strategies we have:

[node() | Node.list()] - all
[node()] - local
[Enum.random([node() | Node.list()])]- random

Okay I think I get it (🤞), given we have 3 nodes:

with "random" every node runs the job on a random node, so every node picks a random node to run the job on and the job is run 3 times.

This is why I see a node which runs the job twice (it got picked twice) and a node which does not run the job (it never got picked, this time).

with "all" the job would be run 3*3 times, each node runs the job on every other node and itself.

with "local" strategy each node runs the job, so it's run 3 times.

What I want is the job to run once across the cluster, it's exporting some data to S3 and we don't want to overwrite the file.

I thinking my options would be to look at highlander as suggested, I'd need to convert my module to a GenServer, or use something like a distributed lock in the module called by Quantum, so when it's called multiple times only one will actually do the work. Umm, we have Oban too so a unique job might work too.

Thanks!