Immediately run jobs

[alexahnfetch]

Previously mentioned in #509, but I'm opening a ticket hoping that I can change your mind. Setting an arbitrary padding to set a future schedule date is not a rigorous solution, and it forces developers to think about all the things that can potentially go wrong (e.g. clock skew) - to the point that they will conservatively use a large padding.

It would be nice if we had something like:

@at now

or allow an empty schedule field:

curl localhost:8080/v1/jobs -XPOST -o - -D '{
  "name": "job1",
  "schedule": "",
  "timezone": "Europe/Berlin",
  "owner": "Platform Team",
  "owner_email": "platform@example.com",
  "disabled": false,
  "tags": {
    "dkron_server": "true:1"
  },
  "concurrency": "allow",
  "executor": "shell",
  "executor_config": {
    "command": "date"
  }
}'

where an empty schedule field specifies an immediately run job (similar to https://mesos.github.io/chronos/docs/api.html#adding-a-scheduled-job).

Expected Behavior

curl -v localhost:8080/v1/jobs -XPOST -d '{
>   "name": "job1",
>   "schedule": "",
>   "disabled": false,
>   "tags": {
>     "dkron_server": "true:1"
>   },
>   "concurrency": "allow",
>   "executor": "shell",
>   "executor_config": {
>     "command": "date"
>   }
> }'
Note: Unnecessary use of -X or --request, POST is already inferred.
*   Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 8080 (#0)
> POST /v1/jobs HTTP/1.1
> Host: localhost:8080
> User-Agent: curl/7.54.0
> Accept: */*
> Content-Length: 204
> Content-Type: application/x-www-form-urlencoded
> 
* upload completely sent off: 214 out of 214 bytes
< HTTP/1.1 201 Created
< Content-Type: application/json; charset=utf-8
< Location: /v1/jobs/job4
< Date: Thu, 25 Jul 2019 00:39:38 GMT
< Content-Length: 427
< 
* Connection #0 to host localhost left intact
{"name":"job4","timezone":"","schedule":"","owner":"","owner_email":"","success_count":0,"error_count":0,"last_success":"0001-01-01T00:00:00Z","last_error":"0001-01-01T00:00:00Z","disabled":false,"tags":{"dkron_server":"true:1"},"retries":0,"dependent_jobs":null,"parent_job":"","processors":null,"concurrency":"allow","executor":"shell","executor_config":{"command":"date"},"status":"","next":"0001-01-01T00:00:00Z"}

Actual Behavior

curl -v localhost:8080/v1/jobs -XPOST -d '{
>   "name": "job4",
>   "schedule": "",
>   "disabled": false,
>   "tags": {
>     "dkron_server": "true:1"
>   },
>   "concurrency": "allow",
>   "executor": "shell",
>   "executor_config": {
>     "command": "date"
>   }
> }'
Note: Unnecessary use of -X or --request, POST is already inferred.
*   Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 8080 (#0)
> POST /v1/jobs HTTP/1.1
> Host: localhost:8080
> User-Agent: curl/7.54.0
> Accept: */*
> Content-Length: 204
> Content-Type: application/x-www-form-urlencoded
> 
* upload completely sent off: 204 out of 204 bytes
< HTTP/1.1 422 Unprocessable Entity
< Date: Thu, 25 Jul 2019 00:38:35 GMT
< Content-Length: 0
< 
* Connection #0 to host localhost left intact

Specifications

• Version: dkron_1.2.4_darwin_amd64
• Platform: Darwin
• Backend store: Default

[yvanoers]

This relates to #460. My (currently only) comment there applies to this as well, including the part about the use case.

@alexahnfetch Regarding the use case: I (dare I say we) would like to understand what it is that you are trying to accomplish with such a feature. Running a job immediately can be accomplished by programmatically running it (has to exist already, oc) using the API: POST to /jobs/{job_name}

Come to think of it, supporting running immediately could be implemented by adding semantics to the API instead of messing with the scheduler. Something like POST /jobs?immediate=1.

[fopina]

When I started using dkron some time ago I faced exactly the same issue: dkron was great for scheduled tasks, but it was also great to run one-time async jobs.

So, I looked for how to run "immediately" (didn't even try@at CURRENT_TIME as that'd very likely miss between my app server and dkron server).
An @at now or just @now would be awesome, but it didn't exist.
As we could trigger jobs manually (as @yvanoers mentioned), I aimed for that.
But the new issue was how to setup a job with no schedule. Empty schedule didn't work and there was no @at never...

So I just went with an hardcoded date in the past for those jobs. I create it and I trigger it and it has been working just fine @alexahnfetch

But I do second the request to have it more official, either by being able to setup jobs with no schedule (to trigger manually right after) or with a "now"/"asap" schedule.

[vcastellm]

Interesting conversation here, I can understand the need to run a job inmediately, but regarding:

but it was also great to run one-time async jobs.

Looks to me like a task for a background jobs queue, not dkron but certainly it can do it.

@fopina and @alexahnfetch, under your use case, do you want the inmediate job to dissapear after being executed or you exepect to have it around?

[fopina]

I, personally, would expect it to lie around so I can check execution times and view logs/stats after it’s done, pretty much like it does now setting an invalid @at and starting manually.

[alexahnfetch]

@yvanoers I'd like the possibility of using dkron as a task queue. It doesn't seem natural to issue two REST calls to run a job immediately, as it leads to more error handling code due to REST calls potentially failing. If dkron is a way to control when jobs get executed (e.g. controlling time), then the schedule attribute should allow us to have a way to say "as early as possible". The more expressive a scheduler is, the more control it gives us over time, and ultimately, the more useful it is for designing applications. Regarding #460, I think if jobs with schedules set in the past were executed, then that would also be an acceptable approach. The only concern there is that it would break backwards compatibility.

@Victorcoder I'd like it to behave just like any other job so I can see if it was executed or not.

[vcastellm]

Given the fact that you can not create an issue without schedule (except when it has a parent job), without looking at the code I would say to add a new special directive @now or something similar.

[yvanoers]

@alexahnfetch Thanks for your explanation.
I suspect that @congphan's (requester from #460) use case is also to be able to start jobs when they arrive in dkron. Maybe he can corroborate.
I am much more in favour of adding "create & start" to dkron than adding a grace period for jobs arriving late.

@Victorcoder, IMHO @now is too ambiguous: it is not clear when such a job is supposed to be running. Alternative suggestion: @creation.
Combined with an API request parameter, @never (somewhat like @fopina mentioned) could also be useful.

Using an API request parameter has the additional benefit of allowing jobs that do have a regular schedule to be started once at creation and then follow their regular schedule, without having to change the schedule afterwards (or set the schedule correctly at creation and kick them off programmatically afterwards).

[vcastellm]

Implemented in #592