Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Run script immediatly after trigger #91

Open
msi-tl opened this issue May 21, 2021 · 18 comments
Open

Run script immediatly after trigger #91

msi-tl opened this issue May 21, 2021 · 18 comments

Comments

@msi-tl
Copy link

msi-tl commented May 21, 2021

Currently the scripts are run with the next cron job.
This is not usable for some cases, e.g. I try to lock a file with an other app if it's opened or updated.
I don't want to trigger cron any 10 secounds because there are other jobs like generating previews and file scan etc.
Other APP like https://github.com/nextcloud/files_accesscontrol are grip immediatly.
How can I configure this direct behavior with workflow_script?

@blizzz
Copy link
Member

blizzz commented May 21, 2021

files_accesscontrol addresses a very different use case. workflow_script always triggers the script per background job, currently.

@msi-tl
Copy link
Author

msi-tl commented May 21, 2021

Yes, it's a different use case.
But the lock strategie in Nextcloud is not useable in corporate environment and I try to workaround this failure.
Currently I use the workflow to trigger a script if a file is opened or updated.
The script uses the app https://github.com/nextcloud/files_lock to lock the file.
Automated unlock time is 5 min.
This seems to be a good solution if workflow_script would be able to run the script immediatly.

I did not find a better solution. The currently lock strategie is unable to handle locks and migrations of multiple users on multiple access ways to the files (Web, WebDAV, Client).

@blizzz
Copy link
Member

blizzz commented May 21, 2021

files_lock could be extended to offer a flow actions itself and act accordingly.

@kousu
Copy link

kousu commented Feb 4, 2023

Putting aside the specific use case above, it would be handy to run scripts immediately after the trigger. When I discovered this app I thought "great, I can do faster push processing instead of pull processing". But this app at the moment just queues itself on the pull-queue that is cron. It would be almost as easy to write my own cronjobs that just enforce their own quit-early event filtering and have the same performance.

In my case I am trying to provision some credentials when new users get created, and I think I can do it with this app, but I don't want my HR people to be waiting around for 5 minutes for cron to wake up and run the triggers when they're trying to onboard someone.

My workaround for the moment is replace this cronjob

*/5  *  *  *  * php -f /var/www/nextcloud/cron.php

with

*  *  *  *  * php -f /var/www/nextcloud/cron.php

which I expect will have some negative consequences down the road.

@blizzz
Copy link
Member

blizzz commented Jun 6, 2023

Currently easiest thing is having a timer/cron/daemon running that looks for registered jobs and executes them. The latter can be achieved with something like:

php occ  background-job:list --output json  --class 'OCA\WorkflowScript\BackgroundJobs\Launcher' | jq -r .[].id | xargs -n1 php occ  background-job:execute 

@radoeka
Copy link

radoeka commented Aug 18, 2023

What's the reason (the background) that the workflow job(s) are not executed immediately?
Is this to prevent that that the Nexcloud engine gets blocked by a long running script?

Just an idea:
Would it be possible that the workflow engine touches a file, when the engine determines that a workflow_script is to be executed?
The file should be touched in a directory that is watched by inotify.

As soon as the file is touched inotify comes into action and execute the workflow scripts.
This script could figure out with the command mentioned in the previous comment the jobs that need to be run.

The directory that could be used as an example is, I think, /run/nextcloud/workflow-scripts (on linux).

Perhaps, multiple files must be created with each file containing the job-name-id that is to be executed. When that particular job has been executed the related file must be removed.

Would something like this be possible (this requires setting up inotify, but with a good procedure that may not be too difficult)?

@radoeka
Copy link

radoeka commented Aug 19, 2023

As soon as the file is touched inotify comes into action and execute the workflow scripts.
This script could figure out with the command mentioned in the previous comment the jobs that need to be run.

Hmmm, I think it should just run the "nextcloud-x.y.z/cron.php" job, that is now run from the cronjob.

@blizzz
Copy link
Member

blizzz commented Sep 11, 2023

Is this to prevent that that the Nexcloud engine gets blocked by a long running script?

Yes, and potential Exceptions that might be thrown.

The file should be touched in a directory that is watched by inotify.

You could watch the database file for oc_jobs.

@radoeka
Copy link

radoeka commented Sep 19, 2023

You could watch the database file for oc_jobs.

How would that be done?

inotify (inode notification) is a buildin Linux kernel module that gets triggered when a file is created, modified, deleted, etc. There is no polling needed. One sets a watch on the directory and as soon as a file in the directory is added, modified or deleted, the watcher activates the specified/configured script.

The script can then start to look for Nextcloud jobs. And as soon as these appear in the Nextcoud table ( I assume oc_jobs), the jobs can be processed (occ cron background-job:list....)

So yes, use oc_jobs but after inotify is triggered by Nextcloud.

@blizzz
Copy link
Member

blizzz commented Sep 26, 2023

May depend on the database you use, but for instance MySQL/MariaDB stores each table in a single file, so you can watch this one for changes. Something like /var/lib/mysql/nextcloud/oc_jobs.ibd.

@radoeka
Copy link

radoeka commented Oct 11, 2023

for instance MySQL/MariaDB stores each table in a single file, so you can watch this one for changes
Nice one. I didn't realize that this was possible. I run only an small Nextcloud setup (with sqlite3), hence for me this does now work.
I don't know about the creator of this ticket, whether he/she can watch the database files.
But would this work with other database as well, like postgress?

Should each job in the oc_jobs.ibd table result in a trigger, or should there only be a trigger for certain tasks/jobs in the script workflow?

Would a DB engine independent solution, be possible and perhaps would that be better?

@blizzz
Copy link
Member

blizzz commented Nov 13, 2023

For sqlite you could only watch the database file itself which would be too noisy i guess. Not sure about postgres, i have not seen dedicated files there, but I did not spend too much time on it.

A different thing doable it to outsource this mechanism into another app. We could dispatch an Event on registering those jobs, and a dedicated app can touch a file or run some other action.

@radoeka
Copy link

radoeka commented Nov 19, 2023

@blizzz thanks for the followup and idea. Another app sounds rather heavy to me, for what I think is something simple.
What is against it to touch a file on the file system?
Touching a file, can be started with besides dispatching an event. Isn't it?

A dedicated app can be created to process the dispatched event later, in the meantime people can use the touched file using inotify.

In case of dispatching an event, must the event be removed? Or will that be taken care of automatically?

Perhaps for both cases (dispatching an event, and creating a file) a configuration option can be created where the admin can enable the desired mode?

@radoeka
Copy link

radoeka commented Feb 10, 2024

And there it is the other app: https://github.com/icewind1991/files_inotify but i don't know if it is suitable for the case discussed above.

@XueSheng-GIT
Copy link

@radoeka did you find a solution or practical workaround for this issue?

@radoeka
Copy link

radoeka commented Aug 19, 2024

@XueSheng-GIT no. I'm still checking every 5 minutes if there is something to update. It would be better with an inotify solution.

@XueSheng-GIT
Copy link

XueSheng-GIT commented Aug 19, 2024

@radoeka I'm using postgresql as database and had a look into trigger functions (https://www.postgresql.org/docs/current/plpgsql-trigger.html).

As an initial proof of concept, I added a trigger and trigger function for the oc_jobs table on INSERT (class OCA\WorkflowScript\BackgroundJobs\Launcher). Job id is sent via pg_notify to a listener script (python) which executes the job via php occ background-job:execute.

First tests were successful and jobs are immediatly executed.

@XueSheng-GIT
Copy link

@radoeka since NC30 there's occ background-job:worker which also should solve your issue.

The following reference (related to Nextcloud AI) can be taken to create a systemd service for this worker. But you need to adapt the required job-classes.
https://docs.nextcloud.com/server/latest/admin_manual/ai/overview.html#ai-overview-improve-ai-task-pickup-speed

You can do a simple test on console first to see if your workflow scripts are triggered immediately. e.g.:

sudo -u www-data occ background-job:worker 'OC\TaskProcessing\SynchronousBackgroundJob' 'OCA\WorkflowScript\BackgroundJobs\Launcher' 

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

5 participants