shell_exec(): Unable to execute

[Bibo-Joshi]

Steps to reproduce

1. Have a workflow that triggers on tag changes and runs a python script (for me it watermarks pdfs and then triggers NC to rescan the directory)
2. Tag multiple files at once
3. Wait for the changes to take effect

Expected behaviour

All files are watermarked

Actual behaviour

Only very few files a watermarked, sometimes only 3-4 out of 40. Prepending a random shell sleep to the command does improve this a bit, but not completey. I am wondering if NC itself or this app imposes some limits on background jobs that can be run at the same time or if this might a a limitation of my host. Maybe there is also a way to configure the number of jobs to run in a single cron-job execution?

Workflow Script app

Workflow Script app version: 1.10.0

Server configuration

Operating system: Linux 5.10.0-23-amd64 x86_64

Web server: Apache

Database: MySQL 10.10.5

PHP version: 7.4.33

Nextcloud Version: 25.0.9

Where did you install Nextcloud from: One-click-install via webhost

Signing status:

No errors have been found.

List of activated apps:

Enabled:
  - activity: 2.17.0
  - bruteforcesettings: 2.5.0
  - circles: 25.0.0
  - cloud_federation_api: 1.8.0
  - contacts: 5.3.2
  - dav: 1.24.0
  - documentserver_community: 0.1.13
  - federatedfilesharing: 1.15.0
  - files: 1.20.1
  - files_accesscontrol: 1.15.1
  - files_automatedtagging: 1.15.3
  - files_pdfviewer: 2.6.0
  - files_retention: 1.14.1
  - files_rightclick: 1.4.0
  - files_sharing: 1.17.0
  - files_trashbin: 1.15.0
  - files_versions: 1.18.0
  - firstrunwizard: 2.14.0
  - forms: 3.3.1
  - group_default_quota: 0.1.7
  - groupfolders: 13.1.4
  - logreader: 2.10.0
  - lookup_server_connector: 1.13.0
  - nextcloud_announcements: 1.14.0
  - notifications: 2.13.1
  - oauth2: 1.13.1
  - onlyoffice: 7.8.0
  - password_policy: 1.15.0
  - photos: 2.0.1
  - polls: 4.1.8
  - provisioning_api: 1.15.0
  - quota_warning: 1.17.0
  - ransomware_protection: 1.14.0
  - recommendations: 1.4.0
  - related_resources: 1.0.4
  - serverinfo: 1.15.0
  - settings: 1.7.0
  - support: 1.8.0
  - systemtags: 1.15.0
  - text: 3.6.0
  - theming: 2.0.1
  - twofactor_backupcodes: 1.14.0
  - updatenotification: 1.15.0
  - user_retention: 1.10.0
  - viewer: 1.9.0
  - workflow_script: 1.10.0
  - workflowengine: 2.7.0
Disabled:
  - admin_audit
  - calendar: 4.4.4
  - comments: 1.15.0
  - contactsinteraction: 1.6.0
  - dashboard: 7.5.0
  - encryption
  - federation: 1.15.0
  - files_external
  - privacy: 1.9.0
  - sharebymail: 1.15.0
  - survey_client: 1.13.0
  - suspicious_login
  - twofactor_totp
  - user_ldap
  - user_status: 1.5.0
  - weather_status: 1.5.0

Nextcloud configuration:

{
    "system": {
        "instanceid": "***REMOVED SENSITIVE VALUE***",
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "trusted_domains": [
            "***REMOVED SENSITIVE VALUE***"
        ],
        "datadirectory": "***REMOVED SENSITIVE VALUE***",
        "dbtype": "mysql",
        "version": "25.0.9.2",
        "overwrite.cli.url": "***REMOVED SENSITIVE VALUE***",
        "dbname": "***REMOVED SENSITIVE VALUE***",
        "dbhost": "***REMOVED SENSITIVE VALUE***",
        "dbport": "",
        "dbtableprefix": "oc_",
        "mysql.utf8mb4": true,
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "installed": true,
        "maintenance": false,
        "theme": "",
        "loglevel": 0,
        "log_type": "file",
        "default_phone_region": "DE",
        "mail_smtpmode": "smtp",
        "mail_smtpsecure": "ssl",
        "mail_sendmailmode": "smtp",
        "mail_smtphost": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpport": "465",
        "mail_from_address": "***REMOVED SENSITIVE VALUE***",
        "mail_domain": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpauth": 1,
        "mail_smtpname": "***REMOVED SENSITIVE VALUE***",
        "mail_smtppassword": "***REMOVED SENSITIVE VALUE***",
        "app_install_overwrite": [
            "apporder",
            "issuetemplate"
        ],
        "memcache.local": "\\OC\\Memcache\\APCu",
        "skeletondirectory": "",
        "updater.release.channel": "stable"
    }
}

Are you using external storage, if yes which one: no

Are you using encryption: no

Server log (data/nextcloud.log)

I see the following error log for each non-watermarked file:

[PHP] Fehler: Error: shell_exec(): Unable to execute 'sh -c 'python /home/www/nextcloud_workflow_scripts/watermark.py --event_type '\''OCP\SystemTag\ISystemTagObjectMapper::assignTags'\'' --file_id '\''2925'\'' --actor_user_id '\''username'\'' --owner_user_id '\''username'\'' --nextcloud_relative_path '\''__groupfolders/1/username/files/path_to_file.pdf'\'' --locally_available_file '\''/home/www/nextcloud/data/__groupfolders/1/path_to_file.pdf'\'' --old_nextcloud_relative_file_path '\'''\''' >/dev/null &' at /home/www/nextcloud/apps/workflow_script/lib/BackgroundJobs/Launcher.php#68 at <<closure>>

0. <<closure>>
   OC\Log\ErrorHandler::onError(2, "shell_exec(): U ... '", "/home/www/nextc ... p", 68, [["python /home/ ... "])
1. /home/www/nextcloud/apps/workflow_script/lib/BackgroundJobs/Launcher.php line 68
   shell_exec("sh -c 'python / ... &")
2. /home/www/nextcloud/lib/public/BackgroundJob/Job.php line 78
   OCA\WorkflowScript\BackgroundJobs\Launcher->run(["python /home/w ... "])
3. /home/www/nextcloud/lib/public/BackgroundJob/QueuedJob.php line 58
   OCP\BackgroundJob\Job->start(["OC\\BackgroundJob\\JobList"])
4. /home/www/nextcloud/lib/public/BackgroundJob/QueuedJob.php line 48
   OCP\BackgroundJob\QueuedJob->start(["OC\\BackgroundJob\\JobList"])
5. /home/www/nextcloud/cron.php line 152
   OCP\BackgroundJob\QueuedJob->execute(["OC\\BackgroundJob\\JobList"], ["OC\\Log"])

at 2023-08-06T16:08:03+00:00

[radoeka]

Maybe there is also a way to configure the number of jobs to run in a single cron-job execution?

See #89

[Bibo-Joshi]

Thanks for the reply! If I understand correctly, there is no limit by default, so that this can not be the issue …