Google Batch run hangs when a job fail to start

[pditommaso]

Bug report

A workflow execution can remain in running status indefinitely when a job fails to start.

This very likely causes by the following snippet

nextflow/plugins/nf-google/src/main/nextflow/cloud/google/batch/GoogleBatchTaskHandler.groovy

Lines 448 to 449 in 62fc001

	if( !tasks.iterator().hasNext() )
	return 'PENDING'

More in details the error arise when a job transition from queued to failed without entering in running status. For example

» gcloud batch jobs describe projects/687213979415/locations/europe-north1/jobs/nf-ddfcf4ff-1732855072018 --format="json(status.statusEvents)"
{
  "status": {
    "statusEvents": [
      {
        "description": "Job state is set from QUEUED to SCHEDULED for job projects/687213979415/locations/europe-north1/jobs/nf-ddfcf4ff-1732855072018.",
        "eventTime": "2024-11-29T04:37:59.862173786Z",
        "type": "STATUS_CHANGED"
      },
      {
        "description": "Job gets non-retryable information Batch Error: code - CODE_GCE_INVALID_FIELD_VALUE, description - Invalid value for field 'operation': ''. No zone supports all of the provided instance templates. The following errors detail the failure. The resource 'projects/687213979415/zones/europe-north1-a/acceleratorTypes/nvidia-tesla-t4' was not found The resource 'projects/687213979415/zones/europe-north1-b/acceleratorTypes/nvidia-tesla-t4' was not found The resource 'projects/687213979415/zones/europe-north1-c/acceleratorTypes/nvidia-tesla-t4' was not found.",
        "eventTime": "2024-11-29T04:38:28.995897436Z",
        "type": "OPERATIONAL_INFO"
      },
      {
        "description": "Job state is set from SCHEDULED to SCHEDULED_PENDING_FAILED for job projects/687213979415/locations/europe-north1/jobs/nf-ddfcf4ff-1732855072018.",
        "eventTime": "2024-11-29T04:38:29.014619222Z",
        "type": "STATUS_CHANGED"
      },
      {
        "description": "Job state is set from SCHEDULED_PENDING_FAILED to FAILED for job projects/687213979415/locations/europe-north1/jobs/nf-ddfcf4ff-1732855072018.",
        "eventTime": "2024-11-29T04:38:40.347887659Z",
        "type": "STATUS_CHANGED"
      }
    ]
  }
}

The task describe reports the following

gcloud batch tasks describe projects/687213979415/locations/europe-north1/jobs/nf-ddfcf4ff-1732855072018/taskGroups/group0/tasks/0 --location=europe-north1
name: projects/687213979415/locations/europe-north1/jobs/nf-ddfcf4ff-1732855072018/taskGroups/group0/tasks/0
status:
  state: PENDING
  statusEvents:
  - eventTime: '2024-11-29T04:37:59.891544112Z'
    taskState: PENDING
    type: CREATED

However the task list request request returns no tasks

»  gcloud batch tasks list --job=nf-ddfcf4ff-1732855072018 --location=europe-north1Listed 0 items.

Worth mentioning this is a regression introduced by this change https://github.com/nextflow-io/nextflow/pull/3892/files#diff-b5373e32074aac589c96f4ce9de7a10a7621721006052186873d610388acaf08R436-R440