ERROR: You cannot call this from an async context - use a thread or sync_to_async

[oswinfox]

Version

  "versions": [
    {
      "component": "core",
      "version": "3.27.1",
      "package": "pulpcore",
      "domain_compatible": true
    },
    {
      "component": "deb",
      "version": "2.21.0",
      "package": "pulp_deb",
      "domain_compatible": false
    },

Describe the bug
When creating a remote with sync_installer set to True and creating a verbatim publication, the sync fail with the following error: ERROR: You cannot call this from an async context - use a thread or sync_to_async

Here is more information of the error:

{
  "pulp_href": "/pulp/api/v3/tasks/0188bac6-291a-715c-802d-46520eb5e4bf/",
  "pulp_created": "2023-06-14T16:38:48.859585Z",
  "state": "failed",
  "name": "pulp_deb.app.tasks.synchronizing.synchronize",
  "logging_cid": "ff8a490cfa3c41258a4fd4b698357ed2",
  "created_by": "/pulp/api/v3/users/1/",
  "started_at": "2023-06-14T16:38:48.984925Z",
  "finished_at": "2023-06-14T16:39:24.424893Z",
  "error": {
    "traceback": "  File \"/usr/local/lib/python3.8/site-packages/pulpcore/tasking/tasks.py\", line 66, in _execute_task\n    result = func(*args, **kwargs)\n  File \"/usr/local/lib/python3.8/site-packages/pulp_deb/app/tasks/synchronizing.py\", line 189, in synchronize\n    DebDeclarativeVersion(first_stage, repository, mirror=mirror).create()\n  File \"/usr/local/lib/python3.8/site-packages/pulpcore/plugin/stages/declarative_version.py\", line 161, in create\n    loop.run_until_complete(pipeline)\n  File \"/usr/lib64/python3.8/asyncio/base_events.py\", line 616, in run_until_complete\n    return future.result()\n  File \"/usr/local/lib/python3.8/site-packages/pulpcore/plugin/stages/api.py\", line 220, in create_pipeline\n    await asyncio.gather(*futures)\n  File \"/usr/local/lib/python3.8/site-packages/pulpcore/plugin/stages/api.py\", line 41, in __call__\n    await self.run()\n  File \"/usr/local/lib/python3.8/site-packages/pulp_deb/app/tasks/synchronizing.py\", line 570, in run\n    await asyncio.gather(\n  File \"/usr/local/lib/python3.8/site-packages/pulp_deb/app/tasks/synchronizing.py\", line 709, in _handle_distribution\n    await asyncio.gather(*sub_tasks)\n  File \"/usr/local/lib/python3.8/site-packages/pulp_deb/app/tasks/synchronizing.py\", line 800, in _handle_component\n    await asyncio.gather(*pending_tasks)\n  File \"/usr/local/lib/python3.8/site-packages/pulp_deb/app/tasks/synchronizing.py\", line 1078, in _handle_installer_file_index\n    for content_artifact in installer_file_index.contentartifact_set.all():\n  File \"/usr/local/lib/python3.8/site-packages/django/db/models/query.py\", line 398, in __iter__\n    self._fetch_all()\n  File \"/usr/local/lib/python3.8/site-packages/django/db/models/query.py\", line 1881, in _fetch_all\n    self._result_cache = list(self._iterable_class(self))\n  File \"/usr/local/lib/python3.8/site-packages/django/db/models/query.py\", line 91, in __iter__\n    results = compiler.execute_sql(\n  File \"/usr/local/lib/python3.8/site-packages/django/db/models/sql/compiler.py\", line 1560, in execute_sql\n    cursor = self.connection.cursor()\n  File \"/usr/local/lib/python3.8/site-packages/django/utils/asyncio.py\", line 24, in inner\n    raise SynchronousOnlyOperation(message)\n",
    "description": "You cannot call this from an async context - use a thread or sync_to_async."
  },
  "worker": "/pulp/api/v3/workers/0188b5ab-bb0f-781b-8e02-212e0ad617bd/",
  "parent_task": null,
  "child_tasks": [],
  "task_group": null,
  "progress_reports": [
    {
      "message": "Downloading Artifacts",
      "code": "sync.downloading.artifacts",
      "state": "canceled",
      "total": null,
      "done": 3,
      "suffix": null
    },
    {
      "message": "Update ReleaseFile units",
      "code": "update.release_file",
      "state": "canceled",
      "total": null,
      "done": 1,
      "suffix": null
    },
    {
      "message": "Update PackageIndex units",
      "code": "update.packageindex",
      "state": "canceled",
      "total": null,
      "done": 3,
      "suffix": null
    },
    {
      "message": "Associating Content",
      "code": "associating.content",
      "state": "canceled",
      "total": null,
      "done": 0,
      "suffix": null
    }
  ],
  "created_resources": [],
  "reserved_resources_record": [
    "/pulp/api/v3/repositories/deb/apt/01889593-ce43-7a7e-8b89-d4f896dc156e/",
    "shared:/pulp/api/v3/remotes/deb/apt/01889593-c6fa-7f4a-b36a-1ef36e91d847/"
  ]
}

To Reproduce

1. Create a deb remote with sync_installer set to True
2. Create a verbatim publication
3. Try to sync the repository

FYI when doing exactly the same for buster or bullseye update or security repository then it work and there is no issue; this only happens with the main repo buster or bullseye!

This is the link used for the remote: http://ftp.debian.org/debian/

Here is an example of on of my created remote:

  {
    "pulp_href": "/pulp/api/v3/remotes/deb/apt/01889593-c6fa-7f4a-b36a-1ef36e91d847/",
    "pulp_created": "2023-06-07T11:17:49.947686Z",
    "name": "buster",
    "url": "http://ftp.debian.org/debian/",
    "ca_cert": null,
    "client_cert": null,
    "tls_validation": true,
    "proxy_url": null,
    "pulp_labels": {},
    "pulp_last_updated": "2023-06-15T07:19:44.137570Z",
    "download_concurrency": 4,
    "max_retries": null,
    "policy": "immediate",
    "total_timeout": null,
    "connect_timeout": null,
    "sock_connect_timeout": null,
    "sock_read_timeout": null,
    "headers": null,
    "rate_limit": null,
    "hidden_fields": [
      {
        "name": "client_key",
        "is_set": false
      },
      {
        "name": "proxy_username",
        "is_set": false
      },
      {
        "name": "proxy_password",
        "is_set": false
      },
      {
        "name": "username",
        "is_set": false
      },
      {
        "name": "password",
        "is_set": false
      }
    ],
    "distributions": "buster",
    "components": null,
    "architectures": "amd64",
    "sync_sources": false,
    "sync_udebs": false,
    "sync_installer": true,
    "gpgkey": null,
    "ignore_missing_package_indices": false
  },

Expected behavior

We expected the sync to work without issue, because of this issue the sync stop.

Additional context

In IRC discussing with quba42, the issue seems to be related with: is most likely in pulp_deb/app/tasks/synchronizing.py line 1078

[quba42]

Probably a duplicate of #597

We should fix this, it should not be that hard to fix!

[alexey-orekhov]

You can try
http --auth admin:admin post $BASE_ADDR/pulp/api/v3/remotes/deb/apt/ name="bullseye-amd64-11-7" url="http://ftp.debian.org/debian/" distributions="bullseye" architectures="amd64" components="main main/debian-installer" sync_udebs=true
This should help avoid crashing the task

[quba42]

@alexey-orekhov It is definitely not meant to work like that by design...

[n-pochet]

Hi @quba42,

In which version of pulp_deb will this be available?

[quba42]

@n-pochet We tried to release this as part of 2.21.1 and 2.20.3 yesterday, but the release pipeline is broken. This means we are blocked until pulp/plugin_template#770 is fixed.

[quba42]

We are not currently planning to backport it to the 2.19 or 2.18 branch since there are merge conflicts and I am not certain it is needed there. Also, we are about to stop supporting those branches (there may be one final dot release for 2.18 and 2.19 each).