Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Batch client not splitting up large batch submissions #14637

Open
patrick-schultz opened this issue Jul 25, 2024 · 0 comments
Open

Batch client not splitting up large batch submissions #14637

patrick-schultz opened this issue Jul 25, 2024 · 0 comments
Assignees

Comments

@patrick-schultz
Copy link
Collaborator

What happened?

Reported by Ben Weisburd here

Version

0.2.132

Relevant log output

File "/Users/weisburd/code/step-pipeline/step_pipeline/batch.py", line 300, in run
    result = self._run_batch_obj()
  File "/Users/weisburd/code/step-pipeline/step_pipeline/batch.py", line 368, in _run_batch_obj
    result = self._batch.run(
  File "/usr/local/lib/python3.9/site-packages/hailtop/batch/batch.py", line 716, in run
    return async_to_blocking(self._async_run(dry_run, verbose, delete_scratch_on_exit, **backend_kwargs))  # type: ignore
  File "/usr/local/lib/python3.9/site-packages/hailtop/utils/utils.py", line 186, in async_to_blocking
    raise exc
  File "/usr/local/lib/python3.9/site-packages/hailtop/utils/utils.py", line 181, in async_to_blocking
    return loop.run_until_complete(task)
  File "/usr/local/lib/python3.9/site-packages/nest_asyncio.py", line 99, in run_until_complete
    return f.result()
  File "/usr/local/Cellar/python@3.9/3.9.18_2/Frameworks/Python.framework/Versions/3.9/lib/python3.9/asyncio/futures.py", line 201, in result
    raise self._exception
  File "/usr/local/Cellar/python@3.9/3.9.18_2/Frameworks/Python.framework/Versions/3.9/lib/python3.9/asyncio/tasks.py", line 256, in __step
    result = coro.send(None)
  File "/usr/local/lib/python3.9/site-packages/hailtop/batch/batch.py", line 747, in _async_run
    run_result = await self._backend._async_run(self, dry_run, verbose, delete_scratch_on_exit, **backend_kwargs)  # pylint: disable=assignment-from-no-return
  File "/usr/local/lib/python3.9/site-packages/hailtop/batch/backend.py", line 901, in _async_run
    await async_batch.submit(disable_progress_bar=disable_progress_bar)
  File "/usr/local/lib/python3.9/site-packages/hailtop/batch_client/aioclient.py", line 1234, in submit
    start_job_group_id, start_job_id = await self._submit(
  File "/usr/local/lib/python3.9/site-packages/hailtop/batch_client/aioclient.py", line 1166, in _submit
    byte_specs_bunches = self._create_bunches(
  File "/usr/local/lib/python3.9/site-packages/hailtop/batch_client/aioclient.py", line 1035, in _create_bunches
    assert n_bytes < max_bunch_bytesize, (
AssertionError: every spec must be less than max_bunch_bytesize, 1048576B, but {"always_run":false,"always_copy_output":false,"job_id":6190,"absolute_parent_ids":[],"in_update_parent_ids":[1073,1069,1082,429
@patrick-schultz patrick-schultz added needs-triage A brand new issue that needs triaging. batch labels Jul 25, 2024
@chrisvittal chrisvittal added bug and removed needs-triage A brand new issue that needs triaging. labels Jul 29, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants