Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Parallelize fetch and expand phase across APKs #775

Closed
Tracked by #781
jonjohnsonjr opened this issue Jul 1, 2023 · 0 comments
Closed
Tracked by #781

Parallelize fetch and expand phase across APKs #775

jonjohnsonjr opened this issue Jul 1, 2023 · 0 comments

Comments

@jonjohnsonjr
Copy link
Contributor

Building on #773

We currently do something that looks like this:

def installPackage(pkg):
  apk = fetchPackage(pkg)
  files = ExpandApk(apk)
  installFiles(files)

def installPackages(packages):
  for pkg in packages:
    installPackage(apk)

Where we do fetch, expand, and install serially for every package.

The only part that really needs to happen serially is the installFiles step. We can concurrently fetch and expand all the packages, then serially run installFiles on all the results.

We probably want the fetch and expand stage to return a promise-like thing so we can start on the installFiles stage as soon as the next result has finished being fetched and expanded.

Here's some kind of weird go python js mashup of what I mean:

def async fetchAndExpand(pkg):
  apk = fetchPackage(pkg)
  files = ExpandApk(apk)
  yield files

def installPackages(packages):
  todos = make([]todo, 0, len(packages))
  for i, pkg in packages:
    go func():
      todo[i] = fetchAndExpand(pkg)

  for todo in todos:
    files = await todo
    installFiles(files)
@jonjohnsonjr jonjohnsonjr changed the title Parallelize fetch and expand Parallelize fetch and expand phase across APKs Jul 1, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant