Download artifacts concurrently

[yaroot]

Proposal

Download artifacts concurrently

Use-cases

I put all the jars separately in job artifacts array instead of one giant tarball.

For JVM programs the dependencies accounts for majority of the resulting program and does not change often between builds. This way each build only results in tiny jars for the actual code change, one would not need to delete any build ever.

But the job initialization is slower because now you have a lot of small files to download. I understand this is very specific to jvm (the only case I could think of), and not affecting most use cases, I could have a go if it's deemed worthy.

// example artifacts array

[
  {
    "destination": "local/lib/HikariCP-5.0.0.jar",
    "mode": "file",
    "options": {
      "checksum": "sha256:cfab85c04fcb590ae4cfdbe1b1e87cfd7277c4ba378229422d8e8bce989a97d6"
    },
    "source": "https://artifacts.example.com/cf/ab/85c04fcb590ae4cfdbe1b1e87cfd7277c4ba378229422d8e8bce989a97d6"
  },
  {
    "destination": "local/lib/case-insensitive_2.13-1.1.4.jar",
    "mode": "file",
    "options": {
      "checksum": "sha256:3b0eae646754430caf86c38e9583c9e3b92c227e14f72a840f0629fe84445c68"
    },
    "source": "https://artifacts.example.com/3b/0e/ae646754430caf86c38e9583c9e3b92c227e14f72a840f0629fe84445c68"
  },
  {
    "destination": "local/lib/cats-core_2.13-2.6.1.jar",
    "mode": "file",
    "options": {
      "checksum": "sha256:263d5982aa88702a3c8024d792f9dcc237cd0c6da3cca610c1fcc107c337b49a"
    },
    "source": "https://artifacts.example.com/26/3d/5982aa88702a3c8024d792f9dcc237cd0c6da3cca610c1fcc107c337b49a"
  },
  ...
]

// task command would be `java -cp local/lib org.example.Foo`

[lgfa29]

Thanks @yaroot.

After a quick look, it seems like we could parallelize this loop with Goroutines.

I've placed it in our backlog for us to investigate further.

[schmichael]

Just a note to the implementer: we should cap the number of concurrent downloads (3?) to ensure a single allocation can't overwhelm the network or disk downloading a large number of artifacts. A client configuration parameter for this would be appropriate, but I don't think necessary.

[yaroot]

Per task group configuration would be nice, it's very dependent on hardware and types of artifacts.

[danishprakash]

mind If I pick this up?

[gowthamgts]

@danishprakash I'm almost done with the implementation. testing it right now. you wanna take over?

[danishprakash]

@gowthamgts oh no, it's fine. I thought of picking it up since there wasn't much activity on this issue but please continue since you've already started with the work.

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.