Story for Hackathon , Task #3

[MattSlm]

Task 3 (RSE selection for concurrent downloads with shared bottleneck) [Stretch^2]
Goal. Scientist picks RSEs with concurrent downloading rate targets.

Workflow:
Step 1. Scientist specifies the concurrent datasets to download and their target rates in a file:
terminal-1 pfn-1 min-rate-1
...
terminal-M pfn-M min-rate-M

Step 2: Scientist conducts scheduling:
rucio-sched-alto --alto-server= --input=
rucio-sched-alto executes the following:
foreach pfn-i, computes its set of replicas RSEs[pfn-i]
foreach combination (r1, r2, ..., rK) in RSEs[pfn-1] x .. x RSEs[pfn-K]
Invoke task 2 to check returned rate
If found a satisfying assignment, uses the assignment and starts rucio download with given RSE

[giralt]

Thanks @MattSlm. Is this the umbrella ticket we discussed to write a user friendly document explaining the complete story? If so, at a high level the idea here will be:

• The story starts from the very beginning (LHC experiment collides atoms generating gigantic datasets, see for instance https://cds.cern.ch/record/2276551)
• The datasets are kept in storage.
• The Rucio server replicates the datasets to various nodes
• Scientists download datasets by choosing from the various available replicas.
• Scientists can use ALTO to optimize the download decision

At each step, please specify the protocols that are being used (e.g., xrootd, Rucio, ALTO).

Suggest that URLS to the various protocol documents be included (xrootd, Rucio, RFCs, etc.) so a reader can dig more into them if he/she wants to learn about the details.

Please also use our hackathon example network to illustrate at each step how things get done (file upload, download, etc.):

              Rucio
      1Mbps    |     5Mbps
      25ms     |     25ms
  s1 --------- s3 ----------- s4 -- XRD3
  |            |              |
  |            | 25ms         | 50ms
  |            | 1Mbps        | 2Mbps
  |            |              |
 XRD1          s2 -- XRD2     s5 -- XRD4

Can I suggest that in the above diagram we separate the Rucio server from the Rucio client (where the scientist is located)? I think it will make it more realistic and functionally easier to understand. (Note: I have created a ticket to modify the network configuration to accommodate for this change: #31)

The document should be written so that both a non-technical person can understand the story flow and a also for a technical person to understand how the various protocols are used end-to end. So good artistic writing skills are needed there :)

The story should also be written in a compelling manner, motivating the use case and the benefits of using ALTO. Thks.

[MattSlm]

Hi Jordi, this is another one for developing the code of demo 3, namely scheduling. The story will be wrapped up under another issue.

[MattSlm]

update: just tested a few cases for demo #2 on my local env. Will push task3 code on a new branch task#3 for ALTO repo and publish the env setup document for task 3.