Architecture
The transfer2go system consists of 3 types of agents, one is a registration agent where client place request (main agent), another is source agents (where data reside for transfer) and finally a destination agent where transfer will go. The system is fully de-centralized, i.e. any agent can have any of these roles. Of course for bookkeeping purposes it is desired to assign specific agent to perform a role of main one. It will only accept requests from clients and assign a transfer from source to destination. The transfer itself can be done either in a push mode (the source push data to destination) or in a pull mode (the destination pull data from the sources).
Here is a data and request flow within pull model. The client place a request to registration agent (main agent). It finds out where data reside by placing request to source agents, and place request into persistent store and its queue. Then it sends request to a destination agent. The destination agent place it into its transfer queue and request data download from a source agent. The source agent can either reply 204 No Content (when data is not yet ready, e.g. reside on tape) or 200 OK (when data is ready). If former happens, the request is put on hold at destination agent. When data is ready it fetches it via HTTP, place in its own disk space and record its meta-data in its Meta-data DB. Finally it sends request to registration agent that the process is finished. The registration agent update its request DB and proceed with handling other requests.
this design is similar to pull model but registration agent send request to source agent instead of destination one. In this scenario source agent initiate a request via HTTP POST (push) to destination one. Data are uploaded to destination agent.
Very first destination agent register a transfer request to the main agent. The main agent has the web interface to approve and delete these requests. After approving the request main agent sends an acknowledge message to source agent to push the data to the destination agent. Before sending the acknowledge message to source agent main agent also verifies that weather destination agent is up or not.
Very first the client will register a request to the main agent. It may ask to transfer the entire dataset, partial data set or may be some specific files/blocks. After, getting the request the main agent will fetch the catalog of each registered agent and will filter out the agents which contain the requested data. After filtering out the agents the main agent will fetch the status of each of these agents. This step will provide the details of bandwidth, network latency and load on agents. After that, the main agent will compare this data with the historical data using linear regression with Multiple Variables to select the agent. It will then send an acknowledge message to the destination to pull the data from source agent.