[TransferEngine] Refactor code to hide transport logics from user APIs

[alogfans]

This PR refactors the code of TransferEngine, so that the following methods are implemented by TransferEngine, i.e., users do not need to obtain Transport pointers beforing calling these methods. This also enables using multiple transports (e.g., rdma and shm) in the same transfer batch.

    BatchID allocateBatchID(size_t batch_size);
    int freeBatchID(BatchID batch_id);
    int submitTransfer(BatchID batch_id,
                       const std::vector<TransferRequest> &entries) ;
    int getTransferStatus(BatchID batch_id, size_t task_id,
                          TransferStatus &status);

We also extract metadata drivers (etcd/redis/http) seperately. User can fill the metadata server parameters with etcd://, redis:// or http:// respectively.

This patch introduces major modification to TransferEngine, and still WIP.

Update on 2024/12/30: We add a patch to extract topology logics outside rdma transport, so that it can be used for other transports/medias

[doujiang24]

Awesome, glad to see this happen.
As we talked in chat, we could add more kinds of devices and protocols base on this PR in the feature.

[doujiang24]

add break after this line would be better?

[doujiang24]

cool, lgtm~

[stmatengss]

LGTM

[ShangmingCai]

I have run some verification tests. Except for metadata_server will be configured to etcd by default when no prefix is given, other changes LGTM.

I will propose a PR to modify the related part in the vllm integration doc later.