Data and/or data copy aliases

[devreal]

Description

Currently there is a fixed mapping between data and data copies. Data copies cannot be used in multiple data. However, there can be instances where a data contains multiple pieces of information and we only care about one part of it. In that case it would be helpful to be able to describe a data as being an alias for another data, potentially with a different view onto that data.

Example: a sparse tensor contains both values and sparsity information in a single blob of data. We may want to synchronize the sparsity information but not the values (or vice versa). Yes we could allocate two different blobs of data but that incurs additional costs.

We cannot simply mark the alias copy as non-owning and point to the memory we want to alias because that memory might be evicted and would then no longer be valid for the alias.

Describe the solution you'd like

Be able to mark a data or data copy as being an alias for a different data or data copy. That way, whenever the alias is used PaRSEC knows where to get the actual data, e.g., if the aliased data has been evicted. I'm not sure whether the data or the data copy is the right entity to alias but I'm leaning towards the data copy.

Describe alternatives you've considered

The application can schedule transfers in and out of the device manually (using cudaMemcpyAsync on the compute stream) but that is more a crutch and mixes compute and transfers on the same stream.

Additional context

Likely relevant for sparsity in TTG.