event array format

[biphasic]

Currently, arrays of events are in NxD format and all floats. We have a separate variable ordering that decides which column encodes x,y,t,p. This provides maximum flexibility, but is not super user-friendly. @neworderofjamie made me rethink again the options to encode event data. Here is what I have until now:

1. Continue as is. Some transformations that work on x and y have to be rounded/truncated to integers. Floats take up a lot of memory, but events are already very space-efficient when it comes to encoding information so in my opinion not too big of an issue. The ordering for each dataset will be different, which is not very user-friendly. Could be somewhat mitigated by deciding a common ordering for all datasets.
2. Unsigned integer event arrays. Even though it makes sense when thinking about the sensor, on second thought not to allow negative timestamps or polarities is prone to underflows (e.g. jittering or offsetting timestamps). The gain in memory efficiency is not worth it imo.
3. Structured arrays. This would be good choice to accomodate the different precisions needed for xytp and also provide names for each column (e.g. to be able to call events['x'] instead of events[:, x_index]). The major downside is that structured arrays cannot be directly converted to tensors, so using a PyTorch dataloader on raw events would throw an exception. Can be mitigated by introducing a Tonic ToTensor() transform that convert structured_array -> ndarray -> tensor, but the reason for it is not obvious for a new user

I would greatly appreciate other opinions @aMarcireau @Jegp @neworderofjamie @eneftci.
Some considerations: bear in mind that while some people use the raw event ndarrays, others use event tensors and yet others create frame representations. Also event dataset encoding is very heterogenous in terms of ordering and timestamp resolution especially.

[Jegp]

It may be worthwhile to map out what concerns Tonic is trying to optimize towards; memory, processing, usability, ...?

I don't have hard data on it, but my gut feeling agrees with you @biphasic in that the memory efficiency is negligent when moving towards byte or uint arrays. However, if there is a need to optimize towards memory, I can't see a reason why the raw event-based data couldn't be encoded in byte tensors.

During processing, the situation will naturally change. One simple heuristic to apply would be to use the least common denominator, in that any filter can choose to "augment" the datatype. If a filter receives a float tensor, it shouldn't truncate it (unless it's a part of its purpose), but it's always easy to increase it.

I realize this won't provide any solid answer, but I have a hard time seeing how else one would go about this - without overengineering some fancy OOP hierarchy of course :-)

[neworderofjamie]

1. Totally agree memory efficiency isn't the biggest concern here.
2. I don't think all integer arrays would be an improvement as integers aren't great for representing time, especially from sensors with variable temporal precision.

Unsurprising, I still think structured arrays would be the nicest solution - no idea about the issues of converting to tensors though 🤷‍♂️ i feel having integer coordinates would be nicer for building frames too

[eneftci]

I have a different view about this.

• Firstly: Memory is the first resource that runs out when scaling. Given that most consumer GPUs are limited to 11GB now, this is a major bottleneck. The representation of event streams is quite minor for now, as their naturally sparse structure does indeed save a lot of space regardless of the datatype. However, once we move into a dense format, the datatype can play a decisive role for long sequences and large batch sizes.
• Secondly and future looking: GPUs cannot handle sparse data structures well, so we generally convert into a dense format (see above). However, future ML accelerators are going to be better at this (e.g. Graphcore IPU, neuromorphic chips). These processors could also perform integer operations and get a speedup from it. There is no reason to believe tonic could not be used to drive IPUs or neuromorphic chips.

I think Dataset should output the raw unsigned addresses and relevant transforms are applied to convert them to the right data type. This way there is an unambiguous mapping between the address of the event and its representation in PyTorch.

[neworderofjamie]

On the GPU, memory is indeed one of the biggest constraints but the data structures produced by tonic are, at least initially, going to be sitting in host memory which is less of a bottleneck so having a format which is numpy friendly for CPU pre-processing/easy to convert to whatever format is being used on GPU seems more important to me

[eneftci]

I should also add that a third bullet point in favor of using integers addresses: A float representation does not allow them to be used as indices. Using as indices is useful for faster conversion from sparse representations (event streams) to dense representations (e.g. count frames). I've compared with torchneuromorphic on this and torchneuromorphic is 6x faster on a nmnist batch creation, possibly by virtue of how the frames are created. See attached script (which needs torchneuromorphic installed)

from torchneuromorphic.nmnist.nmnist_dataloaders import *
from torchneuromorphic.utils import plot_frames_imshow
import torchneuromorphic.transforms as transforms
import time

root = 'data/nmnist/n_mnist.hdf5'
batch_size = 128
n_timebins = 1000
dt = 1000


ds = [1,1] 
low_crop = [0,0]
high_crop = [32,32]
size = [2, np.ceil((high_crop[0]-low_crop[0])/ds[0]).astype('int'), np.ceil((high_crop[1]-low_crop[1])/ds[1]).astype('int')]

print(size)
transform_train = transforms.Compose([
     transforms.CropDims(low_crop,high_crop,[2,3]),
     transforms.Downsample(factor=[dt,1,ds[0],ds[1]]),
     transforms.ToCountFrame(T = n_timebins, size = size),
     transforms.ToTensor()
])



train_d = NMNISTDataset(root,
                        train=True,
                        transform = transform_train, 
                        chunk_size = n_timebins)

train_dl = torch.utils.data.DataLoader(train_d, batch_size=batch_size, collate_fn = list)

it_t = iter(train_dl)


tic = time.time()
for i in range(10):
    next(it_t)

index_based_framing = time.time() - tic


import tonic
import tonic.transforms as transforms
from torch.utils.data import DataLoader

transform = transforms.Compose([transforms.ToFrame(time_window=1000),
                                transforms.ToFloat32()])

trainset = tonic.datasets.NMNIST(save_to='./data',
                                train=True,
                                transform=transform)

trainloader = DataLoader(trainset, shuffle=True, batch_size=batch_size, collate_fn = list)


it_t = iter(trainloader)

tic = time.time()
for i in range(10):
    next(it_t)

array_based_framing = time.time() - tic

print ('torchneuromorphic: {0:1.3}s | tonic: {1:1.3}s'.format(index_based_framing/10, array_based_framing/10))

which returns:

torchneuromorphic: 0.548s | tonic: 3.35s

per iteration. The same framing strategy can be done in tonic as well of course but would require casting to integer, and then back to float.

I can understand the appeal for time in float type, but it is not natural for addresses. So I guess I would vote for option 3 in @biphasic 's message.