---

a stream-based file storage solution for machine learning datasets.

Today, machine learning datasets are abundantly availabe on the internet, while coming in a variety of formats( e.g., pandas dataframes, CSV files, numpy arrays, excel sheets, h5py and many more), which makes generic dataset processing complex. Luckily, almost all recent libraries provide a file-like interface for loading and storing datasets as binary streams, which is also the common ground DataStack builds upon. In DataStack datasets are stored as plain binary streams and loaded via custom iterator implementations specific for each file type. Thereby, the storage itself is completely independent from the file-type. The binary streams can be even lazyily loaded, given that the iterator supports it. The H5Py file format for instance supports this out of the box.

Another important feature of DataStack is its ability to stack iterators. Having a dataset iterator as the foundation, custom higher level iterators like iterator views that allow for arbitrary dataset splits and combined iterators that join dataset splits, can be stacked on top. Higher order iterators in other research projects adopting DataStack already comprise more sophisticated iterators like feature encoding iterators and target class mapping iterators.

So how does DataStack fit into the machine learning engineering work flow? While access to training data is not a limit anymore, integrating datasets into machine learning work flows still requires time-consuming manual preparation. Switching from one project or research paper to another, machine learning engineers and researchers often start from scratch integrating the same datasets over and over again. DataStack offers a solution for integrating these datasets by providing stable interfaces for data access that machine learning algorithms can work against. Having those interfaces in place, allows to reuse datasets and replicate results more easily.

DataStack offers the following key modules:

• Dataset Retrieval: Datasets can be retrieved via the HTTPRetriever. If a custom retriever is needed, e.g., for a custom database, only the Retriever interface needs to be implemented.

• Dataset Storage: DataStack comes with a FileStorageConnector for storing and loading datasets from disk using a predefined dataset identifier. By implementing the StorageConnector interface, any other custom storage solution, e.g. a MongoDB, can be supported. Notably, every dataset is stored as a StreamedResource, which is a wrapper around the Python's IOBase. Therefore, the respective DatasetStorage does not require any knowledge of the encoded data. This is why, the storage is not limited to any specific file-type. Additionally, when accessing the file storage, only a file descriptor to that file is created, offering lazy loading for iterators.

• Iterator: Datastack provides an iterator interface and a few implementations to iterate through datasets. An iterator takes a StreamedResource containing a binarized dataset and provides an iteration routine, customized to to the original filetype of the dataset. For instance, a binary Pytorch Tensor stream needs a different iteration implemenation than a CSV stream. Note, that the StreamedResource only provides a file descriptor to the stream. If this stream is stored on disk, the StreamedResource does not automatically load the stream into memory. This gives the opportunity to lazily load samples with e.g., h5py file streams.

Install

There are two options to install DataStack, the easiest way is to install it from the pip repository:

pip install datastack

For the latest version, one can directly install it from source by cd into the root folder and then running

pip install src/

Usage

The following tutorial is also available as a Jupyter notebook here.

The existing examplary MNIST dataset can be loaded and iterated as follows:

from data_stack.mnist.factory import MNISTFactory
from data_stack.io.storage_connectors import FileStorageConnector

Create the file storage connector to store and retrieve the dataset.

dataset_path = "./datasets/"   # specify dataset path
storage_connector = FileStorageConnector(root_path=dataset_path)

We instantiate the MNIST factory by passing the storage_connector. If the dataset has been already downloaded it is being only loaded from the local disc.

In this example here, the MNIST dataset is not present locally and is thus being donwloaded.

mnist_factory = MNISTFactory(storage_connector)
mnist_iterator, _ = mnist_factory.get_dataset_iterator(config={"split": "train"})

The following files are stored by the storage_connector:

datasets/
└── mnist
    ├── preprocessed
    │   ├── test
    │   │   ├── samples.pt
    │   │   └── targets.pt
    │   └── train
    │       ├── samples.pt
    │       └── targets.pt
    └── raw
        ├── labels_train.gz
        ├── samples_test.gz
        ├── samples_train.gz
        └── targets.gz

Now, we can directly index elements or easily iterate through the dataset.

img, target, tag = mnist_iterator[0]
print(f"Sample has target {target} and tag {tag} and has a shape of {img.shape}")

# visualize the sample
from matplotlib import pyplot as plt
plt.imshow(img)
plt.show()

NOTE: This library is still under heavy development. It's most likely not free of bugs and interfaces can still change.

To implement a new dataset, one has to implement 3 classes:

• DatasetFactory: Retrieves, prepares, stores and loads the dataset using a Retriever and Preprocessor implementation and a StorageConnector.
• Preprocessing: Datasets often come compressed, split up over many files or in who knows what structure. Therefore, for each dataset we need a Preprocessing class that transforms the datasets into a StreamedResource.
• Iterator: Provides the iteration implementation on top of the binary stream StreamedResource

DataStack provides a examplary MNIST implementation.

Copyright

Copyright (c) 2020 Max Lübbering For license see: https://github.com/le1nux/datastack/blob/master/LICENSE