-
Notifications
You must be signed in to change notification settings - Fork 3
/
4_Database.tex
117 lines (103 loc) · 7.33 KB
/
4_Database.tex
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
\section{The Dataset: NE15-MNIST}
\label{sec:data}
%Experiment setup/ collection method/ properties of each class/ etc.
The name of the first proposed dataset in the benchmarking system is NE15-MNIST which stands for Neuromorphic Engineering 2015 on MNIST.
The original MNIST dataset is downloaded from the website\footnote{http://yann.lecun.com/exdb/mnist/} of THE MNIST DATABASE of handwritten digits~\citep{lecun1998gradient}.
The NE15-MNIST is converted into a spiking version of the original dataset consisting of four subsets which were generated for different purposes:
\begin{itemize}
\item \textit{Poissonian}
to benchmarking existing methods of rate-based spiking models.
\item \textit{FoCal}
to promote the study of spatio-temporal algorithms applied to recognition tasks using few input spikes.
\item \textit{DVS recorded flashing input}
to encourage research on fast recognition methods which are found in the primate visual pathway.
\item \textit{DVS recorded moving input}
to trigger the study of algorithms targeting on continuous input from real-world sensors and to implement them on mobile neuromorphic robots.
\end{itemize}
The dataset can be found in the GitHub repository at: https://github.com/qian-liu/benchmarking.
\subsection{File~Formats}
Two file formats are supported in the dataset: jAER format~\citep{delbruck2008frame} (.dat or .aedat), and binary file in NumPy .npy format.
The address event representation (AER) interface has been widely used in neuromorphic systems, especially for vision sensors.
The spikes are encoded as time events with corresponding addresses to convey information.
The spikes in jAER format, both recorded from a DVS retina and artificially generated, can be displayed in jAER software.
Figure~\ref{Fig:jaer} is a snapshot of the software displaying an .aedat file which is recorded by a DVS retina~\citep{serrano2013128}.
The resolution of the DVS recorded data is 128$\times$128.
%, while the original MNIST data is 28$\times$28.
The other format of spikes used is a list of spike source arrays in PyNN~\citep{davison2008pynn}, a description language for building spiking neuronal network models.
Python code for converting one file format to and from the other is also provided.
\begin{figure*}[hbt]
\centering
\subfloat[DVS recording.]{
\label{Fig:jaer}
\includegraphics[width=0.28\textwidth]{images/dvs-128.pdf}
}
\subfloat[Poisson spike trains.]{
\label{Fig:poisson}
\includegraphics[width=0.28\textwidth]{images/zero-28-2.pdf}
}\\
\subfloat[The raster plot of the Poisson spike trains.]{
\label{Fig:raster}
\includegraphics[width=0.65\textwidth]{images/zero.pdf}
}
\caption{
Snapshots of jAER software playing spike presented videos.
The same image of digit ``0'' is transformed to spikes by DVS recording and the Poissonian generation respectively.
A raster plot of the Poissonian spike trains is also provided.}
\label{fig:zero}
\end{figure*}
\subsection{Data Description}
\subsubsection{Poissonian}
In the cortex, the timing of spikes is highly irregular~\citep{squire1998findings}.
It can be interpreted that the inter-spike interval reflects a random process driven by the instantaneous firing rate.
If the generation of each spike is assumed to be independent of all the other spikes, the spike train is seen as a Poisson process.
The spiking rate can be estimated by averaging the pooled responses of the neurons.
As stated above, rate coding is exclusively used in presenting images with spikes.
The spiking rate of each neuron is in accordance with its corresponding pixel intensity.
Instead of providing exact spike arrays, we share the Python code for generating the spikes.
Every recognition system may require different spiking rates and various lengths of their durations.
The generated Poissonian spikes can be in the formats of both jAER and PyNN spike source array.
Thus, it is easy to visualise the digits and also to build spiking neural networks.
Because different simulators generate random Poissonian spike trains with various mechanisms, languages and codes, using the same dataset enables performance evaluation on different simulators without the interference created by non-unified input.
The same digit displayed in Fig.~\ref{Fig:jaer} is converted to Poissonian spike trains, see Fig.~\ref{Fig:poisson}.
The raster plot can be found in Fig.~\ref{Fig:raster}, indicating the intensities of the pixels.
\subsubsection{Rank-Order-Encoding}
\input{4-2-2_Rank-ordered_encoding}
\subsubsection{DVS Sensor Output with Flashing Input}
\label{subsec_flash}
The purpose of including the subset of DVS recorded flashing digits is to promote the application of Rank-Order-Coding to DVS output, and accelerate the fast on-set recognition by using just the beginning part of spike trains within less than 30~ms.
Each digit and a blank image was shown alternately and each display lasted one second.
The digits were displayed on an LCD monitor in front of the DVS retina~\citep{serrano2013128} and were placed in the centre of the visual field of the camera.
% Each recording was cut into 10 sub sections.
Since there are two polarities of the spikes: 'ON' indicates the increase of the intensity while 'OFF' reflects the opposite, there are 'ON' and 'OFF' flashing recordings respectively per digit.
In Fig.~\ref{fig:flash}, the burstiness of the spikes is illustrated where most of the spikes occur in a 30~ms slot.
In total, the subset of the database contains 2$\times$$60,000$ recordings for training and 2$\times$$10,000$ for testing.
% Due to the size limit of online repositories, only the third of every sequence of flashes is published.
\begin{figure*}[b!]
\centering
\subfloat[The raster plot of the spike trains of a flashing digit.]{
\label{fig:flash_all}
\includegraphics[width=0.55\textwidth]{flash_full.pdf}
}
\\
\subfloat[The same spike trains ordered by the spike sequence.]{
\label{fig:flash_a}
\includegraphics[width=0.55\textwidth]{flash_full_order.pdf}
}
% \\
% \subfloat[Bursty spikes appearing in a 30~ms slot.]{
% \label{fig:flash_b}
% \includegraphics[width=0.48\textwidth]{flash_100.png}
% }
\caption{DVS sensor with flashing input.
Blue is used for `ON' events and green for `OFF' events.
(a) The raster plot.
(b) The bursty nature of the resulting spikes is illustrated, where most of the spikes occur in a 30~ms time slot.}
\label{fig:flash}
\end{figure*}
% \subsubsection{DVS Sensor Output with Oscillating Input}
\subsubsection{DVS Sensor Output with Moving Input}
In order to address the problems of position- and scale- invariance, a subset of DVS recorded moving digits is presented.
MNIST digits were scaled to three different sizes, by using smooth interpolation algorithms to increase their size from the original 28x28 pixel size, and displayed on the monitor with slow motion.
The same DVS~\citep{serrano2013128} used in Section~\ref{subsec_flash} captured the movements of the digits and generated spike trains for each pixel of its 128$\times$128 resolution.
A total of 30,000 recordings were made: 10 digits, at 3 different scales, 1000 different handwritten samples for each.
% The subset is available at the website\footnote {http://imse-cnm.csic.es/caviar/MNIST\_DVS}.