Regression testing notes

Stand-in document for integration of testing framework

The purpose of this document is as a "temporary" repository of information arising from manual regression and integration testing that doesn't appear in commit messages. In the long term, we envision integrating a test framework into the BrainGrid provenance facility.

Comparison of summer 2015 refactor-stable-cuda branch with 12/2013

The problem is the inhibitory neurons index specified in the test-medium.xml. The size of the network is 30x30, so the neuron index must be between 0 and 799. Indexes of 901 and 920 are out of bounds. Using these wrong index values, LIFModel::generateNeuronTypeMap function overwrote wrong values to spikeCount of neuron of index number 16.

Solution: remove 901 and 920 from the inhibitory neurons list below.

<!-- 0-indexed positions of inhibitory neurons in the list -->
<I>33 66 111 159 200 299 349 380 411 555 590 655 701 788 810 840 901 920</I>

Michael StiberAug 12

OK, now that we have this resolved, there are still numerical differences between the old and new code. Some possible causes:

1. Different seeds for the random number generator.
2. Compiler differences that cause intermediate mathematical results to be truncated to 64 bits from 81 (but this, at first thought, doesn't seem to be something that would cause so large of differences).
3. The bad test file, while not causing the old code to crash, caused errors (I'm skeptical).
4. Bugs.

Michael StiberAug 26

First comparison: old code, old run results vs. old code, new run results.

There was a single neuron radius history output difference -- neuron 18 (Matlab index 19). All other radii were identical. Tentatively, we've concluded that this was a result of the broken test-medium.xml parameter file used in 2013. Still need to compare a new run of the old code using the old test-medium.xml, to see if that duplicates the old result.

Michael StiberAug 26

Second comparison: old code, new run vs. new code, new run.

Radius history identical for both simulations, except for endogenously active neurons. So, we should examine in more detail the code for endogenously active neurons to see if any bug was introduced. Largest differences in radii were around 6e-3.

Fumitaka KawasakiEdited Sep 03

I confirmed that using the 12/16 2013 code and modified:

Note: The difference of the results were made after the commit on Aug 8, 2014, Implemented external stimulus input. There are two reasons that caused the difference.

1. Changed the default precision from double to single (BGTypes.h).

2. Changed the following code (LIFSingleThreadModel.cpp, cleanupSim()):

        for (int i = 0; i < neurons.totalSpikeCount[neuron_index]; i++) {
            DEBUG_MID (cout << i << " ";);
            int idx1 = static_cast<int>( static_cast<double>( pSpikes[i] ) * sim_info.deltaT );
            //int idx1 = pSpikes[i] * sim_info.deltaT;
            m_conns->burstinessHist[idx1] = m_conns->burstinessHist[idx1] + 1.0;
            int idx2 = static_cast<int>( static_cast<double>( pSpikes[i] ) * sim_info.deltaT * 100 );
            //int idx2 = pSpikes[i] * sim_info.deltaT * 100;
            m_conns->spikesHistory[idx2] = m_conns->spikesHistory[idx2] + 1.0;
        }
   

Note: Single precision (float) gives you 23 bits of significand, 8 bits of exponent, and 1 sign bit. Double precision (double) gives you 52 bits of significand, 11 bits of exponent, and 1 sign bit. Therefore, single precision can only handle 2^23 = 8,388,608 simulation steps or 8 epochs (1 epoch = 100s, 1 simulation step = 0.1ms). This change only affects when the default precision is single.

3. Run w/ test-medium.xml
4. Got the identical results of the r-s-c version

Fumitaka KawasakiSep 03

Note: Cuda version was not available until spring in 2014, so I can only validate on cpu version.