clean up v1.0.3

README.md

@@ -12,6 +12,7 @@ Some of the most commonly used neural networks in neuroscience research are incl
 
 ## Table of contents
 - [Acknowledgements](#acknowledgements)
+- [Change Logs](#change-logs)
 - [Install](#install)
     - [Install Using pip](#install-using-pip)
     - [Install From GitHub](#install-from-github)
@@ -23,11 +24,14 @@ Some of the most commonly used neural networks in neuroscience research are incl
     - [Random Input](#random-input)
 - [Criterion](#criterion)
     - [RNNLoss](#rnnloss)
-- [Change Logs](#change-logs)
 - [Others](#others)
 
 ## Acknowledgements
-Immense thanks to Christopher J. Cueva for his mentorship in developing this project. This project can't be done without his invaluable help.
+Immense thanks to [Christopher J. Cueva](https://www.metaconscious.org/author/chris-cueva/) for his mentorship in developing this project. This project can't be done without his invaluable help.
+
+## Change Logs
+- [v1.0.3 [stable]](https://github.com/zhaozewang/NN4Neurosci/blob/main/docs/change_logs/v1.0.3.md)
+- [v1.1.0 [newest]](https://github.com/zhaozewang/NN4Neurosci/blob/main/docs/change_logs/v1.1.0.md)
 
 ## Install
 ### Install using pip
@@ -36,60 +40,50 @@ pip install nn4n
 ```
 
 ### Install from GitHub
+#### Clone the repository
 ```
 git clone https://github.com/zhaozewang/NN4Neurosci.git
 ```
 #### Navigate to the NN4Neurosci directory
 ```
 cd NN4Neurosci/
-python setup.py install
-```
-#### Install
-```
-cd NN4Neurosci/
-pip install .
+pip install -e .
 ```
 
 
 ## Model
 ### CTRNN
-The implementation of standard continuous-time RNN (CTRNN). This implementation supports enforcing sparsity constraint (i.e. preventing new synapses from being created) and E/I constraints (i.e. enforcing Dale's law). </br>
-
+The implementation of standard continuous-time RNN (CTRNN). This implementation supports enforcing sparsity constraints (i.e., preventing new synapses from being created) and E/I constraints (i.e., enforcing Dale's law). This ensures that the gradient descent will update synapses with biological plausible constraints. <br>
 - [Documentation](https://github.com/zhaozewang/NN4Neurosci/blob/main/docs/model/CTRNN/index.md)
 - [Examples](https://github.com/zhaozewang/NN4Neurosci/blob/main/examples/CTRNN.ipynb)
+<p align="center"><img src="https://github.com/zhaozewang/NN4Neurosci/blob/main/docs/images/basics/EIRNN_structure.png" width="400"></p>
+
 
 ## Structure
-The detailed structure (e.g. whether its modular or hierarchical etc.) of any standard 3-layer RNN (as shown in figure above) can be specified using masks in our `model` module implementation. Easy implementations of a few RNN structures is included in the `structure` module.
+In CTRNN implementation, the hidden layer structure can be easily controlled by specifying sparsity masks and E/I masks. We put all RNN update logic in the `model` module and all structure-related logic in the `structure` module to streamline the implementation process. <br>
+We also emphasize on the structure more as it is often more informative to the underlying biological mechanisms. For instance, we might require different module sizes, or we require a multi-module network with E/I constraints; the implementation might be verbose and error-prone. Our following implementation will allow you to easily achieve these goals by simply specifying a few parameters. <br>
 - [Documentation](https://github.com/zhaozewang/NN4Neurosci/blob/main/docs/structure/index.md)
 
-### Multi-Area
-The HiddenLayer of a RNN is often defined using a connectivity matrix, depicting a somewhat 'random' connectivity between neurons. The connectivity matrix is often designed to imitate the connectivity of a certain brain area or a few brain areas. When modeling a single brain area, the connectivity matrix is often a fully connected matrix. </br>
-However, to model multiple brain areas, it would be more reasonable to use a connectivity matrix with multiple areas. In each areas is densely connected within itself and sparsely connected between areas. The `MultiArea` class in the `structure` module is designed to implement such a connectivity matrix. </br>
-
+#### Multi-Area
+The following implementation groups neurons into areas depending on your specification. Areas can have different sizes and different connectivities to other areas. <br>
 - [Examples](https://github.com/zhaozewang/NN4Neurosci/blob/main/examples/MultiArea.ipynb)
+<p align="center"><img src="https://github.com/zhaozewang/NN4Neurosci/blob/main/docs/images/basics/Multi_Area.png" width="400"></p>
 
-### Multi-Area with E/I constraints
-On top of modeling brain with multi-area hidden layer, another critical constraint would be the Dale's law, as proposed in the paper [Training Excitatory-Inhibitory Recurrent Neural Networks for Cognitive Tasks: A Simple and Flexible Framework](https://doi.org/10.1371/journal.pcbi.1004792) by Song et al. 2016. The `MultiAreaEI` class in the `structure` module is designed to implement such a connectivity matrix. </br>
-This class allows for a much easier implementation of the E/I constraints particularly when there are multiple areas in the hidden layer. It provides flexible control over the excitatory-excitatory, excitatory-inhibitory, inhibitory-excitatory, and inhibitory-inhibitory connections on top of the basic `MultiArea` features. </br>
-
+#### Multi-Area with E/I constraints
+On top of modeling brain with multi-area hidden layer, another critical constraint would be Dale's law, as proposed in the paper [Training Excitatory-Inhibitory Recurrent Neural Networks for Cognitive Tasks: A Simple and Flexible Framework](https://doi.org/10.1371/journal.pcbi.1004792) by Song et al. 2016. The `MultiAreaEI` class in the `structure` module is designed to implement such a connectivity matrix. <br>
+Additionally, how different E/I regions connect to each other could also be tricky; we parameterized this process such that it can be controlled with only two lines of code. <br>
 - [Examples](https://github.com/zhaozewang/NN4Neurosci/blob/main/examples/MultiArea.ipynb)
+<p align="center"><img src="https://github.com/zhaozewang/NN4Neurosci/blob/main/docs/images/basics/Multi_Area_EI.png" width="400"></p>
 
-### Random Input
-Neurons' dynamic receiving input will be heavily driven by the inputting signal. Injecting signal to only part of the neuron will result in more versatile and hierarchical dynamics. See [A Versatile Hub Model For Efficient Information Propagation And Feature Selection](https://arxiv.org/abs/2307.02398) <br>
-
+#### Random Input
+Neurons' dynamic receiving input will be heavily driven by the inputting signal. Injecting signals to only part of the neuron will result in more versatile and hierarchical dynamics. See [A Versatile Hub Model For Efficient Information Propagation And Feature Selection](https://arxiv.org/abs/2307.02398). This is supported by `RandomInput` class <br>
 - Example to be added
 
 ## Criterion
-### RNNLoss
-The loss function is modularized. The `RNNLoss` class is designed in modular fashion and included the most commonly used loss functions in neuroscience research. </br>
-
+#### RNNLoss
+The loss function is modularized. The `RNNLoss` class is designed in a modular fashion and includes the most commonly used loss functions, such as Frobenius norm on connectivity, metabolic cost, reconstruction loss, etc. <br>
 - [Documentation](https://github.com/zhaozewang/NN4Neurosci/blob/main/docs/criterion/index.md)
 
-
-## Change Logs
-- [0.1.3](https://github.com/zhaozewang/NN4Neurosci/blob/main/docs/change_logs/v1.0.3.md)
-
-
 ## Others
 For similar projects:
 - [nn-brain](https://github.com/gyyang/nn-brain)

docs/change_logs/v1.0.3.md

@@ -1,12 +1,13 @@
-## Change Logs
-- Removed `ei_balance`. It will automatically balanced such that all weights add up to 1.
-- Added Multi Layer Perceptron (MLP) model.
-- Removed `allow_negative`
-- Renamed `use_dale` as `positivity_constraint`, support list definition.
-- Renamed `new_synapses` as `sparsity_constraint`.
-- Remove auto E/I balance. Potentially move it to a separate function.
+## Verson 1.0.3 Change Logs
+1. Removed `ei_balance`. It will automatically balanced such that all weights add up to 1.
+2. Added Multi Layer Perceptron (MLP) model.
+3. Removed `allow_negative`
+4. Renamed `use_dale` as `positivity_constraints`, support list definition.
+5. Renamed `new_synapses` as `sparsity_constraints`.
+6. Remove auto E/I balance. Potentially move it to a separate function.
+7. Removed `spectral_radius`, this will be redesigned in future versions.
 
 
-## TODO
-- [x] Change `use_dale` to `positivity_constraint`
-- [x] Change `new_synapses` to `sparsity_constraint`
+## TODOs
+- [x] Change `use_dale` to `positivity_constraints`
+- [x] Change `new_synapses` to `sparsity_constraints`

docs/change_logs/v1.1.0.md

@@ -0,0 +1,5 @@
+## Verson 1.1.0 Change Logs
+
+## TODOs
+- [ ] Put initialization to structures, i.e., the distribution of weights, biases, etc.
+- [ ] Add custom controls to the update speeds of different parts of the network.

docs/model/CTRNN/index.md

@@ -70,7 +70,7 @@ fr^{t+1} = f((1-\alpha) v^t + \alpha( W_{hid}^T f(v^t) + W_{in}^T u^t + b_{hid}
 ## Excitatory-Inhibitory constrained continuous-time RNN
 The implementation of CTRNN also supports Excitatory-Inhibitory constrained continuous-time RNN (EIRNN). EIRNN is proposed by H. Francis Song, Guangyu R. Yang, and Xiao-Jing Wang in [Training Excitatory-Inhibitory Recurrent Neural Networks for Cognitive Tasks: A Simple and Flexible Framework](https://doi.org/10.1371/journal.pcbi.1004792)
 
-The original [code](https://github.com/frsong/pycog) is implemented in [Theano](https://pypi.org/project/Theano/) and may be deprecated due to the unsupported Python version. Theano is no longer maintained after Jul 2020. In this repo, the PyTorch version of EIRNN is implemented. It is implicitly included in the CTRNN class and can be enabled by setting `positivity_constraint` to `True` and use appropriate masks.
+The original [code](https://github.com/frsong/pycog) is implemented in [Theano](https://pypi.org/project/Theano/) and may be deprecated due to the unsupported Python version. Theano is no longer maintained after Jul 2020. In this repo, the PyTorch version of EIRNN is implemented. It is implicitly included in the CTRNN class and can be enabled by setting `positivity_constraints` to `True` and use appropriate masks.
 
 A visual illustration of the EIRNN is shown below.
 
@@ -119,9 +119,9 @@ These parameters primarily determine the training process of the network. The `t
 These parameters primarily determine the constraints of the network. By default, the network is initialized using the most lenient constraints, i.e., no constraints being enforced.
 | Parameter                | Default       | Type                       | Description                                |	
 |:-------------------------|:-------------:|:--------------------------:|:-------------------------------------------|
-| sign                 | False         | `boolean`/`list`           | Whether to enforce Dale's law. Either a `boolean` or a `list` of three `boolean`s. If the given value is a list, from the first element to the last element, corresponds to the InputLayer, HiddenLayer, and ReadoutLayer, respectively. |
-| sparsity_constraint  | True         | `boolean`/`list`            | Whether a neuron can grow new connections. See [constraints and masks](#constraints-and-masks). If it's a list, it must have precisely three elements. Note: this must be checked even if your mask is sparse, otherwise the new connection will still be generated                  |
-| layer_masks              | `None` or `list` | `list` of `np.ndarray`               | Layer masks if `sparsity_constraint/positivity_constraint is set to true. From the first to the last, the list elements correspond to the mask for Input-Hidden, Hidden-Hidden, and Hidden-Readout weights, respectively. Each mask must have the same dimension as the corresponding weight matrix. See [constraints and masks](#constraints-and-masks) for details.                   |
+| positivity_constraints    | False         | `boolean`/`list`           | Whether to enforce Dale's law. Either a `boolean` or a `list` of three `boolean`s. If the given value is a list, from the first element to the last element, corresponds to the InputLayer, HiddenLayer, and ReadoutLayer, respectively. |
+| sparsity_constraints  | True         | `boolean`/`list`            | Whether a neuron can grow new connections. See [constraints and masks](#constraints-and-masks). If it's a list, it must have precisely three elements. Note: this must be checked even if your mask is sparse, otherwise the new connection will still be generated                  |
+| layer_masks              | `None` or `list` | `list` of `np.ndarray`               | Layer masks if `sparsity_constraints/positivity_constraints is set to true. From the first to the last, the list elements correspond to the mask for Input-Hidden, Hidden-Hidden, and Hidden-Readout weights, respectively. Each mask must have the same dimension as the corresponding weight matrix. See [constraints and masks](#constraints-and-masks) for details.                   |
 
 
 ## Parameter Specifications
@@ -147,15 +147,15 @@ then add noise for $t+2$
 ### Constraints and masks
 Constraints are enforced before each forward pass
 #### Dale's law:
-Masks (input, hidden, and output) cannot be `None` if `positivity_constraint` is `True`.<br>
+Masks (input, hidden, and output) cannot be `None` if `positivity_constraints` is `True`.<br>
 Only entry signs matter for the enforcement of Dale's law. All edges from the same neuron must be all excitatory or all inhibitory. This is enforced across training using the `relu()` and `-relu()` functions.<br>
-When `positivity_constraint` is set to true, it will automatically balance the excitatory/inhibitory such that all synaptic strengths add up to zero.
+When `positivity_constraints` is set to true, it will automatically balance the excitatory/inhibitory such that all synaptic strengths add up to zero.
 #### New synapse:
-`sparsity_constraint` defines whether a neuron can 'grow' new connections.<br>
-If plasticity is set to False, neurons cannot 'grow' new connections. A mask must be provided if `sparsity_constraint` is set to False.<br>
+`sparsity_constraints` defines whether a neuron can 'grow' new connections.<br>
+If plasticity is set to False, neurons cannot 'grow' new connections. A mask must be provided if `sparsity_constraints` is set to False.<br>
 Only zeros entries matter. All entries that correspond to a zero value in the mask will remain zero across all time.
 #### Self connections:
-Whether a neuron can connect to itself. This feature is enforced along with the `sparsity_constraint` mask. If mask is not specified but `self_connections` is set, a mask that only has zero entires on the diagonal will be generated automatically.
+Whether a neuron can connect to itself. This feature is enforced along with the `sparsity_constraints` mask. If mask is not specified but `self_connections` is set, a mask that only has zero entires on the diagonal will be generated automatically.
 
 ## Methods
 | Method                                        | Description                                |
@@ -171,8 +171,8 @@ Whether a neuron can connect to itself. This feature is enforced along with the
 - [x] Test different activation functions
 - [x] Bias when using Dale's law?
 - [ ] If the masks are not set, there need default values.
-- [x] Potentially user can choose to enforce `sparsity_constraint` or not for a specific layer
-- [x] Re-write Dale's law such that it can still work when `sparsity_constraint` is not enforced.
+- [x] Potentially user can choose to enforce `sparsity_constraints` or not for a specific layer
+- [x] Re-write Dale's law such that it can still work when `sparsity_constraints` is not enforced.
 - [x] Can InputLayer and ReadoutLayer weights be negative when Dale's law is enforced?
 - [x] Check if bias does not change when use_bias = False
 - [x] Merge hidden_bias, input_bias, readout_bias to a single parameter

docs/structure/index.md

@@ -48,7 +48,7 @@ Methods that are shared by all structures. <br>
 
 ### MultiArea
 See [Examples](https://github.com/zhaozewang/NN4Neurosci/blob/main/examples/MultiArea.ipynb) <br>
-This will generate a multi-area RNN without E/I constraints. Therefore, by default, the input/hidden/readout masks are binary masks. Use cautious when the `positivity_constraint` parameter of CTRNN is set to `True`, because it will make all neurons to be excitatory.
+This will generate a multi-area RNN without E/I constraints. Therefore, by default, the input/hidden/readout masks are binary masks. Use cautious when the `positivity_constraints` parameter of CTRNN is set to `True`, because it will make all neurons to be excitatory.
 **NOTE:** This also implicitly covers single area case. If `n_area` is set to 1. All other parameters that conflict this setting will be ignored.
 #### MultiArea Parameters
 | Parameter            | Default       | Required      | Type                      | Description                                |

examples/CTRNN.ipynb

@@ -497,13 +497,13 @@
     "    \n",
     "    # hyperparameters\n",
     "    \"tau\": 50,\n",
-    "    \"positivity_constraint\": True,\n",
+    "    \"positivity_constraintsssss\": True,\n",
     "    \"scaling\": 1.0,\n",
     "    \"dt\": 1,\n",
     "    \"activation\": \"relu\",\n",
     "    \"preact_noise\": 0.0,\n",
     "    \"postact_noise\": 0.0,\n",
-    "    \"sparsity_constraint\": False,\n",
+    "    \"sparsity_constraints\": False,\n",
     "    \"self_connections\": False,\n",
     "\n",
     "    # bias and distribution\n",