Skip to content

Commit

Permalink
use kaiming initialization for conv layers (#98)
Browse files Browse the repository at this point in the history
use kaiming initialization for convolutional layers in dacapo models.
This is important for our standard UNets since the default
initialization is pretty terrible when following convolutions with
ReLU's.
might want to move this into individual architectures if you are playing
around with non ReLU activations
  • Loading branch information
rhoadesScholar authored Feb 15, 2024
2 parents 4fdc297 + f9a58bd commit 0e834d0
Showing 1 changed file with 7 additions and 0 deletions.
7 changes: 7 additions & 0 deletions dacapo/experiments/model.py
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,13 @@ def __init__(
)
self.eval_activation = eval_activation

# UPDATE WEIGHT INITIALIZATION TO USE KAIMING
# TODO: put this somewhere better, there might be
# conv layers that aren't follwed by relus?
for _name, layer in self.named_modules():
if isinstance(layer, torch.nn.modules.conv._ConvNd):
torch.nn.init.kaiming_normal_(layer.weight, nonlinearity="relu")

def forward(self, x):
result = self.chain(x)
if not self.training and self.eval_activation is not None:
Expand Down

0 comments on commit 0e834d0

Please sign in to comment.