Credits to Ajay Arasanipalai for his help in this project and to Javier Ideami, founder of losslandscape.com for creating the stunning visualizations
Check out Javier's talk at the Synthetic Intelligence Forum on more details regarding these observations.
Note - For unlabelled images, the order is ReLU -> Mish -> Swish (from left to right)