There is some conditions that categorical focal loss does not improve from start

[statcom]

This may not be a problem in your code but in the algorithm itself. The loss function (categorical focal loss, CFL) worked well in some model/data but did not converge at all in other cases. For example, I tried CFL with ResNet50 for bacterial detection which didn't converge at all with 26% train accuracy for 4 classes while categorical cross entropy didn't have this problem with 90% test accuracy. But CFL for 1D-CNN with the same data converged well with 99% test accuracy.