⚡️Optimized Transformer

`highway` fcnn head should be used in order to prevent rank collapse
carefree0910 · Mar 20, 2021 · 00bd2c4 · 00bd2c4
1 parent 969a6c8
commit 00bd2c4
Showing 1 changed file with 1 addition and 1 deletion.
diff --git a/cflearn/models/transformer.py b/cflearn/models/transformer.py
@@ -2,7 +2,7 @@
 
 
 @ModelBase.register("transformer")
-@ModelBase.register_pipe("transformer", head="linear")
+@ModelBase.register_pipe("transformer", head="fcnn", head_config="highway")
 class Transformer(ModelBase):
     pass