Code to convert domain-specific BERT

[maevezhw]

Hi! I read your paper and am really interested to apply it in ClimateBERT, a domain-specific BERT that is built on DistilRoBERTa. When I want to convert the model, it convert the Roberta model that is the base for ClimateBERT, with type(model) like this
<class 'lsg_converter.roberta.modeling_lsg_roberta.LSGRobertaForMaskedLM'>

Because the model is Roberta converted, I'm confused if it is still a ClimateBERT model or not. I know that you had modify another domain specific BERT like LegalBERT using LSG attention, so I'm really curious about the code. I hope you can give me the code or explain it to me clearly. Thank you so much!