Import Transformers into Spark NLP 🚀

[maziyarpanahi]

Import Transformers into Spark NLP

Overview

We have extended support for HuggingFace 🤗 and TF Hub exported models since 3.1.0 to equivalent Spark NLP 🚀 annotators. Starting this release, you can easily use the saved_model feature in HuggingFace within a few lines of codes and import any BERT, DistilBERT, CamemBERT, RoBERTa, DeBERTa, XLM-RoBERTa, Longformer, BertForTokenClassification, DistilBertForTokenClassification, AlbertForTokenClassification, RoBertaForTokenClassification, DeBertaForTokenClassification, XlmRoBertaForTokenClassification, XlnetForTokenClassification, LongformerForTokenClassification, CamemBertForTokenClassification, CamemBertForSequenceClassification, CamemBertForQuestionAnswering, BertForSequenceClassification, DistilBertForSequenceClassification, AlbertForSequenceClassification, RoBertaForSequenceClassification, DeBertaForSequenceClassification, XlmRoBertaForSequenceClassification, XlnetForSequenceClassification, LongformerForSequenceClassification, AlbertForQuestionAnswering, BertForQuestionAnswering, DeBertaForQuestionAnswering, DistilBertForQuestionAnswering, LongformerForQuestionAnswering, RoBertaForQuestionAnswering, XlmRoBertaForQuestionAnswering, TapasForQuestionAnswering, Vision Transformers (ViT), HubertForCTC, SwinForImageClassification, and ConvNextForImageClassification models to Spark NLP. We will work on the remaining annotators and extend this support to the rest with each release 😊

Compatibility

Spark NLP: The equivalent annotator in Spark NLP
TF Hub: Models from TF Hub
HuggingFace: Models from HuggingFace
ONNX: Models from HuggingFace in ONNX format
Model Architecture: Which architecture is compatible with that annotator
Flags:

• Fully supported ✅
• Partially supported (requires workarounds) ✔️
• Under development ❎
• Not supported ❌

Spark NLP	TF Hub	HuggingFace	ONNX	Model Architecture
AlbertEmbeddings	✅	✅	✅	ALBERT
AlbertForQuestionAnswering		✅	❎	TFAlbertForQuestionAnswering
AlbertForSequenceClassification		✅	❎	TFAlbertForSequenceClassification
AlbertForTokenClassification		✅	❎	TFAlbertForTokenClassification
Automatic Speech Recognition (Wav2Vec2ForCTC)		❎	❎	TFWav2Vec2ForCTC
BartForZeroShotClassification		✅	❎	TFBartForSequenceClassification
BartTransformer		✅	❎	TFBartForConditionalGeneration
BertEmbeddings	✅	✅	✅	BERT - Small BERT - ELECTRA
BertForQuestionAnswering		✅	✅	TFBertForQuestionAnswering
BertForSequenceClassification		✅	✅	TFBertForSequenceClassification
BertForTokenClassification		✅	✅	TFBertForTokenClassification
BertForZeroShotClassification		✅	❎	TFBertForSequenceClassification
BertSentenceEmbeddings	✅	✅	❎	BERT - Small BERT - ELECTRA
CamemBertEmbeddings		✅	✅	CamemBERT
CamemBertForQuestionAnswering		✅	❎	TFCamembertForQuestionAnswering
CamemBertForSequenceClassification		✅	❎	TFCamemBertForSequenceClassification
CamemBertForTokenClassification		✅	❎	TFCamemBertForTokenClassification
CLIPForZeroShotClassification		✅	✅	CLIP
ConvNextForImageClassification		❎	❎	TFConvNextForImageClassification
DeBertaEmbeddings		✅	✅	DeBERTa-v2 - DeBERTa-v3
DeBertaForQuestionAnswering		✅	❎	TFDebertaV2ForQuestionAnswering
DeBertaForSequenceClassification		✅	❎	TFDebertaV2ForSequenceClassification
DeBertaForTokenClassification		✅	❎	TFDebertaV2ForTokenClassification
DistilBertEmbeddings		✅	✅	DistilBERT
DistilBertForQuestionAnswering		✅	✅	TFDistilBertForQuestionAnswering
DistilBertForSequenceClassification		✅	✅	TFDistilBertForSequenceClassification
DistilBertForTokenClassification		✅	✅	TFDistilBertForTokenClassification
DistilBertForZeroShotClassification		✅	❎	TFDistilBertForSequenceClassification
E5Embeddings		✅	✅	SentenceTransformer
ElmoEmbeddings	❎		❌
HubertForCTC		❎	❎	TFHubertForCTC
InstructorEmbeddings		✅	❎	INSTRUCTOR
LongformerEmbeddings		✅	❌	Longformer
LongformerForQuestionAnswering		✅	❎	TFLongformerForQuestionAnswering
LongformerForSequenceClassification		✅	❎	TFLongformerForSequenceClassification
LongformerForTokenClassification		✅	❎	TFLongformerForTokenClassification
MarianTransformer		❌	❎
MPNetEmbeddings		✅	✅	SentenceTransformer
OpenAI GPT2		❌	❎
RoBertaEmbeddings		✅	✅	RoBERTa - DistilRoBERTa
RoBertaForQuestionAnswering		✅	❎	TFRobertaForQuestionAnswering
RoBertaForSequenceClassification		✅	❎	TFRobertaForSequenceClassification
RoBertaForTokenClassification		✅	❎	TFRobertaForTokenClassification
RoBertaForZeroShotClassification		✅	❎	TFRobertaForSequenceClassification
RoBertaSentenceEmbeddings		✅	✅	RoBERTa - DistilRoBERTa
SwinForImageClassification		❎	❎	TFSwinForImageClassification
T5Transformer		❌	❎
TapasForQuestionAnswering		❎	❎	TFTapasForQuestionAnswering
UniversalSentenceEncoder	❎		❌
VisionEncoderDecoderForImageCaptioning		✅	❎	VisionEncoderDecoderModel
ViTForImageClassification	❌	✅	❎	TFViTForImageClassification
WhisperForCTC		✅	✅	WhisperForConditionalGeneration
XlmRoBertaEmbeddings		✅	✅	XLM-RoBERTa
XlmRoBertaForQuestionAnswering		✅	❎	TFXLMRobertaForQuestionAnswering
XlmRoBertaForSequenceClassification		✅	❎	TFXLMRobertaForSequenceClassification
XlmRoBertaForTokenClassification		✅	❎	TFXLMRobertaForTokenClassification
XlmRoBertaForZeroShotClassification		✅	❎	TFXLMRobertaForSequenceClassification
XlmRoBertaSentenceEmbeddings		✅	❎	SentenceTransformer
XlnetEmbeddings		✅	❌	XLNet
XlnetForSequenceClassification		✅	❎	TFXLNetForSequenceClassification
XlnetForTokenClassification		✅	❎	TFXLNetForTokenClassificationet
ZeroShotNerModel		✅	❎	TFRobertaForSequenceClassification

Example Notebooks

HuggingFace, Optimum, PyTorch, and ONNX Runtime to Spark NLP (ONNX)

Spark NLP	Notebooks	Colab
AlbertForQuestionAnswering	HuggingFace ONNX in Spark NLP AlbertForQuestionAnswering
AlbertForSequenceClassification	HuggingFace ONNX in Spark NLP AlbertForSequenceClassification
AlbertForTokenClassification	HuggingFace ONNX in Spark NLP AlbertForTokenClassification
BertEmbeddings	HuggingFace ONNX in Spark NLP BERT
BertForQuestionAnswering	HuggingFace ONNX in Spark NLP BertForQuestionAnswering
BertForSequenceClassification	HuggingFace ONNX in Spark NLP BertForSequenceClassification
BertForTokenClassification	HuggingFace ONNX in Spark NLP BertForTokenClassification
BertSentenceEmbeddings	HuggingFace ONNX in Spark NLP BertSentenceEmbeddings
CLIPForZeroShotClassification	HuggingFace ONNX in Spark NLP CLIP
DeBertaEmbeddings	HuggingFace ONNX in Spark NLP DeBERTa
DistilBertEmbeddings	HuggingFace ONNX in Spark NLP DistilBERT
DistilBertForQuestionAnswering	HuggingFace ONNX in Spark NLP DistilBertForQuestionAnswering
DistilBertForSequenceClassification	HuggingFace ONNX in Spark NLP DistilBertForSequenceClassification
DistilBertForTokenClassification	HuggingFace ONNX in Spark NLP DistilBertForTokenClassification
E5Embeddings	HuggingFace ONNX in Spark NLP E5
MarianTransformer	HuggingFace ONNX in Spark NLP Marian
MPNet	HuggingFace ONNX in Spark NLP MPNet
RoBertaEmbeddings	HuggingFace ONNX in Spark NLP RoBERTa
RobertaForQuestionAnswering	HuggingFace ONNX in Spark NLP RoBertaForQuestionAnswering
RoBertaForSequenceClassification	HuggingFace ONNX in Spark NLP RoBertaForSequenceClassification
RoBertaForTokenClassification	HuggingFace ONNX in Spark NLP RoBertaForTokenClassification
T5Transformer	HuggingFace ONNX in Spark NLP T5
WhisperForCTC	HuggingFace ONNX in Spark NLP MPNet
XlmRoBertaSentenceEmbeddings	HuggingFace ONNX in Spark NLP XlmRoBertaSentenceEmbeddings

HuggingFace to Spark NLP (TensorFlow)

Spark NLP	Notebooks	Colab
AlbertEmbeddings	HuggingFace in Spark NLP - ALBERT
AlbertForQuestionAnswering	HuggingFace in Spark NLP - AlbertForQuestionAnswering
AlbertForSequenceClassification	HuggingFace in Spark NLP - AlbertForSequenceClassification
AlbertForTokenClassification	HuggingFace in Spark NLP - AlbertForTokenClassification
BertEmbeddings	HuggingFace in Spark NLP - BERT
BertForQuestionAnswering	HuggingFace in Spark NLP - BertForQuestionAnswering
BertForSequenceClassification	HuggingFace in Spark NLP - BertForSequenceClassification
BertForTokenClassification	HuggingFace in Spark NLP - BertForTokenClassification
BertForZeroShotClassification	HuggingFace in Spark NLP - BertForZeroShotClassification
BertSentenceEmbeddings	HuggingFace in Spark NLP - BERT Sentence
BERTSentenceEmbeddings	HuggingFace in Spark NLP - BERT Sentence
CamemBertEmbeddings	HuggingFace in Spark NLP - CamemBERT
CamemBertForQuestionAnswering	HuggingFace in Spark NLP - CamemBertForQuestionAnswering
CamemBertForSequenceClassification	HuggingFace in Spark NLP - CamemBertForSequenceClassification
CamemBertForTokenClassification	HuggingFace in Spark NLP - CamemBertForTokenClassification
ConvNextForImageClassification	HuggingFace in Spark NLP - ConvNextForImageClassification
DeBertaEmbeddings	HuggingFace in Spark NLP - DeBERTa
DeBertaForQuestionAnswering	HuggingFace in Spark NLP - DeBertaForQuestionAnswering
DistilBertEmbeddings	HuggingFace in Spark NLP - DistilBERT
DistilBertForQuestionAnswering	HuggingFace in Spark NLP - DistilBertForQuestionAnswering
DistilBertForSequenceClassification	HuggingFace in Spark NLP - DistilBertForSequenceClassification
DistilBertForTokenClassification	HuggingFace in Spark NLP - DistilBertForTokenClassification
DistilBertForZeroClassification	HuggingFace in Spark NLP - DistilBertForZeroClassification
DistilBertForZeroShotClassification	HuggingFace in Spark NLP - DistilBertForZeroShotClassification
LongformerEmbeddings	HuggingFace in Spark NLP - Longformer
LongformerForQuestionAnswering	HuggingFace in Spark NLP - LongformerForQuestionAnswering
LongformerForSequenceClassification	HuggingFace in Spark NLP - LongformerForSequenceClassification
RoBertaEmbeddings	HuggingFace in Spark NLP - RoBERTa
RoBertaForQuestionAnswering	HuggingFace in Spark NLP - RoBertaForQuestionAnswering
RoBertaForSequenceClassification	HuggingFace in Spark NLP - RoBertaForSequenceClassification
RoBertaForTokenClassification	HuggingFace in Spark NLP - RoBertaForTokenClassification
RoBertaForZeroShotClassification	HuggingFace in Spark NLP - RoBertaForZeroShotClassification
SwinForImageClassification	HuggingFace in Spark NLP - SwinForImageClassification
ViTForImageClassification	HuggingFace in Spark NLP - ViTForImageClassification
WhisperForCTC	HuggingFace in Spark NLP - WhisperForCTC
XlmRoBertaEmbeddings	HuggingFace in Spark NLP - XLM-RoBERTa
XlmRobertaForQuestionAnswering	HuggingFace in Spark NLP - XlmRobertaForQuestionAnswering
XlmRoBertaForSequenceClassification	HuggingFace in Spark NLP - XlmRoBertaForSequenceClassification
XlmRoBertaForTokenClassification	HuggingFace in Spark NLP - XlmRoBertaForTokenClassification
XlnetEmbeddings	HuggingFace in Spark NLP - XLNet
XlnetForSequenceClassification	HuggingFace in Spark NLP - XlnetForSequenceClassification
T5Transformer	HuggingFace in Spark NLP - T5

TF Hub to Spark NLP

Spark NLP	TF Hub Notebooks	Colab
AlbertEmbeddings	TF Hub in Spark NLP - ALBERT
BertEmbeddings	TF Hub in Spark NLP - BERT
BertSentenceEmbeddings	TF Hub in Spark NLP - BERT Sentence