Skip to content

Commit

Permalink
Increase transformers version
Browse files Browse the repository at this point in the history
  • Loading branch information
@julian-risch
julian-risch committed Jun 1, 2021
1 parent c9235f2 commit 1073b57
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion requirements.txt
View file
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ sklearn
seqeval
mlflow<=1.13.1
# huggingface repository
transformers==4.5.0
transformers==4.6.1
# accessing dictionary elements with dot notation
dotmap
# for inference-rest-apis
Expand Down

2 comments on commit 1073b57

@HaystackBot
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Benchmarks: QA Accuracy

✔️ QA Accuracy Benchmark Passed

run f1_change em_change tnacc_change elapsed_change f1 em tnacc elapsed f1_gold em_gold tnacc_gold elapsed_gold
0 FARM internal evaluation 0.017 0.0001 0.0169 0.0073 82.6841 78.4722 84.3763 40.0073 82.6671 78.4721 84.3594 40
1 outside eval script -0.0045 -0.0077 - 0.1838 82.9125 79.8703 - 27.1838 82.917 79.878 - 27
2 train evaluation 0.4809 -0.2106 0.1685 -8.4684 82.6359 78.4469 97.5406 1126.53 82.155 78.6575 97.3721 1135

CML watermark

@HaystackBot
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Benchmarks: QA per component

model name question preproc language model prediction head total passages per sec batch_size document_size num_processes max_seq_len doc_stride gpu sample_file error
0 deepset/bert-base-cased-squad2 When were the first traces of Human life found in France? 1.76083 17.7697 1.86714 21.4396 111.942 50 1000000 1 384 128 True samples/question_answering_sample.txt
1 deepset/bert-base-cased-squad2 How many pretrained models are available in Transformers? 1.71552 17.7762 1.881 21.3701 112.306 50 1000000 1 384 128 True samples/question_answering_sample.txt
2 deepset/minilm-uncased-squad2 When were the first traces of Human life found in France? 1.63631 5.9916 1.63462 9.30593 257.9 50 1000000 1 384 128 True samples/question_answering_sample.txt
3 deepset/minilm-uncased-squad2 How many pretrained models are available in Transformers? 1.6336 5.99122 1.71741 9.29177 258.293 50 1000000 1 384 128 True samples/question_answering_sample.txt
4 deepset/roberta-base-squad2 When were the first traces of Human life found in France? 1.49182 16.1216 1.81467 19.4202 123.583 50 1000000 1 384 128 True samples/question_answering_sample.txt
5 deepset/roberta-base-squad2 How many pretrained models are available in Transformers? 1.53201 16.1194 1.63063 19.312 124.275 50 1000000 1 384 128 True samples/question_answering_sample.txt
6 deepset/bert-large-uncased-whole-word-masking-squad2 When were the first traces of Human life found in France? 1.6376 48.7961 1.78648 52.2626 45.9219 50 1000000 1 384 128 True samples/question_answering_sample.txt
7 deepset/bert-large-uncased-whole-word-masking-squad2 How many pretrained models are available in Transformers? 1.58626 48.9031 1.69727 52.1906 45.9853 50 1000000 1 384 128 True samples/question_answering_sample.txt
8 deepset/xlm-roberta-large-squad2 When were the first traces of Human life found in France? 59.4019 52.2125 1.87353 113.537 21.1386 50 1000000 1 384 128 True samples/question_answering_sample.txt
9 deepset/xlm-roberta-large-squad2 How many pretrained models are available in Transformers? 59.4084 52.2134 1.85796 113.535 21.1388 50 1000000 1 384 128 True samples/question_answering_sample.txt

CML watermark

Please sign in to comment.