Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add sigmoid to softmax loss #7616

Merged
merged 61 commits into from
Feb 9, 2021
Merged

Add sigmoid to softmax loss #7616

merged 61 commits into from
Feb 9, 2021

Conversation

dakshvar22
Copy link
Contributor

@dakshvar22 dakshvar22 commented Dec 21, 2020

Proposed changes:

  • Constrain similarity values to an approximate range in DotProductLoss by applying sigmoid over them during training.
  • Also, added an option model_confidence to each ML component. It affects how model's confidence for each label is computed during inference. It can take three values -
  1. softmax - Similarities between input and label embeddings are post-processed with a softmax function, as a result of which confidence for all labels sum up to 1.
  2. cosine - Cosine similarity between input label embeddings. Confidence for each label is in the range [-1,1].
  3. inner - Dot product similarity between input and label embeddings. Confidence for each label in in an unbounded range.

Change autoconfig to use constrain_similarities=True and model_confidence=cosine.

Status (please check what you already did):

  • added some tests for the functionality
  • updated the documentation
  • updated the changelog (please check changelog for instructions)
  • reformat files using black (please check Readme for instructions)

@github-actions
Copy link
Contributor

Commit: fafa17e, The full report is available as an artifact.

Dataset: curekart_full, Dataset repository branch: hint3

Configuration Intent Classification Micro F1 Entity Recognition Micro F1 Response Selection Micro F1
BERT + DIET(bow) + ResponseSelector(bow)
test: 2m47s, train: 3m31s, total: 6m17s
0.7960 (0.00) no data no data
BERT + DIET(seq) + ResponseSelector(t2t)
test: 2m31s, train: 6m34s, total: 9m5s
0.8291 (0.00) no data no data
ConveRT + DIET(bow) + ResponseSelector(bow)
test: 38s, train: 1m23s, total: 2m1s
0.8473 (0.00) no data no data
ConveRT + DIET(seq) + ResponseSelector(t2t)
test: 33s, train: 3m33s, total: 4m6s
0.8539 (0.00) no data no data
Sparse + BERT + DIET(bow) + ResponseSelector(bow)
test: 2m57s, train: 3m53s, total: 6m49s
0.8453 (0.00) no data no data
Sparse + BERT + DIET(seq) + ResponseSelector(t2t)
test: 2m10s, train: 4m56s, total: 7m6s
0.8205 (0.00) no data no data
Sparse + ConveRT + DIET(bow) + ResponseSelector(bow)
test: 45s, train: 2m0s, total: 2m45s
0.8438 (0.00) no data no data
Sparse + ConveRT + DIET(seq) + ResponseSelector(t2t)
test: 50s, train: 6m9s, total: 6m58s
0.8536 (0.00) no data no data
Sparse + DIET(bow) + ResponseSelector(bow)
test: 18s, train: 1m14s, total: 1m31s
0.8277 (0.00) no data no data
Sparse + DIET(seq) + ResponseSelector(t2t)
test: 19s, train: 4m2s, total: 4m20s
0.7950 (0.00) no data no data

Dataset: curekart_subset, Dataset repository branch: hint3

Configuration Intent Classification Micro F1 Entity Recognition Micro F1 Response Selection Micro F1
BERT + DIET(bow) + ResponseSelector(bow)
test: 2m28s, train: 2m34s, total: 5m2s
0.7315 (0.00) no data no data
BERT + DIET(seq) + ResponseSelector(t2t)
test: 2m4s, train: 4m0s, total: 6m4s
0.8027 (0.00) no data no data
ConveRT + DIET(bow) + ResponseSelector(bow)
test: 33s, train: 58s, total: 1m31s
0.8183 (0.00) no data no data
ConveRT + DIET(seq) + ResponseSelector(t2t)
test: 37s, train: 2m59s, total: 3m35s
0.8298 (0.00) no data no data
Sparse + BERT + DIET(bow) + ResponseSelector(bow)
test: 2m21s, train: 2m46s, total: 5m7s
0.8162 (0.00) no data no data
Sparse + BERT + DIET(seq) + ResponseSelector(t2t)
test: 2m48s, train: 5m58s, total: 8m46s
0.8022 (0.00) no data no data
Sparse + ConveRT + DIET(bow) + ResponseSelector(bow)
test: 41s, train: 1m26s, total: 2m6s
0.8103 (0.00) no data no data
Sparse + ConveRT + DIET(seq) + ResponseSelector(t2t)
test: 41s, train: 3m11s, total: 3m51s
0.8309 (0.00) no data no data
Sparse + DIET(bow) + ResponseSelector(bow)
test: 16s, train: 50s, total: 1m6s
0.7884 (0.00) no data no data
Sparse + DIET(seq) + ResponseSelector(t2t)
test: 15s, train: 2m23s, total: 2m38s
0.7694 (0.00) no data no data

Dataset: powerplay11_full, Dataset repository branch: hint3

Configuration Intent Classification Micro F1 Entity Recognition Micro F1 Response Selection Micro F1
BERT + DIET(bow) + ResponseSelector(bow)
test: 1m58s, train: 2m47s, total: 4m44s
0.4655 (0.00) no data no data
BERT + DIET(seq) + ResponseSelector(t2t)
test: 2m27s, train: 6m19s, total: 8m45s
0.5537 (0.00) no data no data
ConveRT + DIET(bow) + ResponseSelector(bow)
test: 50s, train: 1m18s, total: 2m7s
0.4982 (0.00) no data no data
ConveRT + DIET(seq) + ResponseSelector(t2t)
test: 37s, train: 2m58s, total: 3m34s
0.5438 (0.00) no data no data
Sparse + BERT + DIET(bow) + ResponseSelector(bow)
test: 1m57s, train: 2m59s, total: 4m56s
0.6182 (0.00) no data no data
Sparse + BERT + DIET(seq) + ResponseSelector(t2t)
test: 2m35s, train: 7m6s, total: 9m40s
0.5636 (0.00) no data no data
Sparse + ConveRT + DIET(bow) + ResponseSelector(bow)
test: 39s, train: 1m9s, total: 1m48s
0.5501 (0.00) no data no data
Sparse + ConveRT + DIET(seq) + ResponseSelector(t2t)
test: 48s, train: 3m52s, total: 4m39s
0.5491 (0.00) no data no data
Sparse + DIET(bow) + ResponseSelector(bow)
test: 29s, train: 54s, total: 1m23s
0.5667 (0.00) no data no data
Sparse + DIET(seq) + ResponseSelector(t2t)
test: 31s, train: 3m52s, total: 4m22s
0.5537 (0.00) no data no data

Dataset: powerplay11_subset, Dataset repository branch: hint3

Configuration Intent Classification Micro F1 Entity Recognition Micro F1 Response Selection Micro F1
BERT + DIET(bow) + ResponseSelector(bow)
test: 2m2s, train: 2m38s, total: 4m40s
0.3236 (0.00) no data no data
BERT + DIET(seq) + ResponseSelector(t2t)
test: 2m29s, train: 5m0s, total: 7m29s
0.4473 (0.00) no data no data
ConveRT + DIET(bow) + ResponseSelector(bow)
test: 56s, train: 1m6s, total: 2m1s
0.3912 (0.00) no data no data
ConveRT + DIET(seq) + ResponseSelector(t2t)
test: 50s, train: 3m4s, total: 3m53s
0.4545 (0.00) no data no data
Sparse + BERT + DIET(bow) + ResponseSelector(bow)
test: 1m59s, train: 2m27s, total: 4m25s
0.4699 (0.00) no data no data
Sparse + BERT + DIET(seq) + ResponseSelector(t2t)
test: 2m30s, train: 4m51s, total: 7m20s
0.4691 (0.00) no data no data
Sparse + ConveRT + DIET(bow) + ResponseSelector(bow)
test: 46s, train: 1m4s, total: 1m49s
0.3818 (0.00) no data no data
Sparse + ConveRT + DIET(seq) + ResponseSelector(t2t)
test: 55s, train: 3m15s, total: 4m10s
0.4691 (0.00) no data no data
Sparse + DIET(bow) + ResponseSelector(bow)
test: 35s, train: 45s, total: 1m20s
0.4314 (0.00) no data no data
Sparse + DIET(seq) + ResponseSelector(t2t)
test: 28s, train: 1m59s, total: 2m27s
0.5027 (0.00) no data no data

Dataset: sofmattress_full, Dataset repository branch: hint3

Configuration Intent Classification Micro F1 Entity Recognition Micro F1 Response Selection Micro F1
BERT + DIET(bow) + ResponseSelector(bow)
test: 1m34s, train: 1m57s, total: 3m31s
0.6797 (0.00) no data no data
BERT + DIET(seq) + ResponseSelector(t2t)
test: 2m13s, train: 4m38s, total: 6m50s
0.7359 (0.00) no data no data
ConveRT + DIET(bow) + ResponseSelector(bow)
test: 28s, train: 48s, total: 1m16s
0.7100 (0.00) no data no data
ConveRT + DIET(seq) + ResponseSelector(t2t)
test: 31s, train: 2m4s, total: 2m34s
0.7489 (0.00) no data no data
Sparse + BERT + DIET(bow) + ResponseSelector(bow)
test: 1m51s, train: 2m37s, total: 4m28s
0.6926 (0.00) no data no data
Sparse + BERT + DIET(seq) + ResponseSelector(t2t)
test: 1m46s, train: 3m15s, total: 5m1s
0.7056 (0.00) no data no data
Sparse + ConveRT + DIET(bow) + ResponseSelector(bow)
test: 36s, train: 1m7s, total: 1m42s
0.7619 (0.00) no data no data
Sparse + ConveRT + DIET(seq) + ResponseSelector(t2t)
test: 27s, train: 1m53s, total: 2m20s
0.6883 (0.00) no data no data
Sparse + DIET(bow) + ResponseSelector(bow)
test: 15s, train: 41s, total: 55s
0.6883 (0.00) no data no data
Sparse + DIET(seq) + ResponseSelector(t2t)
test: 14s, train: 1m41s, total: 1m54s
0.7056 (0.00) no data no data

Dataset: sofmattress_subset, Dataset repository branch: hint3

Configuration Intent Classification Micro F1 Entity Recognition Micro F1 Response Selection Micro F1
BERT + DIET(bow) + ResponseSelector(bow)
test: 1m47s, train: 2m15s, total: 4m2s
0.4935 (0.00) no data no data
BERT + DIET(seq) + ResponseSelector(t2t)
test: 2m5s, train: 3m28s, total: 5m33s
0.6234 (0.00) no data no data
ConveRT + DIET(bow) + ResponseSelector(bow)
test: 36s, train: 50s, total: 1m25s
0.6190 (0.00) no data no data
ConveRT + DIET(seq) + ResponseSelector(t2t)
test: 30s, train: 1m30s, total: 2m0s
0.6840 (0.00) no data no data
Sparse + BERT + DIET(bow) + ResponseSelector(bow)
test: 2m4s, train: 2m28s, total: 4m31s
0.6074 (0.00) no data no data
Sparse + BERT + DIET(seq) + ResponseSelector(t2t)
test: 1m57s, train: 3m16s, total: 5m12s
0.5844 (0.00) no data no data
Sparse + ConveRT + DIET(bow) + ResponseSelector(bow)
test: 32s, train: 49s, total: 1m20s
0.6190 (0.00) no data no data
Sparse + ConveRT + DIET(seq) + ResponseSelector(t2t)
test: 29s, train: 1m32s, total: 2m1s
0.6364 (0.00) no data no data
Sparse + DIET(bow) + ResponseSelector(bow)
test: 14s, train: 30s, total: 44s
0.5870 (0.00) no data no data
Sparse + DIET(seq) + ResponseSelector(t2t)
test: 13s, train: 1m10s, total: 1m23s
0.6061 (0.00) no data no data

@github-actions
Copy link
Contributor

Commit: fafa17e, The full report is available as an artifact.

Dataset: Carbon Bot, Dataset repository branch: hint3

Configuration Intent Classification Micro F1 Entity Recognition Micro F1 Response Selection Micro F1
BERT + DIET(bow) + ResponseSelector(bow)
test: 1m18s, train: 3m25s, total: 4m43s
0.7650 (-0.02) 0.6260 (-0.13) 0.5762 (0.02)
BERT + DIET(seq) + ResponseSelector(t2t)
test: 1m22s, train: 6m14s, total: 7m36s
0.7981 (0.01) 0.8323 (0.07) 0.5762 (0.01)
ConveRT + DIET(bow) + ResponseSelector(bow)
test: 47s, train: 3m23s, total: 4m9s
0.8330 (0.00) 0.6260 (0.00) 0.6093 (0.00)
ConveRT + DIET(seq) + ResponseSelector(t2t)
test: 55s, train: 6m20s, total: 7m15s
0.8330 (0.00) 0.8497 (0.00) 0.6000 (0.00)
Sparse + BERT + DIET(bow) + ResponseSelector(bow)
test: 1m25s, train: 3m37s, total: 5m2s
0.7786 (-0.02) 0.6260 (-0.13) 0.5762 (-0.01)
Sparse + BERT + DIET(seq) + ResponseSelector(t2t)
test: 1m29s, train: 6m59s, total: 8m28s
0.7767 (-0.03) 0.8289 (0.04) 0.5847 (no data)
Sparse + ConveRT + DIET(bow) + ResponseSelector(bow)
test: 54s, train: 3m31s, total: 4m24s
0.8252 (0.00) 0.6260 (0.00) 0.5933 (0.00)
Sparse + ConveRT + DIET(seq) + ResponseSelector(t2t)
test: 1m0s, train: 6m58s, total: 7m57s
0.8136 (0.00) 0.8378 (0.00) 0.6026 (0.00)
Sparse + DIET(bow) + ResponseSelector(bow)
test: 28s, train: 2m20s, total: 2m48s
0.7515 (0.02) 0.6260 (-0.13) 0.5298 (no data)
Sparse + DIET(seq) + ResponseSelector(t2t)
test: 37s, train: 5m49s, total: 6m26s
0.7320 (-0.01) 0.7183 (0.01) 0.5364 (no data)

Dataset: Hermit, Dataset repository branch: hint3

Configuration Intent Classification Micro F1 Entity Recognition Micro F1 Response Selection Micro F1
BERT + DIET(bow) + ResponseSelector(bow)
test: 3m20s, train: 18m12s, total: 21m31s
0.9015 (0.01) 0.7504 (0.00) no data
BERT + DIET(seq) + ResponseSelector(t2t)
test: 2m39s, train: 24m4s, total: 26m42s
0.9024 (0.01) 0.8113 (0.01) no data
ConveRT + DIET(bow) + ResponseSelector(bow)
test: 1m33s, train: 17m48s, total: 19m21s
0.8931 (0.00) 0.7504 (0.00) no data
ConveRT + DIET(seq) + ResponseSelector(t2t)
test: 1m41s, train: 24m38s, total: 26m18s
0.8848 (0.00) 0.8018 (0.00) no data
Sparse + BERT + DIET(bow) + ResponseSelector(bow)
test: 3m20s, train: 19m53s, total: 23m12s
0.8838 (0.00) 0.7504 (0.00) no data
Sparse + BERT + DIET(seq) + ResponseSelector(t2t)
test: 2m43s, train: 25m19s, total: 28m2s
0.8978 (0.03) 0.7957 (0.00) no data
Sparse + ConveRT + DIET(bow) + ResponseSelector(bow)
test: 1m37s, train: 19m51s, total: 21m28s
0.8996 (0.00) 0.7504 (0.00) no data
Sparse + ConveRT + DIET(seq) + ResponseSelector(t2t)
test: 1m43s, train: 25m59s, total: 27m41s
0.9006 (0.00) 0.8128 (0.00) no data
Sparse + DIET(bow) + ResponseSelector(bow)
test: 57s, train: 17m8s, total: 18m5s
0.8467 (0.02) 0.7504 (0.00) no data
Sparse + DIET(seq) + ResponseSelector(t2t)
test: 1m8s, train: 23m58s, total: 25m6s
0.8606 (0.02) 0.7688 (0.02) no data

Dataset: Sara, Dataset repository branch: hint3

Configuration Intent Classification Micro F1 Entity Recognition Micro F1 Response Selection Micro F1
BERT + DIET(bow) + ResponseSelector(bow)
test: 2m9s, train: 4m19s, total: 6m28s
0.8609 (-0.01) 0.8683 (0.00) 0.8696 (0.00)
BERT + DIET(seq) + ResponseSelector(t2t)
test: 2m19s, train: 6m56s, total: 9m14s
0.8511 (-0.00) 0.8898 (-0.00) 0.8696 (-0.01)
ConveRT + DIET(bow) + ResponseSelector(bow)
test: 1m10s, train: 5m38s, total: 6m47s
0.8913 (0.00) 0.8683 (0.00) 0.9348 (0.00)
ConveRT + DIET(seq) + ResponseSelector(t2t)
test: 1m23s, train: 7m25s, total: 8m47s
0.8923 (0.00) 0.9014 (0.00) 0.9413 (0.00)
Sparse + BERT + DIET(bow) + ResponseSelector(bow)
test: 2m15s, train: 5m36s, total: 7m51s
0.8511 (-0.03) 0.8683 (0.00) 0.8978 (0.00)
Sparse + BERT + DIET(seq) + ResponseSelector(t2t)
test: 2m23s, train: 8m6s, total: 10m28s
0.8541 (-0.02) 0.8934 (-0.00) 0.8978 (0.00)
Sparse + ConveRT + DIET(bow) + ResponseSelector(bow)
test: 1m18s, train: 5m51s, total: 7m8s
0.8923 (0.00) 0.8683 (0.00) 0.9304 (0.00)
Sparse + ConveRT + DIET(seq) + ResponseSelector(t2t)
test: 1m30s, train: 8m36s, total: 10m5s
0.9011 (0.00) 0.9113 (0.00) 0.9326 (0.00)
Sparse + DIET(bow) + ResponseSelector(bow)
test: 41s, train: 4m12s, total: 4m53s
0.8384 (0.00) 0.8683 (0.00) 0.8891 (0.03)
Sparse + DIET(seq) + ResponseSelector(t2t)
test: 51s, train: 6m59s, total: 7m51s
0.8306 (-0.01) 0.8565 (0.00) 0.8848 (0.03)

@github-actions
Copy link
Contributor

Commit: 0255ad6, The full report is available as an artifact.

Dataset: Private 1, Dataset repository branch: master

Configuration Intent Classification Micro F1 Entity Recognition Micro F1 Response Selection Micro F1
BERT + DIET(bow) + ResponseSelector(bow)
test: 1m51s, train: 6m19s, total: 8m9s
0.9064 (-0.00) 0.9612 (0.00) no data
BERT + DIET(seq) + ResponseSelector(t2t)
test: 2m6s, train: 3m36s, total: 5m41s
0.9127 (-0.00) 0.9714 (-0.00) no data
Sparse + DIET(bow) + ResponseSelector(bow)
test: 21s, train: 4m55s, total: 5m15s
0.8971 (0.00) 0.9612 (0.00) no data
Sparse + DIET(seq) + ResponseSelector(t2t)
test: 35s, train: 3m19s, total: 3m54s
0.9044 (0.01) 0.9690 (-0.00) no data

Dataset: Private 2, Dataset repository branch: master

Configuration Intent Classification Micro F1 Entity Recognition Micro F1 Response Selection Micro F1
BERT + DIET(bow) + ResponseSelector(bow)
test: 1m48s, train: 13m13s, total: 15m0s
0.8648 (-0.01) no data no data
Sparse + DIET(bow) + ResponseSelector(bow)
test: 29s, train: 6m25s, total: 6m53s
0.8562 (-0.01) no data no data
Sparse + DIET(seq) + ResponseSelector(t2t)
test: 32s, train: 5m14s, total: 5m46s
0.8605 (0.01) no data no data

Dataset: Private 3, Dataset repository branch: master

Configuration Intent Classification Micro F1 Entity Recognition Micro F1 Response Selection Micro F1
BERT + DIET(bow) + ResponseSelector(bow)
test: 53s, train: 1m31s, total: 2m23s
0.9095 (0.00) no data no data
BERT + DIET(seq) + ResponseSelector(t2t)
test: 53s, train: 53s, total: 1m46s
0.9218 (0.11) no data no data
Sparse + DIET(bow) + ResponseSelector(bow)
test: 27s, train: 1m24s, total: 1m51s
0.8807 (0.03) no data no data
Sparse + DIET(seq) + ResponseSelector(t2t)
test: 28s, train: 46s, total: 1m13s
0.8642 (0.05) no data no data

@github-actions
Copy link
Contributor

github-actions bot commented Feb 9, 2021

Commit: 1a5e454, The full report is available as an artifact.

Dataset: Carbon Bot, Dataset repository branch: main

Configuration Intent Classification Micro F1 Entity Recognition Micro F1 Response Selection Micro F1
BERT + DIET(bow) + ResponseSelector(bow)
test: 1m20s, train: 5m12s, total: 6m31s
0.7864 (0.00) 0.7529 (0.00) 0.5847 (0.00)
BERT + DIET(seq) + ResponseSelector(t2t)
test: 1m27s, train: 4m17s, total: 5m43s
0.8000 (0.00) 0.7757 (0.00) 0.5430 (0.00)
Sparse + BERT + DIET(bow) + ResponseSelector(bow)
test: 1m18s, train: 4m38s, total: 5m56s
0.7864 (0.00) 0.7529 (0.00) 0.5033 (0.00)
Sparse + BERT + DIET(seq) + ResponseSelector(t2t)
test: 1m29s, train: 4m45s, total: 6m14s
0.7961 (0.00) 0.8011 (0.00) 0.5364 (-0.02)
Sparse + DIET(bow) + ResponseSelector(bow)
test: 30s, train: 2m45s, total: 3m14s
0.7437 (0.01) 0.7529 (0.00) 0.5430 (0.00)
Sparse + DIET(seq) + ResponseSelector(t2t)
test: 40s, train: 3m55s, total: 4m35s
0.7379 (0.00) 0.7039 (0.00) 0.5183 (-0.00)

Dataset: Hermit, Dataset repository branch: main

Configuration Intent Classification Micro F1 Entity Recognition Micro F1 Response Selection Micro F1
BERT + DIET(bow) + ResponseSelector(bow)
test: 3m5s, train: 19m1s, total: 22m5s
0.8857 (0.00) 0.7504 (0.00) no data
BERT + DIET(seq) + ResponseSelector(t2t)
test: 2m34s, train: 12m7s, total: 14m40s
0.8968 (0.00) 0.8033 (0.00) no data
Sparse + BERT + DIET(bow) + ResponseSelector(bow)
test: 3m3s, train: 21m43s, total: 24m46s
0.8755 (0.00) 0.7504 (0.00) no data
Sparse + BERT + DIET(seq) + ResponseSelector(t2t)
test: 2m36s, train: 13m4s, total: 15m40s
0.8559 (-0.00) 0.7934 (-0.00) no data
Sparse + DIET(bow) + ResponseSelector(bow)
test: 55s, train: 18m5s, total: 19m0s
0.8336 (0.00) 0.7504 (0.00) no data
Sparse + DIET(seq) + ResponseSelector(t2t)
test: 1m7s, train: 12m1s, total: 13m7s
0.8392 (0.00) 0.7523 (-0.00) no data

Dataset: Private 1, Dataset repository branch: main

Configuration Intent Classification Micro F1 Entity Recognition Micro F1 Response Selection Micro F1
BERT + DIET(bow) + ResponseSelector(bow)
test: 1m45s, train: 3m39s, total: 5m24s
0.9054 (-0.00) 0.9612 (0.00) no data
BERT + DIET(seq) + ResponseSelector(t2t)
test: 1m55s, train: 3m5s, total: 5m0s
0.9148 (0.00) 0.9745 (0.00) no data
Spacy + DIET(bow) + ResponseSelector(bow)
test: 1m12s, train: 3m33s, total: 4m44s
0.7983 (0.00) 0.9574 (0.00) no data
Spacy + DIET(seq) + ResponseSelector(t2t)
test: 1m23s, train: 3m47s, total: 5m9s
0.8368 (0.00) 0.9203 (0.00) no data
Sparse + DIET(bow) + ResponseSelector(bow)
test: 22s, train: 3m1s, total: 3m22s
0.8950 (-0.01) 0.9612 (0.00) no data
Sparse + DIET(seq) + ResponseSelector(t2t)
test: 35s, train: 2m52s, total: 3m27s
0.9054 (0.00) 0.9736 (0.00) no data
Sparse + Spacy + DIET(bow) + ResponseSelector(bow)
test: 1m13s, train: 4m24s, total: 5m37s
0.8940 (-0.00) 0.9574 (0.00) no data
Sparse + Spacy + DIET(seq) + ResponseSelector(t2t)
test: 1m26s, train: 4m4s, total: 5m29s
0.9064 (0.00) 0.9661 (0.00) no data

Dataset: Private 2, Dataset repository branch: main

Configuration Intent Classification Micro F1 Entity Recognition Micro F1 Response Selection Micro F1
BERT + DIET(bow) + ResponseSelector(bow)
test: 1m43s, train: 11m6s, total: 12m49s
0.8712 (0.00) no data no data
Spacy + DIET(bow) + ResponseSelector(bow)
test: 1m14s, train: 6m59s, total: 8m12s
0.5751 (0.00) no data no data
Spacy + DIET(seq) + ResponseSelector(t2t)
test: 1m18s, train: 6m34s, total: 7m52s
0.7039 (0.00) no data no data
Sparse + DIET(bow) + ResponseSelector(bow)
test: 29s, train: 4m44s, total: 5m12s
0.8509 (0.00) no data no data
Sparse + DIET(seq) + ResponseSelector(t2t)
test: 31s, train: 4m49s, total: 5m20s
0.8573 (0.00) no data no data
Sparse + Spacy + DIET(bow) + ResponseSelector(bow)
test: 1m18s, train: 8m33s, total: 9m50s
0.8455 (0.00) no data no data
Sparse + Spacy + DIET(seq) + ResponseSelector(t2t)
test: 1m22s, train: 7m0s, total: 8m21s
0.8519 (0.00) no data no data

Dataset: Private 3, Dataset repository branch: main

Configuration Intent Classification Micro F1 Entity Recognition Micro F1 Response Selection Micro F1
BERT + DIET(bow) + ResponseSelector(bow)
test: 50s, train: 1m2s, total: 1m52s
0.9177 (0.00) no data no data
BERT + DIET(seq) + ResponseSelector(t2t)
test: 51s, train: 43s, total: 1m34s
0.8148 (0.00) no data no data
Spacy + DIET(seq) + ResponseSelector(t2t)
test: 1m12s, train: 1m19s, total: 2m31s
0.2675 (0.00) no data no data
Sparse + DIET(bow) + ResponseSelector(bow)
test: 27s, train: 58s, total: 1m24s
0.8519 (0.00) no data no data
Sparse + DIET(seq) + ResponseSelector(t2t)
test: 29s, train: 39s, total: 1m7s
0.8189 (0.00) no data no data
Sparse + Spacy + DIET(bow) + ResponseSelector(bow)
test: 1m13s, train: 1m49s, total: 3m1s
0.8436 (0.00) no data no data
Sparse + Spacy + DIET(seq) + ResponseSelector(t2t)
test: 1m14s, train: 1m25s, total: 2m39s
0.8683 (0.00) no data no data

Dataset: Sara, Dataset repository branch: main

Configuration Intent Classification Micro F1 Entity Recognition Micro F1 Response Selection Micro F1
BERT + DIET(bow) + ResponseSelector(bow)
test: 2m2s, train: 4m36s, total: 6m38s
0.8668 (0.00) 0.8683 (0.00) 0.8848 (0.00)
BERT + DIET(seq) + ResponseSelector(t2t)
test: 2m14s, train: 3m31s, total: 5m45s
0.8492 (0.00) 0.8833 (0.00) 0.8761 (0.00)
Sparse + BERT + DIET(bow) + ResponseSelector(bow)
test: 2m9s, train: 6m36s, total: 8m44s
0.8629 (-0.00) 0.8683 (0.00) 0.9000 (0.00)
Sparse + BERT + DIET(seq) + ResponseSelector(t2t)
test: 2m19s, train: 4m33s, total: 6m52s
0.8727 (0.00) 0.9113 (0.00) 0.8913 (-0.00)
Sparse + DIET(seq) + ResponseSelector(t2t)
test: 52s, train: 3m45s, total: 4m36s
0.8452 (-0.00) 0.8523 (0.00) 0.8435 (-0.01)

@github-actions
Copy link
Contributor

github-actions bot commented Feb 9, 2021

Hey @dakshvar22! 👋 To run model regression tests, comment with the /modeltest command and a configuration.

Tips 💡: The model regression test will be run on push events. You can re-run the tests by re-add status:model-regression-tests label or use a Re-run jobs button in Github Actions workflow.

Tips 💡: Every time when you want to change a configuration you should edit the comment with the previous configuration.

You can copy this in your comment and customize:

/modeltest

```yml
##########
## Available datasets
##########
# - "Carbon Bot"
# - "Hermit"
# - "Private 1"
# - "Private 2"
# - "Private 3"
# - "Sara"

##########
## Available configurations
##########
# - "BERT + DIET(bow) + ResponseSelector(bow)"
# - "BERT + DIET(seq) + ResponseSelector(t2t)"
# - "Spacy + DIET(bow) + ResponseSelector(bow)"
# - "Spacy + DIET(seq) + ResponseSelector(t2t)"
# - "Sparse + BERT + DIET(bow) + ResponseSelector(bow)"
# - "Sparse + BERT + DIET(seq) + ResponseSelector(t2t)"
# - "Sparse + DIET(bow) + ResponseSelector(bow)"
# - "Sparse + DIET(seq) + ResponseSelector(t2t)"
# - "Sparse + Spacy + DIET(bow) + ResponseSelector(bow)"
# - "Sparse + Spacy + DIET(seq) + ResponseSelector(t2t)"

## Example configuration
#################### syntax #################
## include:
##   - dataset: ["<dataset_name>"]
##     config: ["<configuration_name>"]
#
## Example:
## include:
##  - dataset: ["Carbon Bot"]
##    config: ["Sparse + DIET(bow) + ResponseSelector(bow)"]
#
## Shortcut:
## You can use the "all" shortcut to include all available configurations or datasets
#
## Example: Use the "Sparse + EmbeddingIntent + ResponseSelector(bow)" configuration
## for all available datasets
## include:
##  - dataset: ["all"]
##    config: ["Sparse + DIET(bow) + ResponseSelector(bow)"]
#
## Example: Use all available configurations for the "Carbon Bot" and "Sara" datasets
## and for the "Hermit" dataset use the "Sparse + DIET + ResponseSelector(T2T)" and
## "BERT + DIET + ResponseSelector(T2T)" configurations:
## include:
##  - dataset: ["Carbon Bot", "Sara"]
##    config: ["all"]
##  - dataset: ["Hermit"]
##    config: ["Sparse + DIET(seq) + ResponseSelector(t2t)", "BERT + DIET(seq) + ResponseSelector(t2t)"]
#
## Example: Define a branch name to check-out for a dataset repository. Default branch is 'main'
## dataset_branch: "test-branch"
## include:
##  - dataset: ["Carbon Bot", "Sara"]
##    config: ["all"]


include:
 - dataset: ["Carbon Bot"]
   config: ["Sparse + DIET(bow) + ResponseSelector(bow)"]

```

@github-actions
Copy link
Contributor

github-actions bot commented Feb 9, 2021

/modeltest

dataset_branch: "hint3"
include:
 - dataset: ["all"]
   config: ["all"]

@github-actions
Copy link
Contributor

github-actions bot commented Feb 9, 2021

The model regression tests have started. It might take a while, please be patient.
As soon as results are ready you'll see a new comment with the results.

Used configuration can be found in the comment.

docs/docs/migration-guide.mdx Outdated Show resolved Hide resolved
@github-actions
Copy link
Contributor

github-actions bot commented Feb 9, 2021

Commit: 6d4a27a, The full report is available as an artifact.

Dataset: Private 1, Dataset repository branch: hint3

Configuration Intent Classification Micro F1 Entity Recognition Micro F1 Response Selection Micro F1
Spacy + DIET(bow) + ResponseSelector(bow)
test: 1m11s, train: 3m36s, total: 4m47s
0.7807 (-0.02) 0.9574 (0.00) no data
Spacy + DIET(seq) + ResponseSelector(t2t)
test: 1m26s, train: 3m49s, total: 5m15s
0.8108 (-0.03) 0.9257 (0.01) no data
Sparse + DIET(bow) + ResponseSelector(bow)
test: 22s, train: 3m16s, total: 3m38s
0.9012 (0.00) 0.9612 (-0.00) no data
Sparse + DIET(seq) + ResponseSelector(t2t)
test: 36s, train: 3m0s, total: 3m36s
0.9137 (0.01) 0.9701 (-0.00) no data
Sparse + Spacy + DIET(bow) + ResponseSelector(bow)
test: 1m16s, train: 4m37s, total: 5m53s
0.8971 (0.00) 0.9574 (0.00) no data
Sparse + Spacy + DIET(seq) + ResponseSelector(t2t)
test: 1m30s, train: 4m10s, total: 5m39s
0.9054 (-0.00) 0.9727 (0.01) no data

Dataset: Private 2, Dataset repository branch: hint3

Configuration Intent Classification Micro F1 Entity Recognition Micro F1 Response Selection Micro F1
BERT + DIET(bow) + ResponseSelector(bow)
test: 1m49s, train: 11m29s, total: 13m17s
0.8682 (-0.00) no data no data
Spacy + DIET(bow) + ResponseSelector(bow)
test: 1m16s, train: 6m46s, total: 8m2s
0.5691 (-0.01) no data no data
Spacy + DIET(seq) + ResponseSelector(t2t)
test: 1m20s, train: 6m37s, total: 7m57s
0.6860 (-0.02) no data no data
Sparse + DIET(bow) + ResponseSelector(bow)
test: 29s, train: 4m48s, total: 5m17s
0.8596 (0.01) no data no data
Sparse + DIET(seq) + ResponseSelector(t2t)
test: 32s, train: 4m52s, total: 5m23s
0.8650 (0.01) no data no data
Sparse + Spacy + DIET(bow) + ResponseSelector(bow)
test: 1m21s, train: 8m36s, total: 9m56s
0.8660 (0.02) no data no data
Sparse + Spacy + DIET(seq) + ResponseSelector(t2t)
test: 1m24s, train: 7m4s, total: 8m27s
0.8639 (0.01) no data no data

Dataset: Private 3, Dataset repository branch: hint3

Configuration Intent Classification Micro F1 Entity Recognition Micro F1 Response Selection Micro F1
BERT + DIET(bow) + ResponseSelector(bow)
test: 52s, train: 1m5s, total: 1m58s
0.9259 (0.01) no data no data
BERT + DIET(seq) + ResponseSelector(t2t)
test: 54s, train: 46s, total: 1m39s
0.9053 (0.09) no data no data
Spacy + DIET(bow) + ResponseSelector(bow)
test: 1m14s, train: 1m37s, total: 2m51s
0.0700 (0.00) no data no data
Spacy + DIET(seq) + ResponseSelector(t2t)
test: 1m16s, train: 1m23s, total: 2m39s
0.5103 (0.24) no data no data
Sparse + DIET(bow) + ResponseSelector(bow)
test: 28s, train: 1m3s, total: 1m31s
0.8683 (0.02) no data no data
Sparse + DIET(seq) + ResponseSelector(t2t)
test: 29s, train: 40s, total: 1m9s
0.8642 (0.05) no data no data
Sparse + Spacy + DIET(bow) + ResponseSelector(bow)
test: 1m17s, train: 1m55s, total: 3m11s
0.8765 (0.03) no data no data
Sparse + Spacy + DIET(seq) + ResponseSelector(t2t)
test: 1m18s, train: 1m30s, total: 2m47s
0.8724 (0.00) no data no data

Dataset: Sara, Dataset repository branch: hint3

Configuration Intent Classification Micro F1 Entity Recognition Micro F1 Response Selection Micro F1
BERT + DIET(bow) + ResponseSelector(bow)
test: 2m6s, train: 5m4s, total: 7m10s
0.8580 (-0.01) 0.8683 (0.00) 0.8565 (-0.03)
BERT + DIET(seq) + ResponseSelector(t2t)
test: 2m20s, train: 3m48s, total: 6m7s
0.8570 (0.01) 0.8837 (0.00) 0.8783 (0.00)
Sparse + BERT + DIET(bow) + ResponseSelector(bow)
test: 2m16s, train: 7m6s, total: 9m21s
0.8609 (-0.01) 0.8683 (0.00) 0.8891 (-0.01)
Sparse + BERT + DIET(seq) + ResponseSelector(t2t)
test: 2m25s, train: 4m47s, total: 7m12s
0.8521 (-0.02) 0.8994 (-0.01) 0.8804 (-0.01)
Sparse + DIET(bow) + ResponseSelector(bow)
test: 41s, train: 5m9s, total: 5m51s
0.8335 (0.00) 0.8683 (0.00) 0.8826 (0.02)
Sparse + DIET(seq) + ResponseSelector(t2t)
test: 54s, train: 3m52s, total: 4m46s
0.8384 (-0.01) 0.8372 (-0.02) 0.8804 (0.03)

dakshvar22 and others added 3 commits February 9, 2021 17:10
Co-authored-by: Melinda Loubser <32034278+melindaloubser1@users.noreply.github.com>
@dakshvar22
Copy link
Contributor Author

@wochinge I've added TFLayerConfigException instead of RasaException. So, the PR should be good to merge. Can you flip your review? I can't merge otherwise :)

@wochinge
Copy link
Contributor

wochinge commented Feb 9, 2021

So, the PR should be good to merge. Can you flip your review? I can't merge otherwise :)

Sure, let me just give it a final glance 👍🏻 Sorry, had a busy day and only had time for the review now.

Copy link
Contributor

@wochinge wochinge left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please check the 2 todos. I'm good otherwise 👍🏻

rasa/shared/exceptions.py Outdated Show resolved Hide resolved
rasa/utils/train_utils.py Show resolved Hide resolved
rasa/utils/train_utils.py Show resolved Hide resolved
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants