Analysis of Utterance Embeddings and Clustering Methods Related to Intent Induction for Task-Oriented Dialogue
- Paper: https://arxiv.org/abs/2212.02021
- This repository is basd on the awesome work: https://github.com/amazon-science/dstc11-track2-intent-induction
- Python >= 3.7
- PyTorch==1.11.0
# install dependencies
pip3 install -r requirements.txt
pip3 install setproctitle- DSTC11 dataset
| Dataset | Domain | #Intents | #Utterances |
|---|---|---|---|
| DSTC11-dev | insurance | 22 | 66,875 |
| DSTC11-test | insurance | 22 | 913 |
- A sample segment of conversation transcript
{"dialogue_id": "432", "turns": [{"turn_id": "insurance_432_000", "speaker_role": "Agent", "utterance": "Hello, you are currently speaking with Rivetown Insurance customer service. My name is Julian, How may I be of service to you?", "dialogue_acts": [], "intents": []}, {"turn_id": "insurance_432_001", "speaker_role": "Customer", "utterance": "The services I have being receiving from your company has being encouraging and my intent is to increase or enroll for more plans with you.", "dialogue_acts": ["InformIntent"], "intents": ["EnrollInPlan"]}, {"turn_id": "insurance_432_002", "speaker_role": "Agent", "utterance": "Whoa, that is such an encouraging word coming from you. So, which of the plans do you wish to register for?", "dialogue_acts": [], "intents": []}, {"turn_id": "insurance_432_003", "speaker_role": "Customer", "utterance": "I will like to enrol for the life insurance policy.", "dialogue_acts": ["InformIntent"], "intents": []}, {"turn_id": "insurance_432_004", "speaker_role": "Agent", "utterance": "That won't be an issue, I can help you get registered right away.", "dialogue_acts": [], "intents": []}, {"turn_id": "insurance_432_005", "speaker_role": "Customer", "utterance": "Ok then, but first can I get my current policy number.", "dialogue_acts": ["InformIntent"], "intents": ["GetPolicyNumber"]}, {"turn_id": "insurance_432_006", "speaker_role": "Agent", "utterance": "Of course, I can help you get that.", "dialogue_acts": [], "intents": []}, {"turn_id": "insurance_432_007", "speaker_role": "Customer", "utterance": "Please do!", "dialogue_acts": [], "intents": []}, {"turn_id": "insurance_432_008", "speaker_role": "Agent", "utterance": "Yap, I will need you to provide me with your first name and your last name.", "dialogue_acts": [], "intents": []}, {"turn_id": "insurance_432_009", "speaker_role": "Customer", "utterance": "My first name is Bonnie.", "dialogue_acts": [], "intents": []}, {"turn_id": "insurance_432_010", "speaker_role": "Agent", "utterance": "Bonnie is spelled B.O.N.I.E, correct?", "dialogue_acts": [], "intents": []}, {"turn_id": "insurance_432_011", "speaker_role": "Customer", "utterance": "Nah, thats wrong it has double N, as in B.O.N.N.I.E.", "dialogue_acts": [], "intents": []}, {"turn_id": "insurance_432_012", "speaker_role": "Agent", "utterance": "Oh, sorry about that, please proceed with your last name.", "dialogue_acts": [], "intents": []}, {"turn_id": "insurance_432_013", "speaker_role": "Customer", "utterance": "No offence taken, my last name is Wilson as in W.i.l.s.o.n.", "dialogue_acts": [], "intents": []}, {"turn_id": "insurance_432_014", "speaker_role": "Agent", "utterance": "W.I.L.S.O.N.", "dialogue_acts": [], "intents": []}, {"turn_id": "insurance_432_015", "speaker_role": "Customer", "utterance": "Yep, correct!", "dialogue_acts": [], "intents": []}, {"turn_id": "insurance_432_016", "speaker_role": "Agent", "utterance": "I will need you to provide me with your date of birth.", "dialogue_acts": [], "intents": []}, {"turn_id": "insurance_432_017", "speaker_role": "Customer", "utterance": "My birthdate is second of may nineteen fifty six.", "dialogue_acts": [], "intents": []}, {"turn_id": "insurance_432_018", "speaker_role": "Agent", "utterance": "Gotcha, please give me your customer number.", "dialogue_acts": [], "intents": []}, {"turn_id": "insurance_432_019", "speaker_role": "Customer", "utterance": "That should be the number at the bottom of my card right?", "dialogue_acts": [], "intents": []}, {"turn_id": "insurance_432_020", "speaker_role": "Agent", "utterance": "Yep, correct!", "dialogue_acts": [], "intents": []}, {"turn_id": "insurance_432_021", "speaker_role": "Customer", "utterance": "Ok, let me get my card.", "dialogue_acts": [], "intents": []}, {"turn_id": "insurance_432_022", "speaker_role": "Agent", "utterance": "Sure!", "dialogue_acts": [], "intents": []}, {"turn_id": "insurance_432_023", "speaker_role": "Customer", "utterance": "The number is one one one.", "dialogue_acts": [], "intents": []}, {"turn_id": "insurance_432_024", "speaker_role": "Agent", "utterance": "One one one.", "dialogue_acts": [], "intents": []}, {"turn_id": "insurance_432_025", "speaker_role": "Customer", "utterance": "Zero two one.", "dialogue_acts": [], "intents": []}, {"turn_id": "insurance_432_026", "speaker_role": "Agent", "utterance": "Zero two one.", "dialogue_acts": [], "intents": []}, {"turn_id": "insurance_432_027", "speaker_role": "Customer", "utterance": "Five two.", "dialogue_acts": [], "intents": []}, {"turn_id": "insurance_432_028", "speaker_role": "Customer", "utterance": "Sorry that last number is four and not two.", "dialogue_acts": [], "intents": []}, {"turn_id": "insurance_432_029", "speaker_role": "Agent", "utterance": "Ok, it's five four right?", "dialogue_acts": [], "intents": []}, {"turn_id": "insurance_432_030", "speaker_role": "Customer", "utterance": "That is correct.", "dialogue_acts": [], "intents": []}, {"turn_id": "insurance_432_031", "speaker_role": "Agent", "utterance": "What is the current plan you are enrolled in?", "dialogue_acts": [], "intents": []}, {"turn_id": "insurance_432_032", "speaker_role": "Customer", "utterance": "I'm currently enrolled in Automobile Insurance.", "dialogue_acts": [], "intents": []}, {"turn_id": "insurance_432_033", "speaker_role": "Agent", "utterance": "What plan sir?", "dialogue_acts": [], "intents": []}, {"turn_id": "insurance_432_034", "speaker_role": "Customer", "utterance": "I first did the basic plan for around one thousand dollars I think.", "dialogue_acts": [], "intents": []}, {"turn_id": "insurance_432_035", "speaker_role": "Agent", "utterance": "Yep, correct!", "dialogue_acts": [], "intents": []}, {"turn_id": "insurance_432_036", "speaker_role": "Customer", "utterance": "Then I increased it to that of two thousand.", "dialogue_acts": [], "intents": []}, {"turn_id": "insurance_432_037", "speaker_role": "Agent", "utterance": "You mean the Complete Auto.", "dialogue_acts": [], "intents": []}, {"turn_id": "insurance_432_038", "speaker_role": "Customer", "utterance": "Exactly!", "dialogue_acts": [], "intents": []}, {"turn_id": "insurance_432_039", "speaker_role": "Agent", "utterance": "Gotcha, just hold on while I get your policy number sen to your mobile phone.", "dialogue_acts": [], "intents": []}, {"turn_id": "insurance_432_040", "speaker_role": "Customer", "utterance": "Alright, take your time.", "dialogue_acts": [], "intents": []}, {"turn_id": "insurance_432_041", "speaker_role": "Agent", "utterance": "Huh, your policy number has being sent to your phone. Please confirm reciept.", "dialogue_acts": [], "intents": []}, {"turn_id": "insurance_432_042", "speaker_role": "Customer", "utterance": "Jeez, that was fast!", "dialogue_acts": [], "intents": []}, {"turn_id": "insurance_432_043", "speaker_role": "Agent", "utterance": "That is why we are the best at giving our customers utmost satisfaction.", "dialogue_acts": [], "intents": []}, {"turn_id": "insurance_432_044", "speaker_role": "Agent", "utterance": "speaking about the plan you wanted to enrol for. Can you tell me which particular one it is you want to enrol for?", "dialogue_acts": [], "intents": []}, {"turn_id": "insurance_432_045", "speaker_role": "Customer", "utterance": "Mhm, about that I will like to get a quote for the Life insurance first.", "dialogue_acts": ["InformIntent"], "intents": ["GetQuote"]}, {"turn_id": "insurance_432_046", "speaker_role": "Agent", "utterance": "Ok the, that won be an issue. We have the.", "dialogue_acts": [], "intents": []}, {"turn_id": "insurance_432_047", "speaker_role": "Customer", "utterance": "Can i get that sent to my mail or can I get the information on your website. It will afford me enough time to read understand it well. You know what the say about old age ?", "dialogue_acts": ["InformIntent"], "intents": []}, {"turn_id": "insurance_432_048", "speaker_role": "Agent", "utterance": "[Laughter] No problem, please provide me with your email address.", "dialogue_acts": [], "intents": []}, {"turn_id": "insurance_432_049", "speaker_role": "Customer", "utterance": "Bonnie W Wilson at dftmail dot com.", "dialogue_acts": [], "intents": []}, {"turn_id": "insurance_432_050", "speaker_role": "Agent", "utterance": "Gotcha, just a second.", "dialogue_acts": [], "intents": []}, {"turn_id": "insurance_432_051", "speaker_role": "Agent", "utterance": "The mail has being sent.", "dialogue_acts": [], "intents": []}, {"turn_id": "insurance_432_052", "speaker_role": "Customer", "utterance": "Thanks, for the past few decades of my existence, this is actually one of the best customer service I have ever heard, you are sure the best the company has.", "dialogue_acts": [], "intents": []}, {"turn_id": "insurance_432_053", "speaker_role": "Agent", "utterance": "We are all professionals sir, that is why we are the best around.", "dialogue_acts": [], "intents": []}, {"turn_id": "insurance_432_054", "speaker_role": "Customer", "utterance": "Whoa! I see.", "dialogue_acts": [], "intents": []}, {"turn_id": "insurance_432_055", "speaker_role": "Agent", "utterance": "Is there any other thing I can help you with?", "dialogue_acts": [], "intents": []}, {"turn_id": "insurance_432_056", "speaker_role": "Customer", "utterance": "Nah, that will be all for now. I will contact back when I'm done with the mail.", "dialogue_acts": [], "intents": []}, {"turn_id": "insurance_432_057", "speaker_role": "Agent", "utterance": "That will be great, thank you for chosing Rivertown Insurance. Bye!", "dialogue_acts": [], "intents": []}]}# run intent clustering (Task 1) baselines and evaluation
python3 -m sitod.run_experiment \
--data_root_dir dstc11 \
--experiment_root_dir results \
--config configs/run-intent-clustering-baselines.jsonnet- Best Performance (MiniLM-L12 & K-means)
| Model | NMI | ARI | ACC | Precision | Recall | F1 | Example Coverage | K |
|---|---|---|---|---|---|---|---|---|
| Ours | 63.1 | 38.9 | 54.9 | 68.0 | 54.9 | 60.8 | 100.0 | 31 |
- Experimental Result (Clustering algorithm: K-means)
| Model | NMI | ARI | ACC | Precision | Recall | F1 | Example Coverage | K |
|---|---|---|---|---|---|---|---|---|
| EASE-RoBERTa-Multilingual | 33.4 | 14.2 | 28.0 | 28.0 | 69.0 | 39.9 | 43.9 | 5 |
| EASE-BERT | 36.1 | 18.1 | 35.3 | 36.8 | 58.9 | 45.3 | 53.5 | 8 |
| EASE-BERT-Multilingual | 38.4 | 10.9 | 23.6 | 42.7 | 24.2 | 30.9 | 87.6 | 44 |
| EASE-RoBERTa | 45.8 | 25.4 | 40.0 | 43.7 | 60.7 | 50.9 | 65.6 | 12 |
| DiffCSE-BERT-Trans | 43.8 | 15.5 | 29.4 | 49.5 | 30.6 | 37.8 | 90.1 | 40 |
| DiffCSE-BERT-Sts | 46.6 | 16.9 | 29.9 | 50.4 | 30.8 | 38.2 | 89.9 | 43 |
| DiffCSE-RoBERTa-Sts | 53.9 | 29.0 | 45.1 | 57.2 | 46.8 | 51.5 | 91.1 | 30 |
| DiffCSE-RoBERTa-Trans | 55.0 | 23.1 | 35.7 | 60.6 | 35.7 | 44.9 | 96.7 | 50 |
| SimCSE-BERT-Unsupervised | 31.7 | 13.3 | 27.9 | 27.9 | 65.0 | 39.0 | 43.9 | 5 |
| SimCSE-BERT-Large-Unsupervised | 47.0 | 25.7 | 38.4 | 46.9 | 46.9 | 46.9 | 76.8 | 19 |
| SimCSE-RoBERTa-Unsupervised | 51.2 | 29.0 | 44.4 | 49.7 | 61.6 | 55.0 | 70.5 | 14 |
| SimCSE-BERT-Large-Supervised | 53.0 | 27.8 | 39.8 | 55.0 | 42.2 | 47.8 | 91.0 | 31 |
| SimCSE-BERT-Supervised | 53.1 | 24.4 | 39.0 | 60.5 | 39.5 | 47.8 | 96.3 | 44 |
| SimCSE-RoBERTa-Large-Unsupervised | 53.2 | 25.9 | 42.2 | 56.8 | 43.8 | 49.5 | 91.7 | 32 |
| SimCSE-RoBERTa-Supervised | 56.6 | 28.8 | 42.7 | 60.8 | 43.3 | 50.6 | 91.3 | 36 |
| SimCSE-RoBERTa-Large-Supervised | 56.8 | 28.9 | 41.3 | 62.5 | 41.6 | 49.9 | 98.4 | 42 |
| Glove-Avg | 30.5 | 7.0 | 20.6 | 34.6 | 22.2 | 27.0 | 92.2 | 50 |
| MPNet | 59.3 | 32.3 | 46.1 | 66.0 | 47.1 | 54.9 | 96.5 | 42 |
| MiniLM-L6 | 59.3 | 35.7 | 52.6 | 62.2 | 54.9 | 58.4 | 92.4 | 28 |
| MiniLM-MultiQA | 61.7 | 38.2 | 55.1 | 66.6 | 55.4 | 60.5 | 98.8 | 30 |
| MiniLM-L12 | 63.1 | 38.9 | 54.9 | 68.0 | 54.9 | 60.8 | 100.0 | 31 |
# run open intent induction (Task 2) baselines and evaluation
python3 -m sitod.run_experiment \
--data_root_dir dstc11 \
--experiment_root_dir results \
--config configs/run-open-intent-induction-baselines.jsonnet- Best Performance (MiniLM-MultiQA & Agglomerative)
| Model | NMI | ARI | ACC | Precision | Recall | F1 | Example Coverage | K |
|---|---|---|---|---|---|---|---|---|
| Ours | 81.0 | 64.2 | 66.5 | 75.5 | 80.6 | 78.0 | 86.2 | 25 |
- Experimental Result (Clustering algorithm: K-means)
| Model | NMI | ARI | ACC | Precision | Recall | F1 | Example Coverage | K |
|---|---|---|---|---|---|---|---|---|
| EASE-BERT-Multilingual | 27.0 | 12.3 | 23.9 | 24.1 | 72.9 | 36.2 | 26.4 | 5 |
| EASE-BERT | 53.1 | 25.6 | 42.8 | 50.2 | 57.4 | 53.5 | 80.0 | 26 |
| EASE-RoBERTa-Multilingual | 59.6 | 41.8 | 53.7 | 57.8 | 68.8 | 62.8 | 86.0 | 34 |
| EASE-RoBERTa | 60.5 | 50.7 | 52.7 | 55.8 | 70.0 | 62.1 | 83.1 | 28 |
| DiffCSE-BERT-Trans | 21.0 | 11.5 | 23.3 | 23.7 | 77.5 | 36.3 | 35.5 | 6 |
| DiffCSE-BERT-Sts | 49.6 | 22.3 | 40.2 | 46.1 | 58.4 | 51.5 | 93.2 | 31 |
| DiffCSE-RoBERTa-Sts | 65.4 | 42.3 | 53.7 | 59.4 | 65.9 | 62.5 | 89.8 | 31 |
| DiffCSE-RoBERTa-Trans | 65.4 | 53.3 | 57.6 | 63.1 | 71.1 | 66.8 | 90.0 | 28 |
| SimCSE-BERT-Large-Unsupervised | 33.1 | 19.1 | 28.0 | 28.0 | 77.5 | 41.2 | 33.4 | 5 |
| SimCSE-BERT-Unsupervised | 58.3 | 29.6 | 46.4 | 58.5 | 59.1 | 58.8 | 83.1 | 32 |
| SimCSE-BERT-Large-Supervised | 59.4 | 29.1 | 47.0 | 54.2 | 60.5 | 57.2 | 86.2 | 30 |
| SimCSE-RoBERTa-Supervised | 61.4 | 34.3 | 47.1 | 54.3 | 62.0 | 57.9 | 80.3 | 29 |
| SimCSE-BERT-Supervised | 62.7 | 37.3 | 52.6 | 63.4 | 59.8 | 61.6 | 89.6 | 34 |
| SimCSE-RoBERTa-Large-Unsupervised | 67.8 | 51.1 | 57.7 | 64.2 | 70.2 | 67.1 | 86.4 | 32 |
| SimCSE-RoBERTa-Unsupervised | 68.6 | 48.0 | 54.7 | 65.6 | 71.2 | 68.3 | 86.1 | 31 |
| SimCSE-RoBERTa-Large-Supervised | 69.2 | 38.7 | 52.1 | 62.4 | 70.0 | 66.0 | 93.8 | 32 |
| Glove-Avg | 35.0 | 18.8 | 29.1 | 36.3 | 51.2 | 42.4 | 77.0 | 29 |
| MPNet | 72.8 | 41.6 | 60.2 | 65.6 | 76.3 | 70.6 | 86.5 | 26 |
| MiniLM-L6 | 73.2 | 47.1 | 57.1 | 66.0 | 72.2 | 69.0 | 83.1 | 25 |
| MiniLM-MultiQA | 74.7 | 52.8 | 61.2 | 70.5 | 74.9 | 72.7 | 83.4 | 25 |
| MiniLM-L12 | 77.4 | 54.5 | 63.2 | 70.5 | 79.1 | 74.6 | 80.0 | 24 |
Yeah where do I go on here to pay my bill? 34
Hi, I need to speak to someone about my bill. 34
Okay. When will I see that in my bill? 34
Would it be on like any of my other bills? 34
Okay. Where at on the bill? 34
Oh okay. Where at on the bill. I'm looking now. 34
Thank you. How much will this make my bill go up? 34
Well I should probably pay my bill while I have you on the line. 34
How can I make payments going forward without calling in? 34
Hi. I need to check my bill? And maybe pay my bill? 34
Oh, man. Is there any other way I can check my bill and pay it? 34
So, my bill after this one won't be til-. 34
And when is that bill due every month? 34
September first alright. And I will get a bill in the mail for that? 34
Hi I am calling because I received a wrong bill. 34
And from here I can pay my bill online right? 34
Hi I need to change my billing address on my policy. 34
- Please consider citing our paper if it is of help to you ☺
@misc{https://doi.org/10.48550/arxiv.2212.02021,
doi = {10.48550/ARXIV.2212.02021},
url = {https://arxiv.org/abs/2212.02021},
author = {Jeiyoon Park, Yoonna Jang, Chanhee Lee and Heuiseok Lim},
title = {Analysis of Utterance Embeddings and Clustering Methods Related to Intent Induction for Task-Oriented Dialogue},
publisher = {arXiv},
year = {2022},
copyright = {arXiv.org perpetual, non-exclusive license}
}