Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Consider removing audio input option for NLU.DevOps #247

Open
rozele opened this issue Feb 3, 2020 · 1 comment
Open

Consider removing audio input option for NLU.DevOps #247

rozele opened this issue Feb 3, 2020 · 1 comment

Comments

@rozele
Copy link
Contributor

rozele commented Feb 3, 2020

My current thinking is that in order to support testing from audio input, we need to look at the end-state of a conversation, not the intermediate state of NLU results.

The main challenge here is described in #241

It begs the question on what we can really test from audio. While I believe it is still useful to test intents, it is challenging to test entities, and the only real way to validate that the correct entities were detected is to look at an end-state (such as a fully resolved / disambiguated entity from text).

It may be that we want to recommend an alternative of testing from labeled ASR generated text, rather than testing from the audio itself.

@saroup
Copy link
Contributor

saroup commented Feb 14, 2020

Are you suggesting here to remove support for audio completely ? Or keep it to test intents but stop support for entities
I like the idea of labeling the ASR generated text instead because there is still value in the comparison NLU.DevOps is providing. However, this would be measuring something slightly different. What if the expected text is :
"call eric on his cell" -> intent : Call_Person
And what we get from ASR is completely different for some reason like
"Caloric on his cell" -> how would we label this intent? would it still be "Call_Person" or "None"

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants