High Latency in the NER v3 (Chain of Thought)

[innocent-charles]

I have tried the NER v3 that including a chain of thought mechanism behind. But it seems that we i compare to other version(v2,v1, then v3 has higher latency that the response takes a lot of time to come and sometime failed.

[rmitsch]

Hi @innocent-charles, the higher latency is probably due to the LLM generating more output tokens due to the CoT mechanism. There is no way around this, I'm afraid - but you can still uses v2 of the NER recipe if you find it works well enough. Most more advanced prompting techniques come the disadvantage of requiring more output tokens and thus having more latency.

...and sometime failed.

Failed how?

[innocent-charles]

Yes, Thank you @rmitsch I get it. It failed because of time out, so I just increased the maximum number of retries...

[innocent-charles]

The problems that i was faced with version 2 is :

1. When the LLM returns the entity that is correct but in a different way as it was written in the document. That entity won't be shown up when you use for ent in doc.ents thing.
   Example : when for case of name, in the it can be like nyakana, innocent charles , so LLms can return the name as innocent charles nyakana... for this case when you for ent in doc.ents it won't show anything like not entity found while in the logs it has been returned from LLMs, this goes also for case the name is innocent, and LLMs has returned Innocent.

2. When the entity has already been extracted previous , spacy LLM won't show it again:
   Example: think of the case of resume and you want to extract educations deatails of that person, you might have the entity like college_name_one, college_name_two. So when in the resume let say someone has studied master degree at University of chicago, also studied bachelor degree at University of chicago , therefore i expect to have both entities like college_name_one = University of chicago, and college_name_two= university of chicago. The LLMs did that perfectly in the logs like they way i want, but when it comes to use spacy llm now for case of for ent in doc.ents i dont see the college_name_two i only see the college_name_one.

So have tried to think of that maybe it might be because of the repeated entities, that spacy llm when it encouter the value of entity has already been extracted it won't shown up even if that same value can be reffered as another entity.

[rmitsch]

Ah, right. I remember those bugs, they were fixed in NER.v3. So you would like to have a faster NER task that doesn't run have these bugs, right?

[innocent-charles]

Yes exactly ..since the problem with NER.v3 with CoT mechanism has introduced higher latency which in turn has affected the usability.

So would just ask or recommend if it is possible to just take the responses from LLM as they are returned. Since i think the problem might be in the spacy framework especially on how .ents work.

stand to be corrected if am wrong.....

[rmitsch]

Since i think the problem might be in the spacy framework especially on how .ents work.

No, the problem was rooted in how the NER.v1 and NER.v2 tasks parsed the LLM response. spaCy itself handles this correctly.
As for how to work around the response times:

• We'll look into a NER task version without CoT, but with the parsing fixes.
• The easiest way for you to get around this is to swap in a modified prompt template for the NER.v3 task. This is the default one for NER.v3 - feel free to experiment with it to reduce the token count.

[innocent-charles]

Thank you @rmitsch for the well explanation