Training an NER model to detect aliases

[BreezySMX]

Hello,

This is my first time working with spaCy, or any type of ML, for a project and I just wanted a quick sanity check to make sure what I think I understand, and what I'm doing is correct.

I'm trying to train spacy to better recognize ALIAS in data. I figured what better training data than super heroes, and used the following formats (please ignore any inaccuracies in the data, as far as Marvel trivia goes, I used a list of ending sentences and random.choice to pick one of the endings):

('Yvette (Hollow), is active in New York.', [(0, 6, 'NAME'), (8, 14, 'ALIAS')])
('Henry McCoy, who also goes by the name Beast, is conspiring against Daredevil.', [(0, 11, 'NAME'), (39, 44, 'ALIAS')])
('Tomi Shishido, aka, Gorgon,  current wherabouts and activities unknown.', [(0, 13, 'NAME'), (20, 26, 'ALIAS')])
('Raven Darkhölme, formerly known as Mystique, is alive.', [(0, 15, 'NAME'), (35, 43, 'ALIAS')])
('Eden Fesi, is also known as Manifold, is currently retired.', [(0, 9, 'NAME'), (28, 36, 'ALIAS')])

With 173 heroes and 6 variations of how aliases are seen in text, (prob more but this is all I could think of) I ended up 1038 sets of training data.

I used the following to create my training data:

import spacy
from spacy.tokens import DocBin

nlp = spacy.blank("en")
db = DocBin()
for text, annotations in sents:
    doc = nlp(text)
    ents = []
    for start, end, label in annotations:
        span = doc.char_span(start, end, label=label)
        ents.append(span)
    doc.ents = ents
    db.add(doc)

db.to_disk("./train.spacy")

and then ran the following from the command line:

spacy train config.cfg --paths.train="train.spacy" --paths.dev="dev" --gpu-id 0 --output="output_folder"

once the training finished I ran the model again against another document that didn't contain any of the names or aliases in my training data with the following:

import spacy
from spacy import displacy

nlp = spacy.load("output_folder\model-last")
with open('0.txt') as f:
    file= f.read()
    doc = nlp(file)

html = displacy.serve(doc, style='ent',page=True)

And it worked, it successfully identified names and aliases in the data.

My next step is to use the tutorial found here and extract the relationships between the name and alias entity. Unfortunately, the data used for the training comes from Prodigy, which I don't have the means to purchase, so I'm left trying to turn my data into the same format as them, or find an alternate way to do the relationship extraction.

Am I on the right track for this? or did I just get lucky?

Thanks for the feedback :).

[ljvmiranda921]

Hi @BreezySMX ,

Interesting project :) You might need to consider what the model is learning. Since NER is heavily dependent on context, your model may be learning the boundary words such as 'a.k.a.', 'also known as,' etc. There might even be an argument for using business rules here (since NAMES and ALIAS can be many-to-one). For example, you can make a two-stage approach of using NER to detect PERSON entities and then match them to a list of known aliases.

As a learning experiment, I think the flow is good. However, it's unclear if you're using actual dev data. You shouldn't need to use extra data to check if it works at all if you have dev data. Lastly, the output_folder also contains model-best. Sometimes, the model produced during the last iteration is not the best-performing one. You can try swapping it out and observing the results.

[BreezySMX]

Thanks for the feedback @ljvmiranda921!

No, I'm not using any dev data. That was a mistake on my part when trying to get the model to run.

Since NER is heavily dependent on context, your model may be learning the boundary words such as 'a.k.a.', 'also known as,' etc. There might even be an argument for using business rules here (since NAMES and ALIAS can be many-to-one). For example, you can make a two-stage approach of using NER to detect PERSON entities and then match them to a list of known aliases.

I understand what you mean by boundary words, and this was a concern of mine, which is why I tried to include as many examples in my training data as I could. With regards to your statement about (since NAMES and ALIAS can be many-to-one), do you mean, for example, how there are thousands of John Smiths, and only one uses the alias J Smith, and I could accidently relate the alias to ANOTHER John Smith? If so, thanks, I hadn't considered that a possibility.

I have another model which uses specific organization names to create entities, which is similar to the two stage approach you suggested. This issue is, I don't always know the aliases.

[ljvmiranda921]

I understand what you mean by boundary words, and this was a concern of mine, which is why I tried to include as many examples in my training data as I could.

One way to "sanity-check" your model is to show it examples of text with different structures. So far the texts in your original post follow a pattern of "NAME, (boundary) ALIAS (boundary) (additional text)." You can try something like news reports, or a piece of text from a literary work ("Spiderman took off his mask, and they saw Peter Parker"), etc. Of course this also depends on the domain of your data, but that's one way to test it.

do you mean, for example, how there are thousands of John Smiths, and only one uses the alias J Smith, and I could accidently relate
the alias to ANOTHER John Smith?

Yup! That and also there could be multiple PERSON entities related to an ALIAS. For example, there are multiple Spider-men ("Miles Morales", "Peter Parker", etc). I think it's possible to scrape these relationships from a website somewhere and use that for matching? So you just probably need to look for names ("Peter Parker", "Miles Morales") and then use the mapping to get to their alias.

[BreezySMX]

So, I'm using this against criminal organizations. I used the super hero name/alias relationships for training because I thought it would be comparable to what I'm using it for, that isn't word for word my data. I based the boundaries on what I found using the content from 100 news articles.

In other words I didn't want to train it on real criminal name/aliases and then nlp(data) the same information for Spacy to turn around and tell me, what I already know. I did use this training set and 'sanity checked' it as you suggested against real news articles when I initially finished the training earlier this week, and it did pick up names and aliases. It also identified names that did not have an alias near the name, which I thought was promising, considering I trained it with the blank model. However, it did not encounter an alias alone, and I reckon it would have difficulty.

So, with that in mind, I took your feedback and added some additional sentences to my test data to see the results of my training. Here, you can see it working as you would think:

And these are the results at the end of the data:

It keeps the alias tag, but only until it sees the pattern again. Would inducing more training which has the reverse, "ALIAS, (boundary) Name (boundary) (additional text)." help? I assume, based on my limited knowledge of NLP/SpaCy, the differences in lemma's would help SpaCy determine if it is a name first or alias first. I could see an issue where "ALIAS, whose brothers name is "NAME", (boundary)", might complicate matters in the future, as I hope to develop a relationship extractor which relates the name to the alias, and it might accidently relate the brothers name to the alias, but I might be really overthinking this.

I'm also having issues with the relationship extractor from the tutorial I mentioned in my initial post. Should I post questions here? or open a new thread? Thanks again for all your help. No one on my team has any experience with NLP/SpaCy, so I'm diving head first, going through the docs and any tutorials I find on YouTube/Udemy.

[ljvmiranda921]

It's fine to continue the thread here :) From the examples you've shown, it seems that the ALIAS is determined by the sentence structure rather than the span. So going with the relation extractor using the generic PERSON entities (you don't have to create a new NAME entity for this one, some models can already detect PERSON entities) should do the trick.