Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Feat/merge ignored nounphrases lists #118

Merged
merged 8 commits into from
Mar 10, 2022
Merged

Feat/merge ignored nounphrases lists #118

Show file tree
Hide file tree
Changes from 5 commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
f3e9911
feat: merge ignored nounphrases
dilyararimovna Mar 9, 2022
6695a39
fix: codestyle
dilyararimovna Mar 9, 2022
baf061b
move ignore lists to separate dir and expose constants in init
mtalimanchuk Mar 9, 2022
e880e1e
fix codestyle
mtalimanchuk Mar 9, 2022
ba3652c
add docstring
mtalimanchuk Mar 9, 2022
a6fb485
add ./common mount to spacy-nounphrases config
mtalimanchuk Mar 9, 2022
4245bed
feat: common for spacy-nounphrases
dilyararimovna Mar 10, 2022
d478901
fix: import
dilyararimovna Mar 10, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
249 changes: 0 additions & 249 deletions annotators/fact_retrieval/remove_lists.py
View file
Open in desktop

This file was deleted.

7 changes: 4 additions & 3 deletions annotators/fact_retrieval/tfidf_ranker.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,8 @@
from deeppavlov.core.models.estimator import Component
from deeppavlov.core.common.file import read_json
from deeppavlov.core.commands.utils import expand_path
from remove_lists import NP_REMOVE_LIST, NP_IGNORE_LIST

from common.ignore_lists import FALSE_POS_NPS_LIST, BAD_NPS_LIST

logger = getLogger(__name__)

Expand All @@ -50,10 +51,10 @@ def __init__(
freq_unigrams = f.read().splitlines()[:1000]

self.np_ignore_expr = re.compile(
"(" + "|".join([r"\b%s\b" % word for word in NP_IGNORE_LIST + freq_unigrams]) + ")", re.IGNORECASE
"(" + "|".join([r"\b%s\b" % word for word in BAD_NPS_LIST + freq_unigrams]) + ")", re.IGNORECASE
)
self.np_remove_expr = re.compile(
"(" + "|".join([r"\b%s\b" % word for word in NP_REMOVE_LIST]) + ")", re.IGNORECASE
"(" + "|".join([r"\b%s\b" % word for word in FALSE_POS_NPS_LIST]) + ")", re.IGNORECASE
)
self.rm_spaces_expr = re.compile(r"\s\s+")

Expand Down
Loading