Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix AttributeError in WikiCorpus #2901

Merged
merged 2 commits into from
Aug 17, 2020
Merged

Fix AttributeError in WikiCorpus #2901

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
42d2df4
bug fix: wikicorpus getstream from data file-path \n Replace fname wi…
Jul 27, 2020
3feeb2b
refactor: use property decorator for input
Aug 16, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions gensim/corpora/wikicorpus.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -620,7 +620,7 @@ def __init__(self, fname, processes=None, lemmatize=utils.has_pattern(), diction
Unless a dictionary is provided, this scans the corpus once, to determine its vocabulary.

"""
self.fname = fname
self.input = fname
Copy link
Owner

@piskvorky piskvorky Jul 27, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we rename the method param too? That would break API (unless we keep both input and fname params) but maybe a little cleaner. @mpenkov WDYT?

self.filter_namespaces = filter_namespaces
self.filter_articles = filter_articles
self.metadata = False
Expand Down Expand Up @@ -677,7 +677,7 @@ def get_texts(self):
texts = \
((text, self.lemmatize, title, pageid, tokenization_params)
for title, text, pageid
in extract_pages(bz2.BZ2File(self.fname), self.filter_namespaces, self.filter_articles))
in extract_pages(bz2.BZ2File(self.input), self.filter_namespaces, self.filter_articles))
pool = multiprocessing.Pool(self.processes, init_to_ignore_interrupt)

try:
Expand Down
5 changes: 5 additions & 0 deletions gensim/test/test_corpora.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -769,6 +769,11 @@ def test_removed_table_markup(self):
for word in table_markup:
self.assertTrue(word not in text)

def test_get_stream(self):
wiki = self.corpus_class(self.enwiki)
sample_text_wiki = next(wiki.getstream()).decode()[1:14]
self.assertEqual(sample_text_wiki, "mediawiki xml")

# #TODO: sporadic failure to be investigated
# def test_get_texts_returns_generator_of_lists(self):
# corpus = self.corpus_class(self.enwiki)
Expand Down