Replace open() with smart_open() in notebooks. Fix #1789 #1812

sharanry · 2017-12-23T12:14:13Z

Replace open() with smart_open() in notebooks to make it compatible with Python 3.x

menshikh-iv · 2017-12-26T04:47:17Z

Thanks for PR @sharanry, two questions:

This replaces all open calls?
Are you test than notebooks works correctly (after replace)?

sharanry · 2017-12-26T08:38:12Z

i have replaced all open calls in docs/notebooks directory.
What is the testing procedure? Should I run the notebooks locally?

menshikh-iv · 2017-12-26T09:11:51Z

@sharanry yes

sharanry · 2018-01-08T12:47:00Z

@menshikh-iv Could you review this?

menshikh-iv

Thanks @sharanry, are you already test it (i.e. re-run all notebooks locally with your changes)?

menshikh-iv · 2018-01-09T07:25:15Z

docs/notebooks/Corpora_and_Vector_Spaces.ipynb

    "class MyCorpus(object):\n",
    "    def __iter__(self):\n",
-    "        for line in open('datasets/mycorpus.txt'):\n",
+    "        for line in smart_open('datasets/mycorpus.txt'):\n",


Better to explicitly add mode, i.e. 'rb', 'r', etc.

sharanry · 2018-01-15T00:09:54Z

@menshikh-iv I made the changes which you had asked.
I have been trying to run the notebooks for the last couple of days. Some of the dependencies like nltk seem to be failing with the notebooks. I made sure they are all at the latest versions. Any suggestions?

I will be trying with docker image provided next.

menshikh-iv · 2018-01-15T06:29:04Z

@sharanry Also please merge develop from upstream to your branch, thanks.
That's all, let me know when you finished - I'll merge your PR.

…into smart_open * 'develop' of https://github.com/RaRe-Technologies/gensim: Fix positional params used for `gensim.models.CoherenceModel` in `gensim.models.callbacks` (piskvorky#1823) Fix parameter setting for `FastText.train`. Fix piskvorky#1818 (piskvorky#1837) Refactor tests for `gensim.corpora.WikiCorpus`(piskvorky#1821) Fix formula in `gensim.summarization.bm25`. Fix piskvorky#1828 (piskvorky#1833) Fix docstrings for `gensim.matutils` (piskvorky#1804) Fix docstrings for `gensim.models.logentropy_model` (piskvorky#1803) Fix docstrings for `gensim.models.normmodel` (piskvorky#1805) Refactor API reference `gensim.topic_coherence`. Fix piskvorky#1669 (piskvorky#1714) Add CircleCI for build documentation. Fix piskvorky#1807 (piskvorky#1822) Fix docstrings for `gensim.models.translation_matrix` (piskvorky#1806) Fix docstrings for `gensim.models.rpmodel` (piskvorky#1802) Fix docstrings for `gensim.utils` (piskvorky#1797) Fix tox.ini/setup.cfg configuration (piskvorky#1815) Add wordnet mammal train file for Poincare notebook (piskvorky#1781)

menshikh-iv · 2018-01-22T10:29:56Z

Hey @sharanry, how about the run of notebooks?

sharanry · 2018-01-22T14:26:33Z

hey @menshikh-iv, sorry for the late reply, I havent been able to build the docker image, i tried docker build -t gensim . as given here. I am getting this error.

menshikh-iv · 2018-01-23T05:43:46Z

@sharanry replace locales=2.23-0ubuntu9 from https://github.com/RaRe-Technologies/gensim/blob/09672b369b2ec94a06581efd060be20662d7006c/docker/Dockerfile#L28 to locales=2.23-0ubuntu10

menshikh-iv · 2018-02-05T08:31:00Z

@sharanry how is going? are you finished with running?

sharanry · 2018-02-05T12:35:58Z

@menshikh-iv Apologies for the delay. I am incurring multiple errors with the docker image. I am unable to set it up.

Is there any alternative way I could run the notebooks?

menshikh-iv · 2018-02-05T12:38:14Z

@sharanry what's kind of errors, can you show it here?

alternative way - locally on your machine.

sharanry · 2018-02-05T13:16:46Z

I even tried with different network connections.

sharanry · 2018-02-06T05:32:51Z

Finally got it to work somehow.
Got a bunch of errors in Poincare Evaluations

There errors are causing even more errors

sharanry · 2018-02-06T05:50:38Z

Getting this error even after manually downloading movie_plots.csv from the link mentioned. Must be a problem with docker.

menshikh-iv · 2018-02-06T06:13:46Z

Big thanks @sharanry, need to fix notebooks first (unfortunately, this doesn't work correctly).

sharanry · 2018-02-06T06:19:36Z

@menshikh-iv I have yet to check all the notebooks. I can help fixing them if possible.

menshikh-iv · 2018-02-06T06:42:25Z

@sharanry yes, it will be really nice, feel free to fix problems.

sharanry · 2018-02-14T17:34:22Z

@menshikh-iv Can we merge this?

menshikh-iv · 2018-02-15T15:34:30Z

@sharanry I want, but we need to run all of this notebooks first and check that all fine :(

UPD: let me check it myself

menshikh-iv · 2018-02-19T04:19:39Z

@sharanry it was much harder than I thought

Some notebooks need very long time for running
I see many py3/py2 incompatibilities -> need to run notebook 2 or 3 times (27, 35, 36).
Some problems with using (like - "let's go to download dataset from X" or using non-trivial dependencies).

I'll try to run, fix, merge this stuff when I have enough time for it (probably, not soon), sorry for waiting.

Also, please move commits from #1893 to current PR (anyway, I will not merge this until we fix it), also, this allows us to avoid painful merge-conflict with notebooks.

menshikh-iv · 2018-03-09T07:12:36Z

By discussion in #1964 (we'll rework most of the notebooks and provide small clear examples instead of most "long-running" stuff) + really hard to check (I spend a lot of time for it, but did not even check half, because of the need to test on 3 different python versions).

For this reason, I close current PR, sorry @sharanry, thanks for the work!

piskvorky · 2018-03-09T16:40:36Z

I see no reason to wait for some undefined future tests or refactorings just to change open to smart_open. These are valid but orthogonal concerns.

I propose merging this (useful) PR.

menshikh-iv · 2018-03-09T18:35:17Z

@piskvorky I close it because this needs a really long time to check that we didn't break something (and all works as expected) with this change.

P/S this thing is not so useful, without this modification everything works.

piskvorky · 2018-03-10T02:57:41Z

smart_open is a drop-in replacement for open, so your checking is more checking of the rest of the notebooks, not the changes in this PR itself.

I agree it may not be a critical feature, but it is a welcome one -- please merge if the changes themselves are OK.

piskvorky

Why these changes in binary/text semantics?

piskvorky · 2018-04-19T05:33:30Z

docs/notebooks/lda_training_tips.ipynb

@@ -67,7 +68,7 @@
    "    files = os.listdir(data_dir + yr_dir)\n",
    "    for filen in files:\n",
    "        # Note: ignoring characters that cause encoding errors.\n",
-    "        with open(data_dir + yr_dir + '/' + filen, errors='ignore') as fid:\n",
+    "        with smart_open(data_dir + yr_dir + '/' + filen, 'rb') as fid:\n",


Bug (not in this PR, notebook already bad): use os.path.join for joining filesystem paths.

Also, why the change in errors='ignore'?

piskvorky · 2018-04-19T05:34:03Z

docs/notebooks/topic_coherence-movies.ipynb

@@ -114,7 +115,7 @@
    "    # as well as pages about a single year.\n",
    "    # As a result, this preprocessing differs from the paper.\n",
    "    \n",
-    "    with open(os.path.join(data_dir, fname)) as f:\n",
+    "    with smart_open(os.path.join(data_dir, fname), 'rb') as f:\n",


This looks like a bug. If you open the file in binary mode, operations like split and lower have a different meaning, compared to text.

I see the same problem in many places in this PR.

Replace open() with smart_open()

725efed

Fix in Corpora_and_Vector_Spaces.ipynb

872c55c

menshikh-iv suggested changes Jan 9, 2018

View reviewed changes

Specify read/write explicitly while calling smart_open()

18a18f3

menshikh-iv changed the title ~~Replace open() with smart_open() in notebooks~~ Replace open() with smart_open() in notebooks. Fix #1789 Feb 1, 2018

sharanry mentioned this pull request Feb 10, 2018

Fix notebooks #1893

Closed

menshikh-iv mentioned this pull request Feb 26, 2018

open('aclImdb/alldata-id.txt', encoding='utf-8') doesn't work in Python 2.x #1789

Closed

menshikh-iv closed this Mar 9, 2018

menshikh-iv reopened this Mar 10, 2018

menshikh-iv merged commit 17f3719 into piskvorky:develop Mar 10, 2018

piskvorky reviewed Apr 19, 2018

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Replace open() with smart_open() in notebooks. Fix #1789 #1812

Replace open() with smart_open() in notebooks. Fix #1789 #1812

sharanry commented Dec 23, 2017

menshikh-iv commented Dec 26, 2017

sharanry commented Dec 26, 2017 •

edited

Loading

menshikh-iv commented Dec 26, 2017

sharanry commented Jan 8, 2018

menshikh-iv left a comment

menshikh-iv Jan 9, 2018

sharanry commented Jan 15, 2018 •

edited

Loading

menshikh-iv commented Jan 15, 2018

menshikh-iv commented Jan 22, 2018

sharanry commented Jan 22, 2018

menshikh-iv commented Jan 23, 2018

menshikh-iv commented Feb 5, 2018

sharanry commented Feb 5, 2018 •

edited

Loading

menshikh-iv commented Feb 5, 2018

sharanry commented Feb 5, 2018

sharanry commented Feb 6, 2018

sharanry commented Feb 6, 2018 •

edited

Loading

menshikh-iv commented Feb 6, 2018

sharanry commented Feb 6, 2018 •

edited

Loading

menshikh-iv commented Feb 6, 2018

sharanry commented Feb 14, 2018

menshikh-iv commented Feb 15, 2018 •

edited

Loading

menshikh-iv commented Feb 19, 2018

menshikh-iv commented Mar 9, 2018

piskvorky commented Mar 9, 2018 •

edited

Loading

menshikh-iv commented Mar 9, 2018 •

edited

Loading

piskvorky commented Mar 10, 2018 •

edited

Loading

piskvorky left a comment •

edited

Loading

piskvorky Apr 19, 2018 •

edited

Loading

piskvorky Apr 19, 2018

Replace open() with smart_open() in notebooks. Fix #1789 #1812

Replace open() with smart_open() in notebooks. Fix #1789 #1812

Conversation

sharanry commented Dec 23, 2017

menshikh-iv commented Dec 26, 2017

sharanry commented Dec 26, 2017 • edited Loading

menshikh-iv commented Dec 26, 2017

sharanry commented Jan 8, 2018

menshikh-iv left a comment

Choose a reason for hiding this comment

menshikh-iv Jan 9, 2018

Choose a reason for hiding this comment

sharanry commented Jan 15, 2018 • edited Loading

menshikh-iv commented Jan 15, 2018

menshikh-iv commented Jan 22, 2018

sharanry commented Jan 22, 2018

menshikh-iv commented Jan 23, 2018

menshikh-iv commented Feb 5, 2018

sharanry commented Feb 5, 2018 • edited Loading

menshikh-iv commented Feb 5, 2018

sharanry commented Feb 5, 2018

sharanry commented Feb 6, 2018

sharanry commented Feb 6, 2018 • edited Loading

menshikh-iv commented Feb 6, 2018

sharanry commented Feb 6, 2018 • edited Loading

menshikh-iv commented Feb 6, 2018

sharanry commented Feb 14, 2018

menshikh-iv commented Feb 15, 2018 • edited Loading

menshikh-iv commented Feb 19, 2018

menshikh-iv commented Mar 9, 2018

piskvorky commented Mar 9, 2018 • edited Loading

menshikh-iv commented Mar 9, 2018 • edited Loading

piskvorky commented Mar 10, 2018 • edited Loading

piskvorky left a comment • edited Loading

Choose a reason for hiding this comment

piskvorky Apr 19, 2018 • edited Loading

Choose a reason for hiding this comment

piskvorky Apr 19, 2018

Choose a reason for hiding this comment

sharanry commented Dec 26, 2017 •

edited

Loading

sharanry commented Jan 15, 2018 •

edited

Loading

sharanry commented Feb 5, 2018 •

edited

Loading

sharanry commented Feb 6, 2018 •

edited

Loading

sharanry commented Feb 6, 2018 •

edited

Loading

menshikh-iv commented Feb 15, 2018 •

edited

Loading

piskvorky commented Mar 9, 2018 •

edited

Loading

menshikh-iv commented Mar 9, 2018 •

edited

Loading

piskvorky commented Mar 10, 2018 •

edited

Loading

piskvorky left a comment •

edited

Loading

piskvorky Apr 19, 2018 •

edited

Loading