pre-processing of text #890 issue #990
Conversation
ashishgit7
changed the title
|
#890 added pre-processing of text |
| "wordseq = words(federalist)\n", | ||
| "wordseq = wordseq[114:-3098]" | ||
| "wordseqs = words(federalist)\n", | ||
| "wordseqs = wordseqs[114:-3098]" |
There was a problem hiding this comment.
Was it necessary to change the name of the variable?
There was a problem hiding this comment.
I haven't change the name if variable actually I have created a new variable
There was a problem hiding this comment.
Wasn't wordseq already present in the repository?
Anyway, its fine if it makes things simpler.
| "outputs": [], | ||
| "source": [ | ||
| "#removing stopwords\n", | ||
| "from nltk.corpus import stopwords\n", |
There was a problem hiding this comment.
We want to try to minimize the use of third-party libraries. The point of the nlp module is to have basic implementations of standard functions used in the domain. Importing from nltk is the opposite of what we want to do.
There was a problem hiding this comment.
Alright , I'll try to create new function in place of nltk library in my next contribution to minimize third-party library
| "source": [ | ||
| "#stemming and lemmatization\n", | ||
| "from nltk.stem.wordnet import WordNetLemmatizer\n", | ||
| "lmtzr = WordNetLemmatizer()\n", |
There was a problem hiding this comment.
Again, we shouldn't use lemmatizers from third parties. Instead, we could have a lemmatizer within the repository, however basic it may be. The point of this repository is to be able to explain the underlying concepts of these algorithms, not directly import from other modules.
There was a problem hiding this comment.
@sagar-sehgal We can, but make sure you read their license first. We might have to cite/acknowledge them. If the license allows, I think we can save a copy of the file in aima-data and carry on from there.
There was a problem hiding this comment.
Okay. I'll try to do that. Thank You!
| "metadata": {}, | ||
| "output_type": "execute_result" | ||
| } | ||
| ], | ||
| "source": [ | ||
| "' '.join(wordseq[:100])" | ||
| "' '.join(wordseqs[:100])" |
| "metadata": { | ||
| "collapsed": true | ||
| }, | ||
| "outputs": [], | ||
| "source": [ | ||
| "wordseq = [w for w in wordseq if w != 'publius']" | ||
| "wordseqs = [w for w in wordseqs if w != 'publius']" |
| @@ -531,7 +559,7 @@ | |||
| "(4, 16, 52)" | |||
| ] | |||
| }, | |||
| "execution_count": 6, | |||
| "execution_count": 41, | |||
There was a problem hiding this comment.
A slightly picky complaint, but you can rerun a notebook to serialize the execution counts.
No description provided.