base_nlp
--------

This package is meant to hold some basic nlp tools, and over time compete
with scikit-learn and nltk in speed (obviously not there yet).

Tutorial
--------

There are currently two modules: bag_of_words and cluster. Bag_of_words is 
undergoing experimentation with the tokenization for speed improvements,
and cluster currently only contains DBSCAN.

[bag_of_words]

The purpose of the bag_of_words module (henceforth called BOW) is to do exactly
what is sounds like, support bag of words representation of text documents. This
representation is helpful for document similarity and clustering analysis. To 
read more see: http://bit.ly/1qlKOtp

The BOW module is class focused, thus there is a definite work flow that can be
followed to keep track of your information. The main class is called 'Bow' and
this tutorial will focus on using it.

    1. The first step is to import the module and start a Bow object. You have
       several options as to how you want to use this object, whether with a sql
       query, a directory or a pandas dataframe. We will use a a directory in
       this example.

            $ipython
            In [1]: from base_nlp import bag_of_word as nlp
            In [2]: bow = nlp.Bow(method = 'dir', topdir = '/work/username/file/'
        
       Here we passed a method parameter that told the object we want to import
       from text files, and the location of those text files. This will then
       create an object we can work this call 'bow.'

    2. Next we will 'vectorize' our text. This will turn to text into 'tokens',
       which mean individual words, and then into a bag of words representation,
       using vectors.

            In [3]: bow.vectorizer(stem = True)

       Here use the 'stem' option, which will cut off the suffix of words for a
       better match (i.e.: Doorknob and Doorknobs will become the same word). 
       This is useful for accuracy and speed, but can hurt readability of the 
       text. Also notice that the vectorizer did not return the matrix or the
       vocab that we would expect. That is because they are stored inside the
       object, to get access to them call:

            In [4]: bow.word_matrix
            Out[4]: <1000x1000 sparse matrix ...>
            In [5]: bow.vocab
            Out[5]: {word1 : 1}...

    3. Now sometimes the size of vectors are too large so we can use rule based
       measures to remove some values.

            In [6]: bow.trim_vocab(max_word_pct = .3, max_vocab_size = 100)

       This will remove all words that appear in more than 30% of all documents
       and restrict the maximum vocabulary size to 100 words.

    4. The final thing we want to do is transform the documents into a tf-idf
       representation. This will weight the terms according to their relative
       importance, for more info see: http://bit.ly/1sIh6Re

           In [7]: bow.tfidf()

       And we are done! By calling bow.word_matrix, you can get acces to your 
       newly transformed data and use it to cluster or any number of things!
    
    5. There is further functionality in this module, such as gensim compatibil-
       ity and csv writers, and these functions are documented in the code. 

----------
Short Term
----------

    1. Investigate stemmer options, particularly Prter and Lovins

---------
Long Term
---------
    
    1. N-gram support
    2. Include tools that can take advantage of bag of words approach, like 
       topic modelling and latent semantic analysis
    3. Include native support for many different distance measures 
       (or just use scipy?)
    4. Cython?
    5. More advanced tokenization methods (Penn Treebank, Stanford)