Ruby Vector Space Model (VSM) with tf*idf weights

Calculates the similarity between texts using a bag-of-words Vector Space Model with Term Frequency-Inverse Document Frequency (tf*idf) weights. If your use case demands performance, use Lucene or similar (see below).

Usage

require 'tf-idf-similarity'

corpus = TfIdfSimilarity::Collection.new
corpus << TfIdfSimilarity::Document.new("Lorem ipsum dolor sit amet...")
corpus << TfIdfSimilarity::Document.new("Pellentesque sed ipsum dui...")
corpus << TfIdfSimilarity::Document.new("Nam scelerisque dui sed leo...")

p corpus.similarity_matrix

The following methods accept a {function: :bm25} options hash to use the Okapi BM25 ranking function instead of tf*idf:

• term_frequency
• inverse_document_frequency
• term_frequency_inverse_document_frequency
• similarity_matrix

Read the documentation at RubyDoc.info.

Optimizations

This gem will use the first available library below, for faster matrix multiplication.

GNU Scientific Library (GSL)

gem install gsl

NArray

gem install narray

NMatrix

The nmatrix gem gives access to Automatically Tuned Linear Algebra Software (ATLAS), which you may know of through Linear Algebra PACKage (LAPACK) or Basic Linear Algebra Subprograms (BLAS). Follow these instructions to install the nmatrix gem. You may need additional instructions for Mac OS X Lion.

Extras

You can access more term frequency, document frequency, and normalization formulas with:

require 'tf-idf-similarity/extras/collection'
require 'tf-idf-similarity/extras/document'

The default tf*idf formula follows the Lucene Conceptual Scoring Formula.

Why?

At the time of writing, no other Ruby gem implemented the tf*idf formula used by Lucene, Sphinx and Ferret.

• rsemantic now uses the same term frequency and document frequency formulas as Lucene.
• treat offers many term frequency formulas, one of which is the same as Lucene.
• similarity uses cosine normalization, which corresponds roughly to Lucene.

Term frequencies

The vss gem does not normalize the frequency of a term in a document; this occurs frequently in the academic literature, but only to demonstrate why normalization is important. The tf_idf and similarity gems normalize the frequency of a term in a document to the number of terms in that document, which never occurs in the literature. The tf-idf gem normalizes the frequency of a term in a document to the number of unique terms in that document, which never occurs in the literature.

Document frequencies

The vss gem does not normalize the inverse document frequency. The treat, tf_idf, tf-idf and similarity gems use variants of the typical inverse document frequency formula.

Normalization

The treat, tf_idf, tf-idf, rsemantic and vss gems have no normalization component.

Additional adapters

Adapters for the following projects were also considered:

• Ruby-LAPACK is a very thin wrapper around LAPACK, which has an opaque Fortran-style naming scheme.
• Linalg and RNum give access to LAPACK from Ruby, but are old and unavailable as gems.

Reference

• G. Salton and C. Buckley. "Term Weighting Approaches in Automatic Text Retrieval."" Technical Report. Cornell University, Ithaca, NY, USA. 1987.
• E. Chisholm and T. G. Kolda. "New term weighting formulas for the vector space method in information retrieval." Technical Report Number ORNL-TM-13756. Oak Ridge National Laboratory, Oak Ridge, TN, USA. 1999.

Further Reading

Lucene implements many more similarity functions, such as:

• a divergence from randomness (DFR) framework
• a framework for the family of information-based models
• a language model with Bayesian smoothing using Dirichlet priors
• a language model with Jelinek-Mercer smoothing

Lucene can even combine similarity measures.

Bugs? Questions?

This gem's main repository is on GitHub: http://github.com/opennorth/tf-idf-similarity, where your contributions, forks, bug reports, feature requests, and feedback are greatly welcomed.

Copyright (c) 2012 Open North Inc., released under the MIT license