1.0.1

.gitignore

@@ -4,3 +4,4 @@
 .python-version
 .idea/
 _build/
+.DS_Store

README.md

@@ -7,16 +7,55 @@ resources, and regular expressions.
 
 ## Installation
 
-To install, you can use `pip`: ::
+To install, you can use `pip`: 
 
     pip install git+https://github.com/pewresearch/pewtils#egg=pewtils
 
-Or you can install from source: ::
+Or you can install from source: 
 
     git clone https://github.com/pewresearch/pewtils.git
     cd pewtils
     python setup.py install
 
+### Installation Troubleshooting
+
+#### Using 64-bit Python
+
+Some of our libraries require the use of 64-bit Python. If you encounter errors during installation \
+that are related to missing libraries, you may be using 32-bit Python. We recommend that you uninstall \
+this version and switch to a 64-bit version instead. On Windows, these will be marked with `x86-64`; you \
+can find the latest 64-bit versions of Python [here](https://www.python.org/downloads).
+
+#### Installing ssdeep
+
+ssdeep is an optional dependency that can be used by the `get_hash` function in Pewtils. \
+Installation instructions for various Linux distributions can be found in the library's \
+[documentation](https://python-ssdeep.readthedocs.io/en/latest/installation.html). The ssdeep \
+Python library is not currently compatible with Windows. \
+Installing ssdeep on Mac OS may involve a few additional steps, detailed below:
+
+1. Install Homebrew
+2. Install xcode
+    ```
+    xcode-select --install
+    ```
+3. Install system dependencies
+    ```
+    brew install pkg-config libffi libtool automake
+    ln -s /usr/local/bin/glibtoolize /usr/local/bin/libtoolize
+    ```
+4. Install ssdeep with an additional flag to build the required libraries
+	```
+    BUILD_LIB=1 pip install ssdeep
+    ```
+5. If step 4 fails, you may need to redirect your system to the new libraries by setting the following flags:
+    ```
+    export LIBTOOL=`which glibtool`
+    export LIBTOOLIZE=`which glibtoolize`
+    ```
+   Do this and try step 4 again.
+6. Now you should be able to run the main installation process detailed above.
+
 ## Documentation
 
 Please refer to the [official documentation](https://pewresearch.github.io/pewtils/) for information on how to use this package.

docs/_modules/pewtils/__init__.html

@@ -383,6 +383,10 @@ <h1>Source code for pewtils.__init__</h1><div class="highlight"><pre>
 <span class="sd">    3. By default the function uses the :py:mod:`ssdeep` algorithm, which generates context-sensitive hashes that are \</span>
 <span class="sd">    useful for computing document similarities at scale.</span>
 
+<span class="sd">    .. note:: Using `hash_function=&#39;ssdeep&#39;` requires the :py:mod:`ssdeep` library, which is not installed by default \</span>
+<span class="sd">    because it requires the installation of additional system libraries on certain operating systems. For help \</span>
+<span class="sd">    installing ssdeep, refer to the pewtils documentation installation section, which provides OS-specific instructions.</span>
+
 <span class="sd">    Usage::</span>
 
 <span class="sd">        from pewtils import get_hash</span>
@@ -403,8 +407,16 @@ <h1>Source code for pewtils.__init__</h1><div class="highlight"><pre>
     <span class="k">elif</span> <span class="n">hash_function</span> <span class="o">==</span> <span class="s2">&quot;md5&quot;</span><span class="p">:</span>
         <span class="n">hashed</span> <span class="o">=</span> <span class="n">md5</span><span class="p">(</span><span class="n">text</span><span class="p">)</span><span class="o">.</span><span class="n">hexdigest</span><span class="p">()</span>
     <span class="k">else</span><span class="p">:</span>
-        <span class="kn">import</span> <span class="nn">ssdeep</span>
-
+        <span class="k">try</span><span class="p">:</span>
+            <span class="kn">import</span> <span class="nn">ssdeep</span>
+        <span class="k">except</span> <span class="ne">ImportError</span><span class="p">:</span>
+            <span class="k">raise</span> <span class="ne">Exception</span><span class="p">(</span>
+                <span class="sd">&quot;&quot;&quot;</span>
+<span class="sd">                To use get_hash with hash_function=&#39;ssdeep&#39; you need to install the ssdeep package. Try running: </span>
+<span class="sd">                    &gt;&gt; BUILD_LIB=1 pip install ssdeep</span>
+<span class="sd">                If you encounter installation problems, refer to the pewtils documentation for troubleshooting help.</span>
+<span class="sd">            &quot;&quot;&quot;</span>
+            <span class="p">)</span>
         <span class="n">hashed</span> <span class="o">=</span> <span class="n">ssdeep</span><span class="o">.</span><span class="n">hash</span><span class="p">(</span><span class="n">text</span><span class="p">)</span>
 
     <span class="k">return</span> <span class="n">hashed</span></div>

docs/_sources/index.rst.txt

@@ -19,17 +19,74 @@ resources, and regular expressions.
 Installation
 ---------------
 
-To install, you can use ``pip``: ::
+To install, you can use ``pip``:
 
-    pip install git+https://github.com/pewresearch/pewtils#egg=pewtils
+    .. code-block:: bash
 
-Or you can install from source: ::
+        pip install git+https://github.com/pewresearch/pewtils#egg=pewtils
 
-    git clone https://github.com/pewresearch/pewtils.git
-    cd pewtils
-    python setup.py install
+Or you can install from source:
+
+    .. code-block:: bash
+
+        git clone https://github.com/pewresearch/pewtils.git
+        cd pewtils
+        python setup.py install
 
 .. note::
     This is a Python 3 package. Though it is compatible with Python 2, many of its dependencies are \
     planning to drop support for earlier versions if they haven't already. We highly recommend \
     you upgrade to Python 3.
+
+Installation Troubleshooting
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Using 64-bit Python
+""""""""""""""""""""
+
+Some of our libraries require the use of 64-bit Python. If you encounter errors during installation \
+that are related to missing libraries, you may be using 32-bit Python. We recommend that you uninstall \
+this version and switch to a 64-bit version instead. On Windows, these will be marked with ``x86-64``; you \
+can find the latest 64-bit versions of Python `here <http://www.python.org/downloads>`_.
+
+Installing ssdeep
+""""""""""""""""""""""""""""
+
+ssdeep is an optional dependency that can be used by the :py:func:`pewtils.get_hash` function in Pewtils. \
+Installation instructions for various Linux distributions can be found in the library's \
+`documentation <https://python-ssdeep.readthedocs.io/en/latest/installation.html>`_. The ssdeep \
+Python library is not currently compatible with Windows. \
+Installing ssdeep on Mac OS may involve a few additional steps, detailed below:
+
+1. Install Homebrew
+
+2. Install xcode
+
+    .. code-block:: bash
+
+        xcode-select --install
+
+3. Install system dependencies
+
+    .. code-block:: bash
+
+        brew install pkg-config libffi libtool automake
+        ln -s /usr/local/bin/glibtoolize /usr/local/bin/libtoolize
+
+4. Install ssdeep with an additional flag to build the required libraries
+
+    .. code-block:: bash
+
+        BUILD_LIB=1 pip install ssdeep
+
+5. If step 4 fails, you may need to redirect your system to the new libraries by setting the following flags:
+
+    .. code-block:: bash
+
+        export LIBTOOL=`which glibtool`
+        export LIBTOOLIZE=`which glibtoolize`
+
+    Do this and try step 4 again.
+
+6. Now you should be able to run the main installation process detailed above.
+

docs/index.html

@@ -167,19 +167,69 @@ <h1>Pewtils<a class="headerlink" href="#pewtils" title="Permalink to this headli
 <div class="section" id="installation">
 <h2>Installation<a class="headerlink" href="#installation" title="Permalink to this headline">¶</a></h2>
 <p>To install, you can use <code class="docutils literal notranslate"><span class="pre">pip</span></code>:</p>
-<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">pip</span> <span class="n">install</span> <span class="n">git</span><span class="o">+</span><span class="n">https</span><span class="p">:</span><span class="o">//</span><span class="n">github</span><span class="o">.</span><span class="n">com</span><span class="o">/</span><span class="n">pewresearch</span><span class="o">/</span><span class="n">pewtils</span><span class="c1">#egg=pewtils</span>
+<blockquote>
+<div><div class="highlight-bash notranslate"><div class="highlight"><pre><span></span>pip install git+https://github.com/pewresearch/pewtils#egg<span class="o">=</span>pewtils
 </pre></div>
 </div>
+</div></blockquote>
 <p>Or you can install from source:</p>
-<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">git</span> <span class="n">clone</span> <span class="n">https</span><span class="p">:</span><span class="o">//</span><span class="n">github</span><span class="o">.</span><span class="n">com</span><span class="o">/</span><span class="n">pewresearch</span><span class="o">/</span><span class="n">pewtils</span><span class="o">.</span><span class="n">git</span>
-<span class="n">cd</span> <span class="n">pewtils</span>
-<span class="n">python</span> <span class="n">setup</span><span class="o">.</span><span class="n">py</span> <span class="n">install</span>
+<blockquote>
+<div><div class="highlight-bash notranslate"><div class="highlight"><pre><span></span>git clone https://github.com/pewresearch/pewtils.git
+<span class="nb">cd</span> pewtils
+python setup.py install
 </pre></div>
 </div>
+</div></blockquote>
 <div class="admonition note">
 <p class="admonition-title">Note</p>
 <p>This is a Python 3 package. Though it is compatible with Python 2, many of its dependencies are planning to drop support for earlier versions if they haven’t already. We highly recommend you upgrade to Python 3.</p>
 </div>
+<div class="section" id="installation-troubleshooting">
+<h3>Installation Troubleshooting<a class="headerlink" href="#installation-troubleshooting" title="Permalink to this headline">¶</a></h3>
+<div class="section" id="using-64-bit-python">
+<h4>Using 64-bit Python<a class="headerlink" href="#using-64-bit-python" title="Permalink to this headline">¶</a></h4>
+<p>Some of our libraries require the use of 64-bit Python. If you encounter errors during installation that are related to missing libraries, you may be using 32-bit Python. We recommend that you uninstall this version and switch to a 64-bit version instead. On Windows, these will be marked with <code class="docutils literal notranslate"><span class="pre">x86-64</span></code>; you can find the latest 64-bit versions of Python <a class="reference external" href="http://www.python.org/downloads">here</a>.</p>
+</div>
+<div class="section" id="installing-ssdeep">
+<h4>Installing ssdeep<a class="headerlink" href="#installing-ssdeep" title="Permalink to this headline">¶</a></h4>
+<p>ssdeep is an optional dependency that can be used by the <code class="xref py py-func docutils literal notranslate"><span class="pre">pewtils.get_hash()</span></code> function in Pewtils. Installation instructions for various Linux distributions can be found in the library’s <a class="reference external" href="https://python-ssdeep.readthedocs.io/en/latest/installation.html">documentation</a>. The ssdeep Python library is not currently compatible with Windows. Installing ssdeep on Mac OS may involve a few additional steps, detailed below:</p>
+<ol class="arabic">
+<li><p>Install Homebrew</p></li>
+<li><p>Install xcode</p>
+<blockquote>
+<div><div class="highlight-bash notranslate"><div class="highlight"><pre><span></span>xcode-select --install
+</pre></div>
+</div>
+</div></blockquote>
+</li>
+<li><p>Install system dependencies</p>
+<blockquote>
+<div><div class="highlight-bash notranslate"><div class="highlight"><pre><span></span>brew install pkg-config libffi libtool automake
+ln -s /usr/local/bin/glibtoolize /usr/local/bin/libtoolize
+</pre></div>
+</div>
+</div></blockquote>
+</li>
+<li><p>Install ssdeep with an additional flag to build the required libraries</p>
+<blockquote>
+<div><div class="highlight-bash notranslate"><div class="highlight"><pre><span></span><span class="nv">BUILD_LIB</span><span class="o">=</span><span class="m">1</span> pip install ssdeep
+</pre></div>
+</div>
+</div></blockquote>
+</li>
+<li><p>If step 4 fails, you may need to redirect your system to the new libraries by setting the following flags:</p>
+<blockquote>
+<div><div class="highlight-bash notranslate"><div class="highlight"><pre><span></span><span class="nb">export</span> <span class="nv">LIBTOOL</span><span class="o">=</span><span class="sb">`</span>which glibtool<span class="sb">`</span>
+<span class="nb">export</span> <span class="nv">LIBTOOLIZE</span><span class="o">=</span><span class="sb">`</span>which glibtoolize<span class="sb">`</span>
+</pre></div>
+</div>
+<p>Do this and try step 4 again.</p>
+</div></blockquote>
+</li>
+<li><p>Now you should be able to run the main installation process detailed above.</p></li>
+</ol>
+</div>
+</div>
 </div>
 </div>
 

docs/pewtils_core.html

@@ -359,6 +359,10 @@ <h1>Core Functions<a class="headerlink" href="#core-functions" title="Permalink
 <p class="admonition-title">Note</p>
 <p>The string will be passed through <code class="xref py py-func docutils literal notranslate"><span class="pre">pewtils.decode_text()</span></code> and the returned value will be used     instead of the original value if it runs successfully, in order to ensure consistent hashing in both Python 2 and     3. By default the function uses the <code class="xref py py-mod docutils literal notranslate"><span class="pre">ssdeep</span></code> algorithm, which generates context-sensitive hashes that are     useful for computing document similarities at scale.</p>
 </div>
+<div class="admonition note">
+<p class="admonition-title">Note</p>
+<p>Using <cite>hash_function=’ssdeep’</cite> requires the <code class="xref py py-mod docutils literal notranslate"><span class="pre">ssdeep</span></code> library, which is not installed by default     because it requires the installation of additional system libraries on certain operating systems. For help     installing ssdeep, refer to the pewtils documentation installation section, which provides OS-specific instructions.</p>
+</div>
 <p>Usage:</p>
 <div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">pewtils</span> <span class="k">import</span> <span class="n">get_hash</span>
 

docs_source/index.rst

@@ -19,17 +19,74 @@ resources, and regular expressions.
 Installation
 ---------------
 
-To install, you can use ``pip``: ::
+To install, you can use ``pip``:
 
-    pip install git+https://github.com/pewresearch/pewtils#egg=pewtils
+    .. code-block:: bash
 
-Or you can install from source: ::
+        pip install git+https://github.com/pewresearch/pewtils#egg=pewtils
 
-    git clone https://github.com/pewresearch/pewtils.git
-    cd pewtils
-    python setup.py install
+Or you can install from source:
+
+    .. code-block:: bash
+
+        git clone https://github.com/pewresearch/pewtils.git
+        cd pewtils
+        python setup.py install
 
 .. note::
     This is a Python 3 package. Though it is compatible with Python 2, many of its dependencies are \
     planning to drop support for earlier versions if they haven't already. We highly recommend \
     you upgrade to Python 3.
+
+Installation Troubleshooting
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Using 64-bit Python
+""""""""""""""""""""
+
+Some of our libraries require the use of 64-bit Python. If you encounter errors during installation \
+that are related to missing libraries, you may be using 32-bit Python. We recommend that you uninstall \
+this version and switch to a 64-bit version instead. On Windows, these will be marked with ``x86-64``; you \
+can find the latest 64-bit versions of Python `here <http://www.python.org/downloads>`_.
+
+Installing ssdeep
+""""""""""""""""""""""""""""
+
+ssdeep is an optional dependency that can be used by the :py:func:`pewtils.get_hash` function in Pewtils. \
+Installation instructions for various Linux distributions can be found in the library's \
+`documentation <https://python-ssdeep.readthedocs.io/en/latest/installation.html>`_. The ssdeep \
+Python library is not currently compatible with Windows. \
+Installing ssdeep on Mac OS may involve a few additional steps, detailed below:
+
+1. Install Homebrew
+
+2. Install xcode
+
+    .. code-block:: bash
+
+        xcode-select --install
+
+3. Install system dependencies
+
+    .. code-block:: bash
+
+        brew install pkg-config libffi libtool automake
+        ln -s /usr/local/bin/glibtoolize /usr/local/bin/libtoolize
+
+4. Install ssdeep with an additional flag to build the required libraries
+
+    .. code-block:: bash
+
+        BUILD_LIB=1 pip install ssdeep
+
+5. If step 4 fails, you may need to redirect your system to the new libraries by setting the following flags:
+
+    .. code-block:: bash
+
+        export LIBTOOL=`which glibtool`
+        export LIBTOOLIZE=`which glibtoolize`
+
+    Do this and try step 4 again.
+
+6. Now you should be able to run the main installation process detailed above.
+

pewtils/__init__.py

@@ -232,6 +232,10 @@ def get_hash(text, hash_function="ssdeep"):
     3. By default the function uses the :py:mod:`ssdeep` algorithm, which generates context-sensitive hashes that are \
     useful for computing document similarities at scale.
 
+    .. note:: Using `hash_function='ssdeep'` requires the :py:mod:`ssdeep` library, which is not installed by default \
+    because it requires the installation of additional system libraries on certain operating systems. For help \
+    installing ssdeep, refer to the pewtils documentation installation section, which provides OS-specific instructions.
+
     Usage::
 
         from pewtils import get_hash
@@ -252,8 +256,16 @@ def get_hash(text, hash_function="ssdeep"):
     elif hash_function == "md5":
         hashed = md5(text).hexdigest()
     else:
-        import ssdeep
-
+        try:
+            import ssdeep
+        except ImportError:
+            raise Exception(
+                """
+                To use get_hash with hash_function='ssdeep' you need to install the ssdeep package. Try running: 
+                    >> BUILD_LIB=1 pip install ssdeep
+                If you encounter installation problems, refer to the pewtils documentation for troubleshooting help.
+            """
+            )
         hashed = ssdeep.hash(text)
 
     return hashed

requirements.txt

@@ -21,7 +21,6 @@ scipy>=1.4; python_version >= '3.0'
 simplejson>=3.10.0
 six>=1.14.0
 sphinx_rtd_theme>=0.4.3
-ssdeep>=3.3
 tldextract>=2.2.1
 tqdm>=4.11.2
 unicode>=2.7