Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Further optimizations to tfidf backend + rearchitecting #336

Merged
merged 6 commits into from
Oct 7, 2019
Merged

Further optimizations to tfidf backend + rearchitecting #336

merged 6 commits into from
Oct 7, 2019

Commits on Oct 4, 2019

  1. Remove dead code: conversion from SubjectCorpus to DocumentCorpus was… 

    … only used by a unit test, not real code
    @osma
    osma committed Oct 4, 2019
    Configuration menu
    Copy the full SHA
    01d1c43 View commit details
    Browse the repository at this point in the history

  2. Move the conversion from document to subject corpus entirely inside T… 

    …FIDFBackend since nothing else uses it
    @osma
    osma committed Oct 4, 2019
    Configuration menu
    Copy the full SHA
    987c6dd View commit details
    Browse the repository at this point in the history

  3. Perform document to subject conversion in memory; remove stale Subjec… 

    …tDirectory class
    @osma
    osma committed Oct 4, 2019
    Configuration menu
    Copy the full SHA
    2cb42da View commit details
    Browse the repository at this point in the history

  4. Tokenize text during conversion to subject corpus instead of within 

    TfidfTransformer, to avoid tokenizing the same text many times if it has
multiple subjects
    @osma
    osma committed Oct 4, 2019
    Configuration menu
    Copy the full SHA
    3cc830b View commit details
    Browse the repository at this point in the history

Commits on Oct 7, 2019

  1. Spool large subject texts into files instead of keeping everything in… 

    … memory
    @osma
    osma committed Oct 7, 2019
    Configuration menu
    Copy the full SHA
    fe4bdc5 View commit details
    Browse the repository at this point in the history

  2. Cleanup unused imports

    @osma
    osma committed Oct 7, 2019
    Configuration menu
    Copy the full SHA
    3238047 View commit details
    Browse the repository at this point in the history