Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Optimizations to tfidf backend training #335

Merged
merged 10 commits into from
Oct 4, 2019
Merged

Optimizations to tfidf backend training #335

merged 10 commits into from
Oct 4, 2019

Commits on Sep 26, 2019

  1. Speed up conversion of document to subject corpus using buffered writing

    @osma
    osma committed Sep 26, 2019
    Configuration menu
    Copy the full SHA
    a4687d4 View commit details
    Browse the repository at this point in the history

Commits on Oct 3, 2019

  1. Also delay file creation when buffering corpus conversion

    @osma
    osma committed Oct 3, 2019
    Configuration menu
    Copy the full SHA
    c01c88d View commit details
    Browse the repository at this point in the history

  2. pre-transform document corpus to subject corpus before vectorizing - for 

    reasons I don't quite understand this brings a small performance boost
    @osma
    osma committed Oct 3, 2019
    Configuration menu
    Copy the full SHA
    466dd5f View commit details
    Browse the repository at this point in the history

  3. add a small cache to Analyzer.is_valid_token() to improve performance

    @osma
    osma committed Oct 3, 2019
    Configuration menu
    Copy the full SHA
    ea6dbeb View commit details
    Browse the repository at this point in the history

Commits on Oct 4, 2019

  1. Merge branch 'master' into buffered-subject-conversion

    @osma
    osma committed Oct 4, 2019
    Configuration menu
    Copy the full SHA
    be2c44c View commit details
    Browse the repository at this point in the history

  2. Move subject vectorizer handling inside tfidf backend, as no other ba… 

    …ckend

needs it and it is unlikely other backends will need it in the future
    @osma
    osma committed Oct 4, 2019
    Configuration menu
    Copy the full SHA
    65033fa View commit details
    Browse the repository at this point in the history

  3. Use TfidfVectorizer.fit_transform as it is more efficient than separa… 

    …te fit

and transform steps
    @osma
    osma committed Oct 4, 2019
    Configuration menu
    Copy the full SHA
    b81f0c7 View commit details
    Browse the repository at this point in the history

  4. Split up TFIDFBackend.initialize

    @osma
    osma committed Oct 4, 2019
    Configuration menu
    Copy the full SHA
    6f49b21 View commit details
    Browse the repository at this point in the history

  5. use named constant instead of hardcoded value

    @osma
    osma committed Oct 4, 2019
    Configuration menu
    Copy the full SHA
    b6dbbcd View commit details
    Browse the repository at this point in the history

  6. Split up TFIDFBackend.train

    @osma
    osma committed Oct 4, 2019
    Configuration menu
    Copy the full SHA
    0c6ee3b View commit details
    Browse the repository at this point in the history