Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Split large model files across multiple buffers. #29

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

jkinkead
Copy link

Fixes #28 by splitting large models into multiple buffers in memory.

Note on formatting: I used tabs throughout, as that seemed to be more common in the files. SearcherImpl was mixed-use. I also tried to keep for-loop (for( vs for () and naming similar to the local code. This was mixed-use as well.

@jkinkead jkinkead mentioned this pull request Aug 24, 2015
@dirkgr
Copy link
Contributor

dirkgr commented Sep 23, 2015

Ping?

@krishnad
Copy link

This fork breaks in the toBinFile method.
for(int i = 0; i < vocab.size(); ++i) {
out.write(String.format("%s ", vocab.get(i)).getBytes(cs));

        DoubleBuffer vectorBuffer = vectors[i / vectorsPerBuffer];

vectors ( with the plural) is a DoubleBuffer[] and is unlikely to be as large as the vocab size. In most of my test cases, vectors is a DoubleBuffer array of length 1.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging this pull request may close these issues.

3 participants