Doc2Vec dropping sentences after training?

[craigpfeifer]

I'm finding that all training sentences & their labels do not occur in the final trained model

I create one sentence per line in my file, where each sentence has a unique key of "EDGE_"+line_number:

label = "EDGE_"+str(line_num)
all_labels.append(label)
a_sent = gensim.models.doc2vec.LabeledSentence(toks, [label])

model = gensim.models.Doc2Vec()
model.build_vocab(sent_list)
model.train(sent_list)

for a_label in all_labels:
if (model.contains(a_label):
print "Label present"

I build the vocab & train the model. Then I want to iterate over all the trained vectors from the labels, however I find that not all of labels are present in the model, based on the code above.

In my current dataset of 94,460 lines (sentences), only 70,818 of the lines are present in the model. My data set contains duplicate lines (34,789 lines are dupes) leaving 59,671 unique lines, however each label is unique.

[cscorley]

I think this might have to do with min_count. It filters out words (in the word2vec case) and documents (in the doc2vec case) that do not meet the count, so setting that to something lower than 5 should help.

[dineshbvadhia]

If "EDGE_"+line_number is the unique identifier (and doesn't appear in the actual content) then min_count=1 has to be set otherwise the labels won't be present in the model - is that correct?

[piskvorky]

@gojomo 's doc2vec improvements will solve this annoyance. Gordon is working on a cleaner API for doc2vec.

[gojomo]

The big docvec changes (which among other things make doc tags/indexes independent of the vocab-focused min_count) are ready for review in PR #356 .

[gojomo]

Closing as fixed by merge of #356 into develop.