[Dy2stat] Add word2vec as unittest #24944
Conversation
test=develop
|
Thanks for your contribution! |
|
❌This PR is not created using PR's template. You can refer to this Demo. |
| indices = indices[np.argsort(-flat[indices])] | ||
| for i in indices: # Remove the input words | ||
| print('for word %s, the similar word is %s' % | ||
| (query_token, str(id2word_dict[i]))) |
There was a problem hiding this comment.
It seems that get_similar_tokens is not used in the unittest.
There was a problem hiding this comment.
This is copied code so I didn't care much about this. The get_similar_tokens is used in the original code but it now cannot run because of API compatibility issue. I will remove it in this unittest.
|
|
||
|
|
||
| #for _, batch in zip(range(10), build_batch(dataset, 128, 3)): | ||
| # print(batch) |
There was a problem hiding this comment.
The code is at the original code. I commented it so that we don't print much info. I will remove line 208-209.
|
|
||
| for epoch in range(epoch_num): | ||
|
|
||
| #random.shuffle(dataset) |
There was a problem hiding this comment.
Is this commented code necessary?
There was a problem hiding this comment.
No, the commented code is in original code. I commented because the random.shuffle can cause the input to dygraph/static be different thus our check can fail.
test=develop
PR types
Others
PR changes
Others
Describe
Add word2vec as ProgramTranslator unit test。
The word2vec model is referred from https://github.com/PaddlePaddle/models/blob/856c428fe0f1f451f8a22acee50c2e3af33c834e/dygraph/word2vec/word2vec.py#L238