Cross-lingual news article comparison system using bi-graph clustering and Siamese LSTM

Desription Given two article, in Japanese and English respectively, calculate the simialrity between them. Basing on the paper Mueller, J and Thyagarajan, A. Siamese Recurrent Architectures for Learning Sentence Similarity as well as the research of mine.

Data Training data: 1000 "relative" Japanese-English news reports pairs + 1000 "unrelavant" Japanese-English news reports pairs Testing data: 1000 Japanese individual reprots and 1000 English individual reports, generated basing on 1000 "relative" Japanese-English news reports

Files: *.ipynb Testing script

lstm.py Modified basing on the description and codes from the paper Mueller, J and Thyagarajan, A. Siamese Recurrent Architectures for Learning Sentence Similarity. Adjust for the multilingual cases.

sentences.py Embedding the words(cluster number )appeared in the original

main.py Core scripts. Convert the origianl news articles to the series of multilingual cluster numbers. And then generate training data and testing data basing on the rules.

./pickles Containing preprocessed training data and testing data

./weights Containing the trained model for MaLSTM e5/e10 refers to the epoches times 1k1k refers to the number of "relative" training pairs (similairty =1) 1k and "unrelavant" training pairs 1k (similairty = 0)

Ignored files: ./log Log histories

./data Dataset and results for multilingual bi-graph clustering system (see my other paper published soon)

Name		Name	Last commit message	Last commit date
Latest commit History 24 Commits
cl-model		cl-model
data_baseline		data_baseline
data_preparation		data_preparation
data_trans		data_trans
log		log
pickles		pickles
pickles_wrong		pickles_wrong
weights		weights
.gitignore		.gitignore
2016-11-21(0).ipynb		2016-11-21(0).ipynb
2016-11-21(1).ipynb		2016-11-21(1).ipynb
2016-11-23(0).ipynb		2016-11-23(0).ipynb
2016-11-23(1).ipynb		2016-11-23(1).ipynb
2016-12-12(1).ipynb		2016-12-12(1).ipynb
2016-12-13(1).ipynb		2016-12-13(1).ipynb
2016-12-14(1).ipynb		2016-12-14(1).ipynb
2016-12-14(2).ipynb		2016-12-14(2).ipynb
2017-1-10(1).ipynb		2017-1-10(1).ipynb
2017-1-15(1).ipynb		2017-1-15(1).ipynb
2017-1-16(1).ipynb		2017-1-16(1).ipynb
2017-1-22(1).ipynb		2017-1-22(1).ipynb
2017-1-3(1).ipynb		2017-1-3(1).ipynb
2017-1-4(1).ipynb		2017-1-4(1).ipynb
2017-1-5(1).ipynb		2017-1-5(1).ipynb
2017-1-9(1).ipynb		2017-1-9(1).ipynb
2017-2-20(1) top1和top5文章配对{错误}.ipynb		2017-2-20(1) top1和top5文章配对{错误}.ipynb
2017-2-20(2).ipynb		2017-2-20(2).ipynb
2017-2-21(1).ipynb		2017-2-21(1).ipynb
2017-2-22(1) 改变batchsize.ipynb		2017-2-22(1) 改变batchsize.ipynb
2017-2-22(2) 跨语言回归.ipynb		2017-2-22(2) 跨语言回归.ipynb
2017-2-23(1)跨语言回归模型(2).ipynb		2017-2-23(1)跨语言回归模型(2).ipynb
2017-2-26(1) 找出top1和top5一下的文章对.ipynb		2017-2-26(1) 找出top1和top5一下的文章对.ipynb
2017-2-27(1) 重新训练batch=64并作图.ipynb		2017-2-27(1) 重新训练batch=64并作图.ipynb
2017-2-9(1).ipynb		2017-2-9(1).ipynb
2017-3-18 proj_baseline 是用全部数据测试.ipynb		2017-3-18 proj_baseline 是用全部数据测试.ipynb
2017-3-18 proj_baseline 测试 batch=64, epochs=50.ipynb		2017-3-18 proj_baseline 测试 batch=64, epochs=50.ipynb
2017-4-10 谷歌翻译(2).ipynb		2017-4-10 谷歌翻译(2).ipynb
2017-4-11 谷歌翻译3.ipynb		2017-4-11 谷歌翻译3.ipynb
2017-4-9 谷歌翻译.ipynb		2017-4-9 谷歌翻译.ipynb
2017-5-11 BOW-baseline-1.ipynb		2017-5-11 BOW-baseline-1.ipynb
2017-5-12 unbalanced LSTM.ipynb		2017-5-12 unbalanced LSTM.ipynb
2017-5-13 doc2vec+svm.ipynb		2017-5-13 doc2vec+svm.ipynb
2017-5-20 论文素材准备.ipynb		2017-5-20 论文素材准备.ipynb
2017-7-10 使用embedding层数来进行学习.ipynb		2017-7-10 使用embedding层数来进行学习.ipynb
2017-7-11 使用svm训练title数据，并获得rank评价.ipynb		2017-7-11 使用svm训练title数据，并获得rank评价.ipynb
2017-7-11 提高lstm的精确度测试.ipynb		2017-7-11 提高lstm的精确度测试.ipynb
2017-7-13 尝试svm的并行计算.ipynb		2017-7-13 尝试svm的并行计算.ipynb
2017-7-14 全新测试lstm的.ipynb		2017-7-14 全新测试lstm的.ipynb
2017-7-15 比较lstm.ipynb		2017-7-15 比较lstm.ipynb
2017-7-15 比较svm.ipynb		2017-7-15 比较svm.ipynb
2017-7-16 使用了embedding layers的lstm结果.ipynb		2017-7-16 使用了embedding layers的lstm结果.ipynb
2017-7-16 比较svm-2.ipynb		2017-7-16 比较svm-2.ipynb
2017-7-23 论文数据库资料.ipynb		2017-7-23 论文数据库资料.ipynb
2017-7-31 实验数据收集：短文章baseline-lstm.ipynb		2017-7-31 实验数据收集：短文章baseline-lstm.ipynb
2017-7-31 实验数据收集：短文章lstm.ipynb		2017-7-31 实验数据收集：短文章lstm.ipynb
2017-7-31 实验数据收集：短文章svm.ipynb		2017-7-31 实验数据收集：短文章svm.ipynb
2017-7-31 实验数据收集：长文章baseline-lstm.ipynb		2017-7-31 实验数据收集：长文章baseline-lstm.ipynb
2017-7-31 实验数据收集：长文章lstm.ipynb		2017-7-31 实验数据收集：长文章lstm.ipynb
2017-7-31 实验数据收集：长文章svm.ipynb		2017-7-31 实验数据收集：长文章svm.ipynb
2017-7-5 提取新闻标题.ipynb		2017-7-5 提取新闻标题.ipynb
2017-7-5 新闻标题前处理.ipynb		2017-7-5 新闻标题前处理.ipynb
2017-7-6 使用head来做keras网络训练.ipynb		2017-7-6 使用head来做keras网络训练.ipynb
2017-7-6 使用quick评价rank方法来评价模型.ipynb		2017-7-6 使用quick评价rank方法来评价模型.ipynb
2017-8-6 获取word2vec案例数据.ipynb		2017-8-6 获取word2vec案例数据.ipynb
2107-7-6 CPU的多线程评价 [失败].ipynb		2107-7-6 CPU的多线程评价 [失败].ipynb
README.md		README.md
Untitled.ipynb		Untitled.ipynb
Untitled1.ipynb		Untitled1.ipynb
Untitled2.ipynb		Untitled2.ipynb
base_doc2vec_svm.py		base_doc2vec_svm.py
base_w2v_bow_svm.py		base_w2v_bow_svm.py
keras_article.py		keras_article.py
keras_lstm.py		keras_lstm.py
keras_title.py		keras_title.py
keras_title_emb.py		keras_title_emb.py
long-lstm-test2.eps		long-lstm-test2.eps
long-lstm-test2test.eps		long-lstm-test2test.eps
lstm-long-test.eps		lstm-long-test.eps
lstm-long^test.eps		lstm-long^test.eps
lstm.py		lstm.py
lstm_proj.py		lstm_proj.py
lstm_trans.py		lstm_trans.py
lstm_unbalance.py		lstm_unbalance.py
main.py		main.py
main_proj.py		main_proj.py
main_regression.py		main_regression.py
main_trans.py		main_trans.py
main_unbalance.py		main_unbalance.py
malstm_title.py		malstm_title.py
model_lstm2_c		model_lstm2_c
sentence_unbalance.py		sentence_unbalance.py
sentences.py		sentences.py
sentences_proj.py		sentences_proj.py
sentences_trans.py		sentences_trans.py
svm_title.py		svm_title.py
top-test.eps		top-test.eps
top-training.eps		top-training.eps
top-training_new.eps		top-training_new.eps
top1-5-10_e40_l0.eps		top1-5-10_e40_l0.eps

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Repository files navigation

Cross-lingual news article comparison system using bi-graph clustering and Siamese LSTM

About

Releases

Packages

Languages

liuenda/LiuSTM

Folders and files

Latest commit

History

Repository files navigation

Cross-lingual news article comparison system using bi-graph clustering and Siamese LSTM

About

Resources

Stars

Watchers

Forks

Releases

Packages 0

Languages

Packages