Issue reproducing results of your experiments

[manuel-dileo]

Hi,
Thank you for your nice work!

I'm trying to reproduce some of the results of your experiments following the commands described in the README but unfortunately, sometimes I obtain very dissimilar results to the one reported in the Appendix of your work.

For instance, I ran the experiments related to the Wikipedia dataset for tgn_attn by following the README. I've included the log related to both training and evaluation steps. If I'm interpreting the results correctly, It seems the AP on transductive task is on average about 0.80, which is 8 points below the result reported in the paper. Reading both the logs, code, and your paper, the hyperparameter configuration seems the same. Is there something I'm missing? Am I making some mistakes? Do you have any idea why it happens?

Thank you in advance for your help and feel free to request me any further additional details.

I obtained the following results on the test set using hist_nre as the negative sampling strategy:

INFO:root:***` Arguments ***
INFO:root:Namespace(use_memory=True, decoder='concat', gpu=0, n_head=2, n_layer=1, lr=0.0001, n_runs=3, drop_out=0.1, node_dim=100, time_dim=100, backprop_every=1, embedding_module='graph_attention', message_function='identity', memory_updater='gru', aggregator='last', memory_update_at_end=False, message_dim=100, memory_dim=172, use_destination_embedding_in_message=False, n_degree=10, use_source_embedding_in_message=False, dyrep=False, data='wikipedia', different_new_nodes=False, randomize_features=False, uniform=False, val_ratio=0.15, test_ratio=0.15, neg_sample='hist_nre', model='tgn')
The dataset has 157474 interactions, involving 8227 different nodes
The training dataset has 77579 interactions, involving 5293 different nodes
The validation dataset has 23621 interactions, involving 2878 different nodes
The test dataset has 23621 interactions, involving 3181 different nodes
The new node validation dataset has 11329 interactions, involving 1987 different nodes
The new node test dataset has 10614 interactions, involving 2292 different nodes
822 nodes were used for the inductive testing, i.e. are never seen during training
INFO:root:Negative Edge Sampling: hist_nre
INFO:root:************************************
INFO:root:*********** Test starts *************
INFO:root:************************************
INFO:root:*********** Run 0 starts *************
INFO:root:Performance of the tgn model for the test set.
INFO:root:Network Name: wikipedia, Model: tgn
INFO:root:Test statistics: *** Old Nodes (Transductive) ***
INFO:root:Test statistics: Old nodes -- auc_inherent: 0.7976978984450922
INFO:root:Test statistics: Old nodes -- ap_inherent: 0.8172550107597417
INFO:root:Test statistics: New nodes -- auc_inherent: 0.7731374622071051
INFO:root:Test statistics: New nodes -- ap_inherent: 0.8209122757636943
INFO:root:Test statistics: **Old nodes -- ap: 0.8172550107597417**
INFO:root:Test statistics: Old nodes -- au_roc_score: 0.7976978984450922
INFO:root:Test statistics: Old nodes -- opt_thr_au_roc: 0.9707613587379456
INFO:root:Test statistics: Old nodes -- acc_auroc_opt_thr: 0.7496818727490999
INFO:root:Test statistics: Old nodes -- prec_auroc_opt_thr: 0.8229704710921953
INFO:root:Test statistics: Old nodes -- rec_auroc_opt_thr: 0.6429791916766706
INFO:root:Test statistics: Old nodes -- f1_auroc_opt_thr: 0.7182537353531723
INFO:root:Test statistics: Old nodes -- au_pr_score: 0.815620639378624
INFO:root:Test statistics: Old nodes -- opt_thr_au_pr: 0.9153319001197815
INFO:root:Test statistics: Old nodes -- acc_aupr_opt_thr: 0.6627731092436976
INFO:root:Test statistics: Old nodes -- prec_aupr_opt_thr: 0.7709254192273766
INFO:root:Test statistics: Old nodes -- rec_aupr_opt_thr: 0.5629411764705882
INFO:root:Test statistics: Old nodes -- f1_aupr_opt_thr: 0.56061605414922
INFO:root:Test statistics: Old nodes -- acc: 0.5977110844337734
INFO:root:Test statistics: Old nodes -- prec: 0.5595330355200728
INFO:root:Test statistics: Old nodes -- rec: 0.9201460584233694
INFO:root:Test statistics: Old nodes -- f1: 0.6957720316818435
INFO:root:Test statistics: New nodes -- ap: 0.8209122757636943
INFO:root:Test statistics: New nodes -- au_roc_score: 0.7731374622071051
INFO:root:Test statistics: New nodes -- opt_thr_au_roc: 0.9758896827697754
INFO:root:Test statistics: New nodes -- acc_auroc_opt_thr: 0.7495701058201057
INFO:root:Test statistics: New nodes -- prec_auroc_opt_thr: 0.8392478032759629
INFO:root:Test statistics: New nodes -- rec_auroc_opt_thr: 0.6241402116402118
INFO:root:Test statistics: New nodes -- f1_auroc_opt_thr: 0.7113762763412118
INFO:root:Test statistics: New nodes -- au_pr_score: 0.8200984568560308
INFO:root:Test statistics: New nodes -- opt_thr_au_pr: 0.9382930397987366
INFO:root:Test statistics: New nodes -- acc_aupr_opt_thr: 0.7085978835978836
INFO:root:Test statistics: New nodes -- prec_aupr_opt_thr: 0.770502035614445
INFO:root:Test statistics: New nodes -- rec_aupr_opt_thr: 0.6584788359788362
INFO:root:Test statistics: New nodes -- f1_aupr_opt_thr: 0.6624386287890781
INFO:root:Test statistics: New nodes -- acc: 0.5308730158730158
INFO:root:Test statistics: New nodes -- prec: 0.5172921650566022
INFO:root:Test statistics: New nodes -- rec: 0.9256481481481481
INFO:root:Test statistics: New nodes -- f1: 0.6635949263928558
INFO:root:Info: Run 0 elapsed time: 15.306545495986938 seconds.
INFO:root:************************************
INFO:root:*********** Run 1 starts *************
INFO:root:Performance of the tgn model for the test set.
INFO:root:Network Name: wikipedia, Model: tgn
INFO:root:Test statistics: *** Old Nodes (Transductive) ***
INFO:root:Test statistics: Old nodes -- auc_inherent: 0.8006257648297415
INFO:root:Test statistics: Old nodes -- ap_inherent: 0.7880019154957626
INFO:root:Test statistics: New nodes -- auc_inherent: 0.7760298658352229
INFO:root:Test statistics: New nodes -- ap_inherent: 0.8240847031941535
INFO:root:Test statistics: **Old nodes -- ap: 0.7880019154957626**
INFO:root:Test statistics: Old nodes -- au_roc_score: 0.8006257648297415
INFO:root:Test statistics: Old nodes -- opt_thr_au_roc: 0.950464129447937
INFO:root:Test statistics: Old nodes -- acc_auroc_opt_thr: 0.7643437374949981
INFO:root:Test statistics: Old nodes -- prec_auroc_opt_thr: 0.825678050739192
INFO:root:Test statistics: Old nodes -- rec_auroc_opt_thr: 0.6737755102040814
INFO:root:Test statistics: Old nodes -- f1_auroc_opt_thr: 0.7393263294649426
INFO:root:Test statistics: Old nodes -- au_pr_score: 0.7849127666858429
INFO:root:Test statistics: Old nodes -- opt_thr_au_pr: 0.9801141023635864
INFO:root:Test statistics: Old nodes -- acc_aupr_opt_thr: 0.5296218487394957
INFO:root:Test statistics: Old nodes -- prec_aupr_opt_thr: 0.34396916890842016
INFO:root:Test statistics: Old nodes -- rec_aupr_opt_thr: 0.10504201680672269
INFO:root:Test statistics: Old nodes -- f1_aupr_opt_thr: 0.7235516263253224
INFO:root:Test statistics: Old nodes -- acc: 0.6061264505802322
INFO:root:Test statistics: Old nodes -- prec: 0.5658443600314311
INFO:root:Test statistics: Old nodes -- rec: 0.913235294117647
INFO:root:Test statistics: Old nodes -- f1: 0.6986153663186325
INFO:root:Test statistics: New nodes -- ap: 0.8240847031941535
INFO:root:Test statistics: New nodes -- au_roc_score: 0.7760298658352229
INFO:root:Test statistics: New nodes -- opt_thr_au_roc: 0.970453679561615
INFO:root:Test statistics: New nodes -- acc_auroc_opt_thr: 0.7601388888888887
INFO:root:Test statistics: New nodes -- prec_auroc_opt_thr: 0.8771052009252576
INFO:root:Test statistics: New nodes -- rec_auroc_opt_thr: 0.6169444444444445
INFO:root:Test statistics: New nodes -- f1_auroc_opt_thr: 0.71860422232696
INFO:root:Test statistics: New nodes -- au_pr_score: 0.8228562648761175
INFO:root:Test statistics: New nodes -- opt_thr_au_pr: 0.9264819622039795
INFO:root:Test statistics: New nodes -- acc_aupr_opt_thr: 0.6732407407407407
INFO:root:Test statistics: New nodes -- prec_aupr_opt_thr: 0.7945601242845284
INFO:root:Test statistics: New nodes -- rec_aupr_opt_thr: 0.5062037037037037
INFO:root:Test statistics: New nodes -- f1_aupr_opt_thr: 0.5834315808799078
INFO:root:Test statistics: New nodes -- acc: 0.5385317460317459
INFO:root:Test statistics: New nodes -- prec: 0.5220552580526202
INFO:root:Test statistics: New nodes -- rec: 0.9143650793650794
INFO:root:Test statistics: New nodes -- f1: 0.6644456454397719
INFO:root:Info: Run 1 elapsed time: 13.319621801376343 seconds.
INFO:root:************************************
INFO:root:*********** Run 2 starts *************
INFO:root:Performance of the tgn model for the test set.
INFO:root:Network Name: wikipedia, Model: tgn
INFO:root:Test statistics: *** Old Nodes (Transductive) ***
INFO:root:Test statistics: Old nodes -- auc_inherent: 0.7880636880942855
INFO:root:Test statistics: Old nodes -- ap_inherent: 0.7586455147685096
INFO:root:Test statistics: New nodes -- auc_inherent: 0.7752561366213152
INFO:root:Test statistics: New nodes -- ap_inherent: 0.8193617652375809
INFO:root:Test statistics: **Old nodes -- ap: 0.7586455147685096**
INFO:root:Test statistics: Old nodes -- au_roc_score: 0.7880636880942855
INFO:root:Test statistics: Old nodes -- opt_thr_au_roc: 0.9707430005073547
INFO:root:Test statistics: Old nodes -- acc_auroc_opt_thr: 0.7483673469387756
INFO:root:Test statistics: Old nodes -- prec_auroc_opt_thr: 0.7816918049254487
INFO:root:Test statistics: Old nodes -- rec_auroc_opt_thr: 0.6947198879551822
INFO:root:Test statistics: Old nodes -- f1_auroc_opt_thr: 0.7329222492843995
INFO:root:Test statistics: Old nodes -- au_pr_score: 0.7554820558951595
INFO:root:Test statistics: Old nodes -- opt_thr_au_pr: 0.9916073083877563
INFO:root:Test statistics: Old nodes -- acc_aupr_opt_thr: 0.5257142857142857
INFO:root:Test statistics: Old nodes -- prec_aupr_opt_thr: 0.29283813081897136
INFO:root:Test statistics: Old nodes -- rec_aupr_opt_thr: 0.10147058823529412
INFO:root:Test statistics: Old nodes -- f1_aupr_opt_thr: 0.7681308461881446
INFO:root:Test statistics: Old nodes -- acc: 0.5846758703481391
INFO:root:Test statistics: Old nodes -- prec: 0.5497639240870785
INFO:root:Test statistics: Old nodes -- rec: 0.9372889155662265
INFO:root:Test statistics: Old nodes -- f1: 0.6929364688234859
INFO:root:Test statistics: New nodes -- ap: 0.8193617652375809
INFO:root:Test statistics: New nodes -- au_roc_score: 0.7752561366213152
INFO:root:Test statistics: New nodes -- opt_thr_au_roc: 0.9804934859275818
INFO:root:Test statistics: New nodes -- acc_auroc_opt_thr: 0.749510582010582
INFO:root:Test statistics: New nodes -- prec_auroc_opt_thr: 0.8521151081683656
INFO:root:Test statistics: New nodes -- rec_auroc_opt_thr: 0.6071031746031748
INFO:root:Test statistics: New nodes -- f1_auroc_opt_thr: 0.7053850552879857
INFO:root:Test statistics: New nodes -- au_pr_score: 0.818206064975554
INFO:root:Test statistics: New nodes -- opt_thr_au_pr: 0.9441109895706177
INFO:root:Test statistics: New nodes -- acc_aupr_opt_thr: 0.6854232804232806
INFO:root:Test statistics: New nodes -- prec_aupr_opt_thr: 0.7826475970528322
INFO:root:Test statistics: New nodes -- rec_aupr_opt_thr: 0.6081613756613756
INFO:root:Test statistics: New nodes -- f1_aupr_opt_thr: 0.6051941071840851
INFO:root:Test statistics: New nodes -- acc: 0.5255687830687832
INFO:root:Test statistics: New nodes -- prec: 0.514144509221895
INFO:root:Test statistics: New nodes -- rec: 0.936362433862434
INFO:root:Test statistics: New nodes -- f1: 0.6637249255428349
INFO:root:Info: Run 2 elapsed time: 13.61383056640625 seconds.
INFO:root:Info: Total elapsed time: 42.24013137817383 seconds.
```

This is the training log related to evaluation above:

```
2023-10-25 02:03:54,255 - root - INFO - Namespace(data='wikipedia', decoder='concat', bs=200, prefix='tgn_attn', n_degree=10, n_head=2, n_epoch=50, n_layer=1, lr=0.0001, patience=5, n_runs=3, drop_out=0.1, gpu=0, node_dim=100, time_dim=100, backprop_every=1, use_memory=True, embedding_module='graph_attention', message_function='identity', memory_updater='gru', aggregator='last', memory_update_at_end=False, message_dim=100, memory_dim=172, different_new_nodes=False, uniform=False, randomize_features=False, use_destination_embedding_in_message=False, use_source_embedding_in_message=False, dyrep=False, val_ratio=0.15, test_ratio=0.15)
2023-10-25 02:03:55,407 - root - INFO - ************************************
2023-10-25 02:03:55,407 - root - INFO - ********** Run 0 starts. **********
2023-10-25 02:03:56,835 - root - INFO - num of training instances: 77579
2023-10-25 02:03:56,835 - root - INFO - num of batches per epoch: 388
2023-10-25 02:03:56,837 - root - INFO - start 0 epoch
2023-10-25 02:04:40,084 - root - INFO - epoch: 0 took 43.25s
2023-10-25 02:04:40,084 - root - INFO - Epoch mean loss: 0.6544743108073461
2023-10-25 02:04:40,084 - root - INFO - val auc: 0.9331831685054974, new node val auc: 0.9372436637685193
2023-10-25 02:04:40,084 - root - INFO - val ap: 0.9369712388836342, new node val ap: 0.9406248053515711
2023-10-25 02:04:40,166 - root - INFO - start 1 epoch
2023-10-25 02:05:23,441 - root - INFO - epoch: 1 took 43.34s
2023-10-25 02:05:23,441 - root - INFO - Epoch mean loss: 0.5117596087535632
2023-10-25 02:05:23,441 - root - INFO - val auc: 0.9381341731930868, new node val auc: 0.9472770082769572
2023-10-25 02:05:23,441 - root - INFO - val ap: 0.9415516471381926, new node val ap: 0.9506987303943254
2023-10-25 02:05:23,500 - root - INFO - start 2 epoch
2023-10-25 02:06:06,774 - root - INFO - epoch: 2 took 43.31s
2023-10-25 02:06:06,774 - root - INFO - Epoch mean loss: 0.4784796604329778
2023-10-25 02:06:06,774 - root - INFO - val auc: 0.9450878818193945, new node val auc: 0.9560554115179482
2023-10-25 02:06:06,774 - root - INFO - val ap: 0.948150704657506, new node val ap: 0.9590644111036896
2023-10-25 02:06:06,852 - root - INFO - start 3 epoch
2023-10-25 02:06:50,793 - root - INFO - epoch: 3 took 44.00s
2023-10-25 02:06:50,793 - root - INFO - Epoch mean loss: 0.44193456452531915
2023-10-25 02:06:50,793 - root - INFO - val auc: 0.9472435571847788, new node val auc: 0.9607007427227403
2023-10-25 02:06:50,793 - root - INFO - val ap: 0.951726385636963, new node val ap: 0.9640512067712843
2023-10-25 02:06:50,880 - root - INFO - start 4 epoch
2023-10-25 02:07:35,619 - root - INFO - epoch: 4 took 44.81s
2023-10-25 02:07:35,620 - root - INFO - Epoch mean loss: 0.4038544486939293
2023-10-25 02:07:35,620 - root - INFO - val auc: 0.9570419191486118, new node val auc: 0.9624153853513358
2023-10-25 02:07:35,620 - root - INFO - val ap: 0.9604094246491608, new node val ap: 0.9656091336553986
2023-10-25 02:07:35,701 - root - INFO - start 5 epoch
2023-10-25 02:08:19,820 - root - INFO - epoch: 5 took 44.18s
2023-10-25 02:08:19,821 - root - INFO - Epoch mean loss: 0.38989225018423856
2023-10-25 02:08:19,821 - root - INFO - val auc: 0.961510514682063, new node val auc: 0.9656054909297159
2023-10-25 02:08:19,821 - root - INFO - val ap: 0.965703519150543, new node val ap: 0.9699055378765158
2023-10-25 02:08:19,876 - root - INFO - start 6 epoch
2023-10-25 02:09:03,821 - root - INFO - epoch: 6 took 43.98s
2023-10-25 02:09:03,821 - root - INFO - Epoch mean loss: 0.3768120009029649
2023-10-25 02:09:03,821 - root - INFO - val auc: 0.9629708316659997, new node val auc: 0.9676784927736081
2023-10-25 02:09:03,821 - root - INFO - val ap: 0.9668167085884218, new node val ap: 0.9713889312226722
2023-10-25 02:09:03,879 - root - INFO - start 7 epoch
2023-10-25 02:09:48,973 - root - INFO - epoch: 7 took 45.13s
2023-10-25 02:09:48,973 - root - INFO - Epoch mean loss: 0.356577481267993
2023-10-25 02:09:48,973 - root - INFO - val auc: 0.9673926513462526, new node val auc: 0.9722944869572826
2023-10-25 02:09:48,973 - root - INFO - val ap: 0.9705689246183211, new node val ap: 0.9749690367571819
2023-10-25 02:09:49,032 - root - INFO - start 8 epoch
2023-10-25 02:10:33,702 - root - INFO - epoch: 8 took 44.71s
2023-10-25 02:10:33,703 - root - INFO - Epoch mean loss: 0.35486559220349667
2023-10-25 02:10:33,703 - root - INFO - val auc: 0.9575684459498085, new node val auc: 0.9658602277243796
2023-10-25 02:10:33,703 - root - INFO - val ap: 0.9626287560176571, new node val ap: 0.9700555521648483
2023-10-25 02:10:33,760 - root - INFO - start 9 epoch
2023-10-25 02:11:18,487 - root - INFO - epoch: 9 took 44.77s
2023-10-25 02:11:18,487 - root - INFO - Epoch mean loss: 0.36354996740203543
2023-10-25 02:11:18,488 - root - INFO - val auc: 0.9692842208311897, new node val auc: 0.9720895241566752
2023-10-25 02:11:18,488 - root - INFO - val ap: 0.9724754457507584, new node val ap: 0.9750473267211728
2023-10-25 02:11:18,557 - root - INFO - start 10 epoch
2023-10-25 02:12:03,538 - root - INFO - epoch: 10 took 45.03s
2023-10-25 02:12:03,538 - root - INFO - Epoch mean loss: 0.33820416544055204
2023-10-25 02:12:03,538 - root - INFO - val auc: 0.9687806265363289, new node val auc: 0.9717812704723169
2023-10-25 02:12:03,538 - root - INFO - val ap: 0.9717646637361462, new node val ap: 0.9753919360163678
2023-10-25 02:12:03,612 - root - INFO - start 11 epoch
2023-10-25 02:12:48,861 - root - INFO - epoch: 11 took 45.31s
2023-10-25 02:12:48,862 - root - INFO - Epoch mean loss: 0.3519376083901248
2023-10-25 02:12:48,862 - root - INFO - val auc: 0.9711291133119915, new node val auc: 0.972577652874901
2023-10-25 02:12:48,862 - root - INFO - val ap: 0.9740046389399605, new node val ap: 0.9767256973206834
2023-10-25 02:12:48,922 - root - INFO - start 12 epoch
2023-10-25 02:13:33,560 - root - INFO - epoch: 12 took 44.68s
2023-10-25 02:13:33,560 - root - INFO - Epoch mean loss: 0.348712871139197
2023-10-25 02:13:33,560 - root - INFO - val auc: 0.9725878942053013, new node val auc: 0.9746857523217333
2023-10-25 02:13:33,560 - root - INFO - val ap: 0.975932449170503, new node val ap: 0.978376767927766
2023-10-25 02:13:33,623 - root - INFO - start 13 epoch
2023-10-25 02:14:17,731 - root - INFO - epoch: 13 took 44.15s
2023-10-25 02:14:17,731 - root - INFO - Epoch mean loss: 0.330714580132482
2023-10-25 02:14:17,731 - root - INFO - val auc: 0.9714708545322893, new node val auc: 0.9747396086288674
2023-10-25 02:14:17,731 - root - INFO - val ap: 0.9750705777348887, new node val ap: 0.978393224442112
2023-10-25 02:14:17,792 - root - INFO - start 14 epoch
2023-10-25 02:15:02,314 - root - INFO - epoch: 14 took 44.57s
2023-10-25 02:15:02,315 - root - INFO - Epoch mean loss: 0.31500264969606373
2023-10-25 02:15:02,315 - root - INFO - val auc: 0.9740591324625089, new node val auc: 0.9758312055302008
2023-10-25 02:15:02,315 - root - INFO - val ap: 0.9768085304230383, new node val ap: 0.9791303519363836
2023-10-25 02:15:02,378 - root - INFO - start 15 epoch
2023-10-25 02:15:47,144 - root - INFO - epoch: 15 took 44.81s
2023-10-25 02:15:47,145 - root - INFO - Epoch mean loss: 0.3123122620367512
2023-10-25 02:15:47,145 - root - INFO - val auc: 0.9741262147716231, new node val auc: 0.9775977128989168
2023-10-25 02:15:47,145 - root - INFO - val ap: 0.9770180006006156, new node val ap: 0.9804193075966515
2023-10-25 02:15:47,215 - root - INFO - start 16 epoch
2023-10-25 02:16:31,513 - root - INFO - epoch: 16 took 44.35s
2023-10-25 02:16:31,514 - root - INFO - Epoch mean loss: 0.3114401551981255
2023-10-25 02:16:31,514 - root - INFO - val auc: 0.9754681691724311, new node val auc: 0.9781997191727891
2023-10-25 02:16:31,514 - root - INFO - val ap: 0.9780162549767836, new node val ap: 0.9806667640554452
2023-10-25 02:16:31,581 - root - INFO - start 17 epoch
2023-10-25 02:17:16,415 - root - INFO - epoch: 17 took 44.88s
2023-10-25 02:17:16,415 - root - INFO - Epoch mean loss: 0.3046062096722962
2023-10-25 02:17:16,415 - root - INFO - val auc: 0.9730418691286037, new node val auc: 0.9768697386079825
2023-10-25 02:17:16,415 - root - INFO - val ap: 0.9761410747278929, new node val ap: 0.9799197406567245
2023-10-25 02:17:16,472 - root - INFO - start 18 epoch
2023-10-25 02:18:01,842 - root - INFO - epoch: 18 took 45.41s
2023-10-25 02:18:01,842 - root - INFO - Epoch mean loss: 0.3093772614278744
2023-10-25 02:18:01,842 - root - INFO - val auc: 0.9736180017244991, new node val auc: 0.9753022127497398
2023-10-25 02:18:01,842 - root - INFO - val ap: 0.9762983170647017, new node val ap: 0.9785149500687266
2023-10-25 02:18:01,930 - root - INFO - start 19 epoch
2023-10-25 02:18:47,138 - root - INFO - epoch: 19 took 45.28s
2023-10-25 02:18:47,138 - root - INFO - Epoch mean loss: 0.3076711413402533
2023-10-25 02:18:47,138 - root - INFO - val auc: 0.9732243864212351, new node val auc: 0.9784260722565383
2023-10-25 02:18:47,138 - root - INFO - val ap: 0.9755864299085354, new node val ap: 0.980568544907364
2023-10-25 02:18:47,203 - root - INFO - start 20 epoch
2023-10-25 02:19:32,179 - root - INFO - epoch: 20 took 45.02s
2023-10-25 02:19:32,179 - root - INFO - Epoch mean loss: 0.29330090994072944
2023-10-25 02:19:32,179 - root - INFO - val auc: 0.9750694920825475, new node val auc: 0.9758891375613181
2023-10-25 02:19:32,179 - root - INFO - val ap: 0.9781389412756277, new node val ap: 0.9787965418296029
2023-10-25 02:19:32,248 - root - INFO - start 21 epoch
2023-10-25 02:20:16,948 - root - INFO - epoch: 21 took 44.75s
2023-10-25 02:20:16,948 - root - INFO - Epoch mean loss: 0.30630029138830517
2023-10-25 02:20:16,948 - root - INFO - val auc: 0.9768620143295412, new node val auc: 0.9793133935471152
2023-10-25 02:20:16,948 - root - INFO - val ap: 0.9797080351660764, new node val ap: 0.982077317572272
2023-10-25 02:20:17,009 - root - INFO - start 22 epoch
2023-10-25 02:21:01,895 - root - INFO - epoch: 22 took 44.93s
2023-10-25 02:21:01,895 - root - INFO - Epoch mean loss: 0.2862508870161993
2023-10-25 02:21:01,895 - root - INFO - val auc: 0.974778128394215, new node val auc: 0.9776281548848383
2023-10-25 02:21:01,895 - root - INFO - val ap: 0.9772892047360646, new node val ap: 0.9802940011384602
2023-10-25 02:21:01,978 - root - INFO - start 23 epoch
2023-10-25 02:21:46,738 - root - INFO - epoch: 23 took 44.82s
2023-10-25 02:21:46,738 - root - INFO - Epoch mean loss: 0.29116634552165405
2023-10-25 02:21:46,738 - root - INFO - val auc: 0.9737933959298005, new node val auc: 0.9782036015990943
2023-10-25 02:21:46,738 - root - INFO - val ap: 0.9762222156286089, new node val ap: 0.980045548799287
2023-10-25 02:21:46,793 - root - INFO - start 24 epoch
2023-10-25 02:22:31,473 - root - INFO - epoch: 24 took 44.72s
2023-10-25 02:22:31,473 - root - INFO - Epoch mean loss: 0.29292712934776066
2023-10-25 02:22:31,473 - root - INFO - val auc: 0.9766972658110863, new node val auc: 0.9776757831929592
2023-10-25 02:22:31,473 - root - INFO - val ap: 0.979007900897018, new node val ap: 0.9805096256762017
2023-10-25 02:22:31,530 - root - INFO - start 25 epoch
2023-10-25 02:23:15,487 - root - INFO - epoch: 25 took 44.00s
2023-10-25 02:23:15,488 - root - INFO - Epoch mean loss: 0.2839326347349231
2023-10-25 02:23:15,488 - root - INFO - val auc: 0.9761068444044285, new node val auc: 0.9771478430203566
2023-10-25 02:23:15,488 - root - INFO - val ap: 0.9788958756763966, new node val ap: 0.9801716689899665
2023-10-25 02:23:15,550 - root - INFO - start 26 epoch
2023-10-25 02:23:59,728 - root - INFO - epoch: 26 took 44.22s
2023-10-25 02:23:59,729 - root - INFO - Epoch mean loss: 0.28708498515956793
2023-10-25 02:23:59,729 - root - INFO - val auc: 0.9732540082699745, new node val auc: 0.9762317367166489
2023-10-25 02:23:59,729 - root - INFO - val ap: 0.976369912799345, new node val ap: 0.9785933935823783
2023-10-25 02:23:59,729 - root - INFO - No improvement over 5 epochs, stop training
2023-10-25 02:23:59,729 - root - INFO - Loading the best model at epoch 21
2023-10-25 02:23:59,736 - root - INFO - Loaded the best model at epoch 21 for inference
2023-10-25 02:24:14,975 - root - INFO - Test statistics: Old nodes -- auc_inherent: 0.9667954998666135
2023-10-25 02:24:14,976 - root - INFO - Test statistics: Old nodes -- ap_inherent: 0.9707907344702982
2023-10-25 02:24:14,976 - root - INFO - Test statistics: New nodes -- auc_inherent: 0.9658363851095992
2023-10-25 02:24:14,976 - root - INFO - Test statistics: New nodes -- ap_inherent: 0.9692621269807276
2023-10-25 02:24:14,976 - root - INFO - Test statistics: Old nodes -- ap: 0.9707907344702982
2023-10-25 02:24:14,976 - root - INFO - Test statistics: Old nodes -- au_roc_score: 0.9667954998666135
2023-10-25 02:24:14,976 - root - INFO - Test statistics: Old nodes -- opt_thr_au_roc: 0.8109609484672546
2023-10-25 02:24:14,976 - root - INFO - Test statistics: Old nodes -- acc_auroc_opt_thr: 0.9135594237695077
2023-10-25 02:24:14,976 - root - INFO - Test statistics: Old nodes -- prec_auroc_opt_thr: 0.9395032277113026
2023-10-25 02:24:14,976 - root - INFO - Test statistics: Old nodes -- rec_auroc_opt_thr: 0.8851020408163267
2023-10-25 02:24:14,976 - root - INFO - Test statistics: Old nodes -- f1_auroc_opt_thr: 0.9107289927511439
2023-10-25 02:24:14,976 - root - INFO - Test statistics: Old nodes -- au_pr_score: 0.9707086833936728
2023-10-25 02:24:14,976 - root - INFO - Test statistics: Old nodes -- opt_thr_au_pr: 0.776494562625885
2023-10-25 02:24:14,976 - root - INFO - Test statistics: Old nodes -- acc_aupr_opt_thr: 0.9131812725090035
2023-10-25 02:24:14,976 - root - INFO - Test statistics: Old nodes -- prec_aupr_opt_thr: 0.9274386192443718
2023-10-25 02:24:14,976 - root - INFO - Test statistics: Old nodes -- rec_aupr_opt_thr: 0.8980432172869148
2023-10-25 02:24:14,976 - root - INFO - Test statistics: Old nodes -- f1_aupr_opt_thr: 0.9117769969829053
2023-10-25 02:24:14,976 - root - INFO - Test statistics: Old nodes -- acc: 0.8766886754701879
2023-10-25 02:24:14,976 - root - INFO - Test statistics: Old nodes -- prec: 0.8318351212638334
2023-10-25 02:24:14,976 - root - INFO - Test statistics: Old nodes -- rec: 0.9465326130452181
2023-10-25 02:24:14,976 - root - INFO - Test statistics: Old nodes -- f1: 0.8849144679671297
2023-10-25 02:24:14,976 - root - INFO - Test statistics: New nodes -- ap: 0.9692621269807276
2023-10-25 02:24:14,976 - root - INFO - Test statistics: New nodes -- au_roc_score: 0.9658363851095992
2023-10-25 02:24:14,976 - root - INFO - Test statistics: New nodes -- opt_thr_au_roc: 0.8327393531799316
2023-10-25 02:24:14,976 - root - INFO - Test statistics: New nodes -- acc_auroc_opt_thr: 0.9090343915343916
2023-10-25 02:24:14,977 - root - INFO - Test statistics: New nodes -- prec_auroc_opt_thr: 0.9309248269410569
2023-10-25 02:24:14,977 - root - INFO - Test statistics: New nodes -- rec_auroc_opt_thr: 0.8845767195767195
2023-10-25 02:24:14,977 - root - INFO - Test statistics: New nodes -- f1_auroc_opt_thr: 0.9062950058572489
2023-10-25 02:24:14,977 - root - INFO - Test statistics: New nodes -- au_pr_score: 0.9691552966213025
2023-10-25 02:24:14,977 - root - INFO - Test statistics: New nodes -- opt_thr_au_pr: 0.7936143279075623
2023-10-25 02:24:14,977 - root - INFO - Test statistics: New nodes -- acc_aupr_opt_thr: 0.9084325396825398
2023-10-25 02:24:14,977 - root - INFO - Test statistics: New nodes -- prec_aupr_opt_thr: 0.9150108844011897
2023-10-25 02:24:14,977 - root - INFO - Test statistics: New nodes -- rec_aupr_opt_thr: 0.9019841269841271
2023-10-25 02:24:14,977 - root - INFO - Test statistics: New nodes -- f1_aupr_opt_thr: 0.9076714670846444
2023-10-25 02:24:14,977 - root - INFO - Test statistics: New nodes -- acc: 0.8646230158730158
2023-10-25 02:24:14,977 - root - INFO - Test statistics: New nodes -- prec: 0.8096289449759637
2023-10-25 02:24:14,977 - root - INFO - Test statistics: New nodes -- rec: 0.9547883597883597
2023-10-25 02:24:14,977 - root - INFO - Test statistics: New nodes -- f1: 0.875921300387986
2023-10-25 02:24:14,977 - root - INFO - Saving the model
2023-10-25 02:24:15,367 - root - INFO - TGN model saved
2023-10-25 02:24:15,368 - root - INFO - Run 0, elapsed time: 1219.9606909751892 seconds.
2023-10-25 02:24:15,368 - root - INFO - ************************************
2023-10-25 02:24:15,368 - root - INFO - ********** Run 1 starts. **********
2023-10-25 02:24:15,476 - root - INFO - num of training instances: 77579
2023-10-25 02:24:15,476 - root - INFO - num of batches per epoch: 388
2023-10-25 02:24:15,478 - root - INFO - start 0 epoch
2023-10-25 02:25:00,276 - root - INFO - epoch: 0 took 44.80s
2023-10-25 02:25:00,277 - root - INFO - Epoch mean loss: 0.6292816959705549
2023-10-25 02:25:00,277 - root - INFO - val auc: 0.9425674350692658, new node val auc: 0.9519797697401368
2023-10-25 02:25:00,277 - root - INFO - val ap: 0.9440838069568095, new node val ap: 0.9536835286169085
2023-10-25 02:25:00,342 - root - INFO - start 1 epoch
2023-10-25 02:25:45,200 - root - INFO - epoch: 1 took 44.90s
2023-10-25 02:25:45,200 - root - INFO - Epoch mean loss: 0.4966244323121518
2023-10-25 02:25:45,200 - root - INFO - val auc: 0.963282409154138, new node val auc: 0.9635682296394341
2023-10-25 02:25:45,200 - root - INFO - val ap: 0.9664021896374815, new node val ap: 0.9663069442595131
2023-10-25 02:25:45,254 - root - INFO - start 2 epoch
2023-10-25 02:26:30,646 - root - INFO - epoch: 2 took 45.43s
2023-10-25 02:26:30,646 - root - INFO - Epoch mean loss: 0.4362970183497852
2023-10-25 02:26:30,646 - root - INFO - val auc: 0.9646992263572096, new node val auc: 0.9659305104597923
2023-10-25 02:26:30,646 - root - INFO - val ap: 0.9677343431893068, new node val ap: 0.9693126765241891
2023-10-25 02:26:30,720 - root - INFO - start 3 epoch
2023-10-25 02:27:15,780 - root - INFO - epoch: 3 took 45.11s
2023-10-25 02:27:15,780 - root - INFO - Epoch mean loss: 0.38676398589285377
2023-10-25 02:27:15,780 - root - INFO - val auc: 0.9607929633758264, new node val auc: 0.9663168553783352
2023-10-25 02:27:15,780 - root - INFO - val ap: 0.9649521860563575, new node val ap: 0.970158214955084
2023-10-25 02:27:15,842 - root - INFO - start 4 epoch
2023-10-25 02:28:00,907 - root - INFO - epoch: 4 took 45.11s
2023-10-25 02:28:00,907 - root - INFO - Epoch mean loss: 0.3885010524278449
2023-10-25 02:28:00,907 - root - INFO - val auc: 0.9637238519217212, new node val auc: 0.967991475372073
2023-10-25 02:28:00,907 - root - INFO - val ap: 0.9676437340074051, new node val ap: 0.9711764028098365
2023-10-25 02:28:00,969 - root - INFO - start 5 epoch
2023-10-25 02:28:45,121 - root - INFO - epoch: 5 took 44.19s
2023-10-25 02:28:45,121 - root - INFO - Epoch mean loss: 0.3815943038801557
2023-10-25 02:28:45,121 - root - INFO - val auc: 0.9646275543550752, new node val auc: 0.9690696363715912
2023-10-25 02:28:45,121 - root - INFO - val ap: 0.968847427102364, new node val ap: 0.9731615508890684
2023-10-25 02:28:45,177 - root - INFO - start 6 epoch
2023-10-25 02:29:30,187 - root - INFO - epoch: 6 took 45.05s
2023-10-25 02:29:30,187 - root - INFO - Epoch mean loss: 0.36954496485978056
2023-10-25 02:29:30,187 - root - INFO - val auc: 0.9635408749213971, new node val auc: 0.9691214704065313
2023-10-25 02:29:30,187 - root - INFO - val ap: 0.9670815809169221, new node val ap: 0.9703944138774232
2023-10-25 02:29:30,239 - root - INFO - start 7 epoch
2023-10-25 02:30:14,518 - root - INFO - epoch: 7 took 44.32s
2023-10-25 02:30:14,519 - root - INFO - Epoch mean loss: 0.35359133499645695
2023-10-25 02:30:14,519 - root - INFO - val auc: 0.9657605208750167, new node val auc: 0.9718511633178253
2023-10-25 02:30:14,519 - root - INFO - val ap: 0.9694420449274964, new node val ap: 0.9754186654544404
2023-10-25 02:30:14,570 - root - INFO - start 8 epoch
2023-10-25 02:30:59,015 - root - INFO - epoch: 8 took 44.48s
2023-10-25 02:30:59,015 - root - INFO - Epoch mean loss: 0.33817019375021923
2023-10-25 02:30:59,015 - root - INFO - val auc: 0.9633181691724308, new node val auc: 0.9698746950567031
2023-10-25 02:30:59,015 - root - INFO - val ap: 0.9676615074591463, new node val ap: 0.9736624683758547
2023-10-25 02:30:59,100 - root - INFO - start 9 epoch
2023-10-25 02:31:43,950 - root - INFO - epoch: 9 took 44.92s
2023-10-25 02:31:43,950 - root - INFO - Epoch mean loss: 0.35249757682232513
2023-10-25 02:31:43,950 - root - INFO - val auc: 0.96760359858229, new node val auc: 0.9704119836390145
2023-10-25 02:31:43,950 - root - INFO - val ap: 0.9711232055322732, new node val ap: 0.9743897226347128
2023-10-25 02:31:44,009 - root - INFO - start 10 epoch
2023-10-25 02:32:28,808 - root - INFO - epoch: 10 took 44.84s
2023-10-25 02:32:28,809 - root - INFO - Epoch mean loss: 0.34701262979009717
2023-10-25 02:32:28,809 - root - INFO - val auc: 0.969990920653976, new node val auc: 0.9714559992915405
2023-10-25 02:32:28,809 - root - INFO - val ap: 0.9735579739050643, new node val ap: 0.9750667001776755
2023-10-25 02:32:28,873 - root - INFO - start 11 epoch
2023-10-25 02:33:14,085 - root - INFO - epoch: 11 took 45.26s
2023-10-25 02:33:14,085 - root - INFO - Epoch mean loss: 0.3456740344768947
2023-10-25 02:33:14,085 - root - INFO - val auc: 0.9711260609005504, new node val auc: 0.9733627032999239
2023-10-25 02:33:14,085 - root - INFO - val ap: 0.9745288122358675, new node val ap: 0.9763044786708059
2023-10-25 02:33:14,147 - root - INFO - start 12 epoch
2023-10-25 02:33:58,506 - root - INFO - epoch: 12 took 44.40s
2023-10-25 02:33:58,506 - root - INFO - Epoch mean loss: 0.34534222248596014
2023-10-25 02:33:58,506 - root - INFO - val auc: 0.9670528892509385, new node val auc: 0.9731457994522089
2023-10-25 02:33:58,506 - root - INFO - val ap: 0.9698179409795671, new node val ap: 0.9751762714698209
2023-10-25 02:33:58,581 - root - INFO - start 13 epoch
2023-10-25 02:34:43,698 - root - INFO - epoch: 13 took 45.17s
2023-10-25 02:34:43,698 - root - INFO - Epoch mean loss: 0.32268958216168214
2023-10-25 02:34:43,698 - root - INFO - val auc: 0.9731947417062062, new node val auc: 0.9756004241268396
2023-10-25 02:34:43,698 - root - INFO - val ap: 0.9763542210835434, new node val ap: 0.9785284855969628
2023-10-25 02:34:43,772 - root - INFO - start 14 epoch
2023-10-25 02:35:29,317 - root - INFO - epoch: 14 took 45.60s
2023-10-25 02:35:29,318 - root - INFO - Epoch mean loss: 0.32996760262656455
2023-10-25 02:35:29,318 - root - INFO - val auc: 0.9702821533375255, new node val auc: 0.9748955736834726
2023-10-25 02:35:29,318 - root - INFO - val ap: 0.9736249776688787, new node val ap: 0.9772405573522723
2023-10-25 02:35:29,382 - root - INFO - start 15 epoch
2023-10-25 02:36:14,972 - root - INFO - epoch: 15 took 45.63s
2023-10-25 02:36:14,972 - root - INFO - Epoch mean loss: 0.30596750049093335
2023-10-25 02:36:14,972 - root - INFO - val auc: 0.9721225185312221, new node val auc: 0.9755747582329419
2023-10-25 02:36:14,972 - root - INFO - val ap: 0.9757803881285207, new node val ap: 0.9791254272561103
2023-10-25 02:36:15,032 - root - INFO - start 16 epoch
2023-10-25 02:36:59,858 - root - INFO - epoch: 16 took 44.87s
2023-10-25 02:36:59,858 - root - INFO - Epoch mean loss: 0.31284615689331724
2023-10-25 02:36:59,858 - root - INFO - val auc: 0.9727249542674212, new node val auc: 0.9760811421167545
2023-10-25 02:36:59,858 - root - INFO - val ap: 0.976116542716109, new node val ap: 0.9792609113419443
2023-10-25 02:36:59,917 - root - INFO - start 17 epoch
2023-10-25 02:37:45,317 - root - INFO - epoch: 17 took 45.44s
2023-10-25 02:37:45,318 - root - INFO - Epoch mean loss: 0.3112061475739651
2023-10-25 02:37:45,318 - root - INFO - val auc: 0.9749811091103107, new node val auc: 0.9770931612841671
2023-10-25 02:37:45,318 - root - INFO - val ap: 0.9777796504391842, new node val ap: 0.9796392830447811
2023-10-25 02:37:45,393 - root - INFO - start 18 epoch
2023-10-25 02:38:29,601 - root - INFO - epoch: 18 took 44.26s
2023-10-25 02:38:29,601 - root - INFO - Epoch mean loss: 0.31151469531901105
2023-10-25 02:38:29,602 - root - INFO - val auc: 0.9732584700546885, new node val auc: 0.9772534813612963
2023-10-25 02:38:29,602 - root - INFO - val ap: 0.9767828157449676, new node val ap: 0.9802881283367896
2023-10-25 02:38:29,658 - root - INFO - start 19 epoch
2023-10-25 02:39:13,764 - root - INFO - epoch: 19 took 44.15s
2023-10-25 02:39:13,765 - root - INFO - Epoch mean loss: 0.30840339634529096
2023-10-25 02:39:13,765 - root - INFO - val auc: 0.975473250728863, new node val auc: 0.9772334748670847
2023-10-25 02:39:13,765 - root - INFO - val ap: 0.9775532966965449, new node val ap: 0.9797899566911549
2023-10-25 02:39:13,824 - root - INFO - start 20 epoch
2023-10-25 02:39:57,939 - root - INFO - epoch: 20 took 44.16s
2023-10-25 02:39:57,939 - root - INFO - Epoch mean loss: 0.30129331039245594
2023-10-25 02:39:57,939 - root - INFO - val auc: 0.9740554164522952, new node val auc: 0.9791588175790716
2023-10-25 02:39:57,939 - root - INFO - val ap: 0.9772426920073104, new node val ap: 0.9813754482013379
2023-10-25 02:39:58,001 - root - INFO - start 21 epoch
2023-10-25 02:40:42,278 - root - INFO - epoch: 21 took 44.32s
2023-10-25 02:40:42,278 - root - INFO - Epoch mean loss: 0.2985713924713356
2023-10-25 02:40:42,278 - root - INFO - val auc: 0.9751040706758892, new node val auc: 0.9796790173709617
2023-10-25 02:40:42,278 - root - INFO - val ap: 0.9774237507470489, new node val ap: 0.9819631323053312
2023-10-25 02:40:42,349 - root - INFO - start 22 epoch
2023-10-25 02:41:27,386 - root - INFO - epoch: 22 took 45.09s
2023-10-25 02:41:27,386 - root - INFO - Epoch mean loss: 0.2870856932143575
2023-10-25 02:41:27,386 - root - INFO - val auc: 0.9758945940280876, new node val auc: 0.9785193730713719
2023-10-25 02:41:27,386 - root - INFO - val ap: 0.9787964403489134, new node val ap: 0.9818636076389861
2023-10-25 02:41:27,451 - root - INFO - start 23 epoch
2023-10-25 02:42:12,790 - root - INFO - epoch: 23 took 45.39s
2023-10-25 02:42:12,791 - root - INFO - Epoch mean loss: 0.2944211192889926
2023-10-25 02:42:12,791 - root - INFO - val auc: 0.9753065854913393, new node val auc: 0.9779005004812675
2023-10-25 02:42:12,791 - root - INFO - val ap: 0.9777305488872171, new node val ap: 0.9805578446426673
2023-10-25 02:42:12,865 - root - INFO - start 24 epoch
2023-10-25 02:42:57,967 - root - INFO - epoch: 24 took 45.16s
2023-10-25 02:42:57,967 - root - INFO - Epoch mean loss: 0.2865448307975666
2023-10-25 02:42:57,967 - root - INFO - val auc: 0.9752627860668077, new node val auc: 0.9755761810820242
2023-10-25 02:42:57,967 - root - INFO - val ap: 0.9785407666987845, new node val ap: 0.9794116109766129
2023-10-25 02:42:58,030 - root - INFO - start 25 epoch
2023-10-25 02:43:43,168 - root - INFO - epoch: 25 took 45.18s
2023-10-25 02:43:43,168 - root - INFO - Epoch mean loss: 0.28283515651278274
2023-10-25 02:43:43,168 - root - INFO - val auc: 0.9744381895615388, new node val auc: 0.9764004144804052
2023-10-25 02:43:43,169 - root - INFO - val ap: 0.9768844486836025, new node val ap: 0.9788494701117919
2023-10-25 02:43:43,222 - root - INFO - start 26 epoch
2023-10-25 02:44:28,589 - root - INFO - epoch: 26 took 45.40s
2023-10-25 02:44:28,589 - root - INFO - Epoch mean loss: 0.2845369040313148
2023-10-25 02:44:28,589 - root - INFO - val auc: 0.9749823300748871, new node val auc: 0.9793261973175533
2023-10-25 02:44:28,589 - root - INFO - val ap: 0.9772501244031683, new node val ap: 0.9805447846319363
2023-10-25 02:44:28,642 - root - INFO - start 27 epoch
2023-10-25 02:45:13,298 - root - INFO - epoch: 27 took 44.70s
2023-10-25 02:45:13,298 - root - INFO - Epoch mean loss: 0.29135604344846044
2023-10-25 02:45:13,299 - root - INFO - val auc: 0.9729865255626059, new node val auc: 0.9765915352537645
2023-10-25 02:45:13,299 - root - INFO - val ap: 0.9758610720695485, new node val ap: 0.97921584473132
2023-10-25 02:45:13,299 - root - INFO - No improvement over 5 epochs, stop training
2023-10-25 02:45:13,299 - root - INFO - Loading the best model at epoch 22
2023-10-25 02:45:13,306 - root - INFO - Loaded the best model at epoch 22 for inference
2023-10-25 02:45:28,767 - root - INFO - Test statistics: Old nodes -- auc_inherent: 0.9735355666075953
2023-10-25 02:45:28,768 - root - INFO - Test statistics: Old nodes -- ap_inherent: 0.9765324504810987
2023-10-25 02:45:28,768 - root - INFO - Test statistics: New nodes -- auc_inherent: 0.9727211451247166
2023-10-25 02:45:28,768 - root - INFO - Test statistics: New nodes -- ap_inherent: 0.9744819197011522
2023-10-25 02:45:28,768 - root - INFO - Test statistics: Old nodes -- ap: 0.9765324504810987
2023-10-25 02:45:28,768 - root - INFO - Test statistics: Old nodes -- au_roc_score: 0.9735355666075953
2023-10-25 02:45:28,768 - root - INFO - Test statistics: Old nodes -- opt_thr_au_roc: 0.6841602921485901
2023-10-25 02:45:28,768 - root - INFO - Test statistics: Old nodes -- acc_auroc_opt_thr: 0.9240116046418567
2023-10-25 02:45:28,768 - root - INFO - Test statistics: Old nodes -- prec_auroc_opt_thr: 0.9496004158920482
2023-10-25 02:45:28,768 - root - INFO - Test statistics: Old nodes -- rec_auroc_opt_thr: 0.8962164865946377
2023-10-25 02:45:28,768 - root - INFO - Test statistics: Old nodes -- f1_auroc_opt_thr: 0.9216186638794472
2023-10-25 02:45:28,768 - root - INFO - Test statistics: Old nodes -- au_pr_score: 0.9764641815867432
2023-10-25 02:45:28,768 - root - INFO - Test statistics: Old nodes -- opt_thr_au_pr: 0.625267505645752
2023-10-25 02:45:28,768 - root - INFO - Test statistics: Old nodes -- acc_aupr_opt_thr: 0.9236124449779911
2023-10-25 02:45:28,768 - root - INFO - Test statistics: Old nodes -- prec_aupr_opt_thr: 0.9373468905606334
2023-10-25 02:45:28,768 - root - INFO - Test statistics: Old nodes -- rec_aupr_opt_thr: 0.9087595038015206
2023-10-25 02:45:28,768 - root - INFO - Test statistics: Old nodes -- f1_aupr_opt_thr: 0.9223337977221638
2023-10-25 02:45:28,768 - root - INFO - Test statistics: Old nodes -- acc: 0.9135914365746297
2023-10-25 02:45:28,768 - root - INFO - Test statistics: Old nodes -- prec: 0.9041644236788484
2023-10-25 02:45:28,768 - root - INFO - Test statistics: Old nodes -- rec: 0.9264285714285716
2023-10-25 02:45:28,768 - root - INFO - Test statistics: Old nodes -- f1: 0.9146945143773952
2023-10-25 02:45:28,768 - root - INFO - Test statistics: New nodes -- ap: 0.9744819197011522
2023-10-25 02:45:28,769 - root - INFO - Test statistics: New nodes -- au_roc_score: 0.9727211451247166
2023-10-25 02:45:28,769 - root - INFO - Test statistics: New nodes -- opt_thr_au_roc: 0.678218424320221
2023-10-25 02:45:28,769 - root - INFO - Test statistics: New nodes -- acc_auroc_opt_thr: 0.9193121693121694
2023-10-25 02:45:28,769 - root - INFO - Test statistics: New nodes -- prec_auroc_opt_thr: 0.9316238022121269
2023-10-25 02:45:28,769 - root - INFO - Test statistics: New nodes -- rec_auroc_opt_thr: 0.9057804232804231
2023-10-25 02:45:28,769 - root - INFO - Test statistics: New nodes -- f1_auroc_opt_thr: 0.9180238681486246
2023-10-25 02:45:28,769 - root - INFO - Test statistics: New nodes -- au_pr_score: 0.9743790494159348
2023-10-25 02:45:28,769 - root - INFO - Test statistics: New nodes -- opt_thr_au_pr: 0.6128215789794922
2023-10-25 02:45:28,769 - root - INFO - Test statistics: New nodes -- acc_aupr_opt_thr: 0.9188492063492064
2023-10-25 02:45:28,769 - root - INFO - Test statistics: New nodes -- prec_aupr_opt_thr: 0.9166739315100995
2023-10-25 02:45:28,769 - root - INFO - Test statistics: New nodes -- rec_aupr_opt_thr: 0.9233994708994709
2023-10-25 02:45:28,769 - root - INFO - Test statistics: New nodes -- f1_aupr_opt_thr: 0.9193780926743038
2023-10-25 02:45:28,769 - root - INFO - Test statistics: New nodes -- acc: 0.9077843915343915
2023-10-25 02:45:28,769 - root - INFO - Test statistics: New nodes -- prec: 0.8863629714265175
2023-10-25 02:45:28,769 - root - INFO - Test statistics: New nodes -- rec: 0.9369841269841268
2023-10-25 02:45:28,769 - root - INFO - Test statistics: New nodes -- f1: 0.9104549306299442
2023-10-25 02:45:28,769 - root - INFO - Saving the model
2023-10-25 02:45:29,162 - root - INFO - TGN model saved
2023-10-25 02:45:29,162 - root - INFO - Run 1, elapsed time: 1273.7943892478943 seconds.
2023-10-25 02:45:29,162 - root - INFO - ************************************
2023-10-25 02:45:29,162 - root - INFO - ********** Run 2 starts. **********
2023-10-25 02:45:29,264 - root - INFO - num of training instances: 77579
2023-10-25 02:45:29,264 - root - INFO - num of batches per epoch: 388
2023-10-25 02:45:29,266 - root - INFO - start 0 epoch
2023-10-25 02:46:13,460 - root - INFO - epoch: 0 took 44.20s
2023-10-25 02:46:13,460 - root - INFO - Epoch mean loss: 0.6409682380323557
2023-10-25 02:46:13,460 - root - INFO - val auc: 0.9306526605880446, new node val auc: 0.9386101307856203
2023-10-25 02:46:13,460 - root - INFO - val ap: 0.9343497569280168, new node val ap: 0.9420159692071832
2023-10-25 02:46:13,530 - root - INFO - start 1 epoch
2023-10-25 02:46:58,773 - root - INFO - epoch: 1 took 45.29s
2023-10-25 02:46:58,773 - root - INFO - Epoch mean loss: 0.5057077399771852
2023-10-25 02:46:58,773 - root - INFO - val auc: 0.9437301877894015, new node val auc: 0.954360772089017
2023-10-25 02:46:58,773 - root - INFO - val ap: 0.9457704712409667, new node val ap: 0.9575065087318037
2023-10-25 02:46:58,838 - root - INFO - start 2 epoch
2023-10-25 02:47:42,933 - root - INFO - epoch: 2 took 44.14s
2023-10-25 02:47:42,933 - root - INFO - Epoch mean loss: 0.44565945702422527
2023-10-25 02:47:42,933 - root - INFO - val auc: 0.9599154776196192, new node val auc: 0.9618876287113735
2023-10-25 02:47:42,933 - root - INFO - val ap: 0.963059203288508, new node val ap: 0.9653978833134207
2023-10-25 02:47:42,992 - root - INFO - start 3 epoch
2023-10-25 02:48:27,296 - root - INFO - epoch: 3 took 44.34s
2023-10-25 02:48:27,296 - root - INFO - Epoch mean loss: 0.42268336164889875
2023-10-25 02:48:27,296 - root - INFO - val auc: 0.9645204219783149, new node val auc: 0.9711399612244963
2023-10-25 02:48:27,296 - root - INFO - val ap: 0.9680393598665945, new node val ap: 0.974495803726751
2023-10-25 02:48:27,358 - root - INFO - start 4 epoch
2023-10-25 02:49:11,607 - root - INFO - epoch: 4 took 44.29s
2023-10-25 02:49:11,608 - root - INFO - Epoch mean loss: 0.4086272277936493
2023-10-25 02:49:11,608 - root - INFO - val auc: 0.9593486418376873, new node val auc: 0.9667297517387302
2023-10-25 02:49:11,608 - root - INFO - val ap: 0.9631302913063472, new node val ap: 0.9706123669366353
2023-10-25 02:49:11,683 - root - INFO - start 5 epoch
2023-10-25 02:49:56,894 - root - INFO - epoch: 5 took 45.27s
2023-10-25 02:49:56,894 - root - INFO - Epoch mean loss: 0.3733020487827124
2023-10-25 02:49:56,894 - root - INFO - val auc: 0.9679124147277959, new node val auc: 0.9691943309538795
2023-10-25 02:49:56,894 - root - INFO - val ap: 0.9706974407897105, new node val ap: 0.9725862150816428
2023-10-25 02:49:56,971 - root - INFO - start 6 epoch
2023-10-25 02:50:41,933 - root - INFO - epoch: 6 took 45.02s
2023-10-25 02:50:41,933 - root - INFO - Epoch mean loss: 0.36313611775943916
2023-10-25 02:50:41,933 - root - INFO - val auc: 0.9657095038015204, new node val auc: 0.9666148719290865
2023-10-25 02:50:41,933 - root - INFO - val ap: 0.9689098214144334, new node val ap: 0.9706011897929471
2023-10-25 02:50:42,010 - root - INFO - start 7 epoch
2023-10-25 02:51:27,227 - root - INFO - epoch: 7 took 45.27s
2023-10-25 02:51:27,228 - root - INFO - Epoch mean loss: 0.35977408743089007
2023-10-25 02:51:27,228 - root - INFO - val auc: 0.9705977286152557, new node val auc: 0.9710125229432273
2023-10-25 02:51:27,228 - root - INFO - val ap: 0.9738659370420584, new node val ap: 0.974532132334208
2023-10-25 02:51:27,299 - root - INFO - start 8 epoch
2023-10-25 02:52:12,469 - root - INFO - epoch: 8 took 45.22s
2023-10-25 02:52:12,470 - root - INFO - Epoch mean loss: 0.35394358200971615
2023-10-25 02:52:12,470 - root - INFO - val auc: 0.966604530383582, new node val auc: 0.9695881906030024
2023-10-25 02:52:12,470 - root - INFO - val ap: 0.9692223678869957, new node val ap: 0.9722397424930633
2023-10-25 02:52:12,552 - root - INFO - start 9 epoch
2023-10-25 02:52:57,591 - root - INFO - epoch: 9 took 45.10s
2023-10-25 02:52:57,592 - root - INFO - Epoch mean loss: 0.35059861316508856
2023-10-25 02:52:57,592 - root - INFO - val auc: 0.9667160059261798, new node val auc: 0.9690749368237612
2023-10-25 02:52:57,592 - root - INFO - val ap: 0.9705440500086135, new node val ap: 0.9726860843298846
2023-10-25 02:52:57,652 - root - INFO - start 10 epoch
2023-10-25 02:53:43,196 - root - INFO - epoch: 10 took 45.58s
2023-10-25 02:53:43,196 - root - INFO - Epoch mean loss: 0.32447411427178335
2023-10-25 02:53:43,196 - root - INFO - val auc: 0.9647541726214295, new node val auc: 0.9718404208797337
2023-10-25 02:53:43,196 - root - INFO - val ap: 0.9684473357551386, new node val ap: 0.9749245834442843
2023-10-25 02:53:43,272 - root - INFO - start 11 epoch
2023-10-25 02:54:28,654 - root - INFO - epoch: 11 took 45.44s
2023-10-25 02:54:28,654 - root - INFO - Epoch mean loss: 0.32617636789212523
2023-10-25 02:54:28,654 - root - INFO - val auc: 0.9710504597076927, new node val auc: 0.9737543032585971
2023-10-25 02:54:28,654 - root - INFO - val ap: 0.974516749782103, new node val ap: 0.976706807422538
2023-10-25 02:54:28,715 - root - INFO - start 12 epoch
2023-10-25 02:55:14,142 - root - INFO - epoch: 12 took 45.47s
2023-10-25 02:55:14,142 - root - INFO - Epoch mean loss: 0.3366305930635978
2023-10-25 02:55:14,142 - root - INFO - val auc: 0.9724774752758246, new node val auc: 0.9747177308054403
2023-10-25 02:55:14,142 - root - INFO - val ap: 0.9753756566176394, new node val ap: 0.9778361164381631
2023-10-25 02:55:14,203 - root - INFO - start 13 epoch
2023-10-25 02:55:59,269 - root - INFO - epoch: 13 took 45.11s
2023-10-25 02:55:59,270 - root - INFO - Epoch mean loss: 0.3297029490553841
2023-10-25 02:55:59,270 - root - INFO - val auc: 0.9671162326835496, new node val auc: 0.9715637332017623
2023-10-25 02:55:59,270 - root - INFO - val ap: 0.9700785459287284, new node val ap: 0.9744455891888278
2023-10-25 02:55:59,330 - root - INFO - start 14 epoch
2023-10-25 02:56:43,974 - root - INFO - epoch: 14 took 44.68s
2023-10-25 02:56:43,974 - root - INFO - Epoch mean loss: 0.32676374232492494
2023-10-25 02:56:43,974 - root - INFO - val auc: 0.9750781512605041, new node val auc: 0.9758128103595326
2023-10-25 02:56:43,974 - root - INFO - val ap: 0.9775296912756609, new node val ap: 0.978999945352562
2023-10-25 02:56:44,039 - root - INFO - start 15 epoch
2023-10-25 02:57:29,328 - root - INFO - epoch: 15 took 45.34s
2023-10-25 02:57:29,328 - root - INFO - Epoch mean loss: 0.3047039597151206
2023-10-25 02:57:29,328 - root - INFO - val auc: 0.9713880783265688, new node val auc: 0.9760516797974144
2023-10-25 02:57:29,328 - root - INFO - val ap: 0.9745573263237365, new node val ap: 0.9792037972825437
2023-10-25 02:57:29,400 - root - INFO - start 16 epoch
2023-10-25 02:58:14,033 - root - INFO - epoch: 16 took 44.69s
2023-10-25 02:58:14,033 - root - INFO - Epoch mean loss: 0.30857773394007043
2023-10-25 02:58:14,033 - root - INFO - val auc: 0.9719275574515521, new node val auc: 0.9746010836688501
2023-10-25 02:58:14,034 - root - INFO - val ap: 0.975452915930981, new node val ap: 0.978251506619975
2023-10-25 02:58:14,128 - root - INFO - start 17 epoch
2023-10-25 02:58:59,100 - root - INFO - epoch: 17 took 45.04s
2023-10-25 02:58:59,101 - root - INFO - Epoch mean loss: 0.3014318261333962
2023-10-25 02:58:59,101 - root - INFO - val auc: 0.9725620310028773, new node val auc: 0.9746207976863316
2023-10-25 02:58:59,101 - root - INFO - val ap: 0.9761164739758832, new node val ap: 0.9779584173887048
2023-10-25 02:58:59,163 - root - INFO - start 18 epoch
2023-10-25 02:59:43,834 - root - INFO - epoch: 18 took 44.71s
2023-10-25 02:59:43,834 - root - INFO - Epoch mean loss: 0.29646211194316135
2023-10-25 02:59:43,834 - root - INFO - val auc: 0.9760326878370396, new node val auc: 0.9785357619681678
2023-10-25 02:59:43,834 - root - INFO - val ap: 0.9789281749270833, new node val ap: 0.9811834709379075
2023-10-25 02:59:43,911 - root - INFO - start 19 epoch
2023-10-25 03:00:29,173 - root - INFO - epoch: 19 took 45.32s
2023-10-25 03:00:29,173 - root - INFO - Epoch mean loss: 0.29385695422125846
2023-10-25 03:00:29,173 - root - INFO - val auc: 0.9749946002210408, new node val auc: 0.9762073523752894
2023-10-25 03:00:29,173 - root - INFO - val ap: 0.9779569582515746, new node val ap: 0.9787948808671445
2023-10-25 03:00:29,240 - root - INFO - start 20 epoch
2023-10-25 03:01:14,419 - root - INFO - epoch: 20 took 45.23s
2023-10-25 03:01:14,419 - root - INFO - Epoch mean loss: 0.2886570751590213
2023-10-25 03:01:14,419 - root - INFO - val auc: 0.9705603503306083, new node val auc: 0.9756359618549408
2023-10-25 03:01:14,419 - root - INFO - val ap: 0.973517365377568, new node val ap: 0.9783281166491625
2023-10-25 03:01:14,475 - root - INFO - start 21 epoch
2023-10-25 03:01:59,866 - root - INFO - epoch: 21 took 45.43s
2023-10-25 03:01:59,867 - root - INFO - Epoch mean loss: 0.3043287709432164
2023-10-25 03:01:59,867 - root - INFO - val auc: 0.973404382705463, new node val auc: 0.9774093081766975
2023-10-25 03:01:59,867 - root - INFO - val ap: 0.9764475271271, new node val ap: 0.9797461428378587
2023-10-25 03:01:59,930 - root - INFO - start 22 epoch
2023-10-25 03:02:44,881 - root - INFO - epoch: 22 took 45.00s
2023-10-25 03:02:44,881 - root - INFO - Epoch mean loss: 0.29711095593178394
2023-10-25 03:02:44,881 - root - INFO - val auc: 0.9739799931877513, new node val auc: 0.9762475717209765
2023-10-25 03:02:44,881 - root - INFO - val ap: 0.9768856987539579, new node val ap: 0.9789657889470696
2023-10-25 03:02:44,951 - root - INFO - start 23 epoch
2023-10-25 03:03:29,785 - root - INFO - epoch: 23 took 44.88s
2023-10-25 03:03:29,785 - root - INFO - Epoch mean loss: 0.307691945876811
2023-10-25 03:03:29,785 - root - INFO - val auc: 0.9717923517025858, new node val auc: 0.9766088046380901
2023-10-25 03:03:29,785 - root - INFO - val ap: 0.9746079479629582, new node val ap: 0.9785508854811479
2023-10-25 03:03:29,785 - root - INFO - No improvement over 5 epochs, stop training
2023-10-25 03:03:29,785 - root - INFO - Loading the best model at epoch 18
2023-10-25 03:03:29,793 - root - INFO - Loaded the best model at epoch 18 for inference
2023-10-25 03:03:45,438 - root - INFO - Test statistics: Old nodes -- auc_inherent: 0.9683572219363936
2023-10-25 03:03:45,439 - root - INFO - Test statistics: Old nodes -- ap_inherent: 0.9713584927392781
2023-10-25 03:03:45,439 - root - INFO - Test statistics: New nodes -- auc_inherent: 0.9654133503401361
2023-10-25 03:03:45,439 - root - INFO - Test statistics: New nodes -- ap_inherent: 0.9676717389492436
2023-10-25 03:03:45,439 - root - INFO - Test statistics: Old nodes -- ap: 0.9713584927392781
2023-10-25 03:03:45,439 - root - INFO - Test statistics: Old nodes -- au_roc_score: 0.9683572219363936
2023-10-25 03:03:45,439 - root - INFO - Test statistics: Old nodes -- opt_thr_au_roc: 0.7578691244125366
2023-10-25 03:03:45,439 - root - INFO - Test statistics: Old nodes -- acc_auroc_opt_thr: 0.9142627050820329
2023-10-25 03:03:45,439 - root - INFO - Test statistics: Old nodes -- prec_auroc_opt_thr: 0.9335685108239372
2023-10-25 03:03:45,439 - root - INFO - Test statistics: Old nodes -- rec_auroc_opt_thr: 0.8930432172869146
2023-10-25 03:03:45,439 - root - INFO - Test statistics: Old nodes -- f1_auroc_opt_thr: 0.9120532051575434
2023-10-25 03:03:45,439 - root - INFO - Test statistics: Old nodes -- au_pr_score: 0.971280060971713
2023-10-25 03:03:45,439 - root - INFO - Test statistics: Old nodes -- opt_thr_au_pr: 0.6977196931838989
2023-10-25 03:03:45,439 - root - INFO - Test statistics: Old nodes -- acc_aupr_opt_thr: 0.9135694277711084
2023-10-25 03:03:45,439 - root - INFO - Test statistics: Old nodes -- prec_aupr_opt_thr: 0.9174424313057916
2023-10-25 03:03:45,439 - root - INFO - Test statistics: Old nodes -- rec_aupr_opt_thr: 0.9108183273309323
2023-10-25 03:03:45,439 - root - INFO - Test statistics: Old nodes -- f1_aupr_opt_thr: 0.9134029522823885
2023-10-25 03:03:45,439 - root - INFO - Test statistics: Old nodes -- acc: 0.8959123649459785
2023-10-25 03:03:45,439 - root - INFO - Test statistics: Old nodes -- prec: 0.8636924139084887
2023-10-25 03:03:45,439 - root - INFO - Test statistics: Old nodes -- rec: 0.9415126050420168
2023-10-25 03:03:45,439 - root - INFO - Test statistics: Old nodes -- f1: 0.900595188577182
2023-10-25 03:03:45,439 - root - INFO - Test statistics: New nodes -- ap: 0.9676717389492436
2023-10-25 03:03:45,439 - root - INFO - Test statistics: New nodes -- au_roc_score: 0.9654133503401361
2023-10-25 03:03:45,440 - root - INFO - Test statistics: New nodes -- opt_thr_au_roc: 0.7347480058670044
2023-10-25 03:03:45,440 - root - INFO - Test statistics: New nodes -- acc_auroc_opt_thr: 0.9097420634920637
2023-10-25 03:03:45,440 - root - INFO - Test statistics: New nodes -- prec_auroc_opt_thr: 0.9251166856853058
2023-10-25 03:03:45,440 - root - INFO - Test statistics: New nodes -- rec_auroc_opt_thr: 0.8926587301587303
2023-10-25 03:03:45,440 - root - INFO - Test statistics: New nodes -- f1_auroc_opt_thr: 0.9077115773150979
2023-10-25 03:03:45,440 - root - INFO - Test statistics: New nodes -- au_pr_score: 0.967566430034555
2023-10-25 03:03:45,440 - root - INFO - Test statistics: New nodes -- opt_thr_au_pr: 0.6911842226982117
2023-10-25 03:03:45,440 - root - INFO - Test statistics: New nodes -- acc_aupr_opt_thr: 0.9090013227513231
2023-10-25 03:03:45,440 - root - INFO - Test statistics: New nodes -- prec_aupr_opt_thr: 0.9122298127993518
2023-10-25 03:03:45,440 - root - INFO - Test statistics: New nodes -- rec_aupr_opt_thr: 0.9062698412698414
2023-10-25 03:03:45,440 - root - INFO - Test statistics: New nodes -- f1_aupr_opt_thr: 0.9086763856791563
2023-10-25 03:03:45,440 - root - INFO - Test statistics: New nodes -- acc: 0.8912103174603175
2023-10-25 03:03:45,440 - root - INFO - Test statistics: New nodes -- prec: 0.8599840504050421
2023-10-25 03:03:45,440 - root - INFO - Test statistics: New nodes -- rec: 0.9364285714285714
2023-10-25 03:03:45,440 - root - INFO - Test statistics: New nodes -- f1: 0.8960690517402545
2023-10-25 03:03:45,440 - root - INFO - Saving the model
2023-10-25 03:03:45,710 - root - INFO - TGN model saved
2023-10-25 03:03:45,710 - root - INFO - Run 2, elapsed time: 1096.5476849079132 seconds.
```