{"payload":{"feedbackUrl":"https://github.com/orgs/community/discussions/53140","repo":{"id":631755045,"defaultBranch":"main","name":"recurrent-memory-transformer-pytorch","ownerLogin":"lucidrains","currentUserCanPush":false,"isFork":false,"isEmpty":false,"createdAt":"2023-04-24T01:35:47.000Z","ownerAvatar":"https://avatars.githubusercontent.com/u/108653?v=4","public":true,"private":false,"isOrgOwned":false},"refInfo":{"name":"","listCacheKey":"v0:1707676130.0","currentOid":""},"activityList":{"items":[{"before":"d45ef72a40324c6224ffacb890d5593a69db73de","after":"35cd18deeb7965491873fcba4a15d581106eae39","ref":"refs/heads/main","pushedAt":"2024-02-11T18:28:45.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"lucidrains","name":"Phil Wang","path":"/lucidrains","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/108653?s=80&v=4"},"commit":{"message":"address https://github.com/lucidrains/recurrent-memory-transformer-pytorch/issues/22","shortMessageHtmlLink":"address <a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"2129108815\" data-permission-text=\"Title is private\" data-url=\"https://github.com/lucidrains/recurrent-memory-transformer-pytorch/issues/22\" data-hovercard-type=\"issue\" data-hovercard-url=\"/lucidrains/recurrent-memory-transformer-pytorch/issues/22/hovercard\" href=\"https://github.com/lucidrains/recurrent-memory-transformer-pytorch/issues/22\">#22</a>"}},{"before":"90de2ac64c1ce2d2ef90f3b63dbdcecf8af2a024","after":"d45ef72a40324c6224ffacb890d5593a69db73de","ref":"refs/heads/main","pushedAt":"2023-08-31T21:14:24.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"lucidrains","name":"Phil Wang","path":"/lucidrains","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/108653?s=80&v=4"},"commit":{"message":"address https://github.com/lucidrains/recurrent-memory-transformer-pytorch/issues/19 again","shortMessageHtmlLink":"address <a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"1873764721\" data-permission-text=\"Title is private\" data-url=\"https://github.com/lucidrains/recurrent-memory-transformer-pytorch/issues/19\" data-hovercard-type=\"issue\" data-hovercard-url=\"/lucidrains/recurrent-memory-transformer-pytorch/issues/19/hovercard\" href=\"https://github.com/lucidrains/recurrent-memory-transformer-pytorch/issues/19\">#19</a> again"}},{"before":"3be7d43604c6921a7dbdc68f88c7f3c534f82d2a","after":"90de2ac64c1ce2d2ef90f3b63dbdcecf8af2a024","ref":"refs/heads/main","pushedAt":"2023-08-31T19:39:23.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"lucidrains","name":"Phil Wang","path":"/lucidrains","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/108653?s=80&v=4"},"commit":{"message":"address https://github.com/lucidrains/recurrent-memory-transformer-pytorch/issues/19 by allowing for an option to attend to raw read memory positional embeddings on first step","shortMessageHtmlLink":"address <a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"1873764721\" data-permission-text=\"Title is private\" data-url=\"https://github.com/lucidrains/recurrent-memory-transformer-pytorch/issues/19\" data-hovercard-type=\"issue\" data-hovercard-url=\"/lucidrains/recurrent-memory-transformer-pytorch/issues/19/hovercard\" href=\"https://github.com/lucidrains/recurrent-memory-transformer-pytorch/issues/19\">#19</a> by allowing for an option to attend to raw read memory po…"}},{"before":"8f9c3ad32b05dcef028c9de96939234cfa95be3c","after":"3be7d43604c6921a7dbdc68f88c7f3c534f82d2a","ref":"refs/heads/main","pushedAt":"2023-08-29T16:50:08.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"lucidrains","name":"Phil Wang","path":"/lucidrains","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/108653?s=80&v=4"},"commit":{"message":"allow for customizing whether read memory is stop gradded, the one added to the next write mem, if that option is turned on","shortMessageHtmlLink":"allow for customizing whether read memory is stop gradded, the one ad…"}},{"before":"8b82aaafdb6d4de4c8855aef8e7bbc2e0d0e1486","after":"8f9c3ad32b05dcef028c9de96939234cfa95be3c","ref":"refs/heads/main","pushedAt":"2023-08-29T16:30:08.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"lucidrains","name":"Phil Wang","path":"/lucidrains","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/108653?s=80&v=4"},"commit":{"message":"reinject the write memory positions","shortMessageHtmlLink":"reinject the write memory positions"}},{"before":"976af1ecba5c1b421a4e204aab31b297c4302b81","after":"8b82aaafdb6d4de4c8855aef8e7bbc2e0d0e1486","ref":"refs/heads/main","pushedAt":"2023-08-29T16:26:06.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"lucidrains","name":"Phil Wang","path":"/lucidrains","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/108653?s=80&v=4"},"commit":{"message":"address https://github.com/lucidrains/recurrent-memory-transformer-pytorch/issues/17","shortMessageHtmlLink":"address <a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"1871943731\" data-permission-text=\"Title is private\" data-url=\"https://github.com/lucidrains/recurrent-memory-transformer-pytorch/issues/17\" data-hovercard-type=\"issue\" data-hovercard-url=\"/lucidrains/recurrent-memory-transformer-pytorch/issues/17/hovercard\" href=\"https://github.com/lucidrains/recurrent-memory-transformer-pytorch/issues/17\">#17</a>"}},{"before":"14ecab42c63822df406b862c1e0eeceecb8136fb","after":"976af1ecba5c1b421a4e204aab31b297c4302b81","ref":"refs/heads/main","pushedAt":"2023-08-09T14:18:54.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"lucidrains","name":"Phil Wang","path":"/lucidrains","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/108653?s=80&v=4"},"commit":{"message":"give a null key / value to protect against entirely masked out row, as well as to give attention the ability to attend to nothing","shortMessageHtmlLink":"give a null key / value to protect against entirely masked out row, a…"}},{"before":"8d560e973752f8dbcd9b59ef6deafe27ada251c2","after":"14ecab42c63822df406b862c1e0eeceecb8136fb","ref":"refs/heads/main","pushedAt":"2023-08-08T21:04:44.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"lucidrains","name":"Phil Wang","path":"/lucidrains","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/108653?s=80&v=4"},"commit":{"message":"address https://github.com/lucidrains/recurrent-memory-transformer-pytorch/issues/10","shortMessageHtmlLink":"address <a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"1840904350\" data-permission-text=\"Title is private\" data-url=\"https://github.com/lucidrains/recurrent-memory-transformer-pytorch/issues/10\" data-hovercard-type=\"issue\" data-hovercard-url=\"/lucidrains/recurrent-memory-transformer-pytorch/issues/10/hovercard\" href=\"https://github.com/lucidrains/recurrent-memory-transformer-pytorch/issues/10\">#10</a>"}},{"before":"98bf3091a29fbd65dbbb30ce00dd1cadd05fef2d","after":"8d560e973752f8dbcd9b59ef6deafe27ada251c2","ref":"refs/heads/main","pushedAt":"2023-08-08T15:55:26.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"lucidrains","name":"Phil Wang","path":"/lucidrains","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/108653?s=80&v=4"},"commit":{"message":"address https://github.com/lucidrains/recurrent-memory-transformer-pytorch/issues/12","shortMessageHtmlLink":"address <a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"1841062514\" data-permission-text=\"Title is private\" data-url=\"https://github.com/lucidrains/recurrent-memory-transformer-pytorch/issues/12\" data-hovercard-type=\"issue\" data-hovercard-url=\"/lucidrains/recurrent-memory-transformer-pytorch/issues/12/hovercard\" href=\"https://github.com/lucidrains/recurrent-memory-transformer-pytorch/issues/12\">#12</a>"}},{"before":"eee6c349ae61321b4973cd796baf057886497344","after":"98bf3091a29fbd65dbbb30ce00dd1cadd05fef2d","ref":"refs/heads/main","pushedAt":"2023-06-22T18:19:04.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"lucidrains","name":"Phil Wang","path":"/lucidrains","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/108653?s=80&v=4"},"commit":{"message":"update","shortMessageHtmlLink":"update"}},{"before":"a8d434544f1ed3d11984887245f9cf69da67f8de","after":"eee6c349ae61321b4973cd796baf057886497344","ref":"refs/heads/main","pushedAt":"2023-05-27T01:17:36.613Z","pushType":"push","commitsCount":1,"pusher":{"login":"lucidrains","name":"Phil Wang","path":"/lucidrains","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/108653?s=80&v=4"},"commit":{"message":"turns out resiDual is prone to overflowing in fp16. add the scaling solution proposed in the paper","shortMessageHtmlLink":"turns out resiDual is prone to overflowing in fp16. add the scaling s…"}},{"before":"59f4600e9b029017eb36d6dfda07743b9d0bc5c0","after":"a8d434544f1ed3d11984887245f9cf69da67f8de","ref":"refs/heads/main","pushedAt":"2023-05-25T22:34:44.671Z","pushType":"push","commitsCount":1,"pusher":{"login":"lucidrains","name":"Phil Wang","path":"/lucidrains","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/108653?s=80&v=4"},"commit":{"message":"adopt the resiDual norm pattern, may help with vanishing gradients in the context of recurrent transformers","shortMessageHtmlLink":"adopt the resiDual norm pattern, may help with vanishing gradients in…"}},{"before":"d1f37abd751aa7ca3cccfcc03f6d3add599b0788","after":"59f4600e9b029017eb36d6dfda07743b9d0bc5c0","ref":"refs/heads/main","pushedAt":"2023-05-03T20:08:12.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"lucidrains","name":"Phil Wang","path":"/lucidrains","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/108653?s=80&v=4"},"commit":{"message":"incorporate a lesson from block recurrent transformer - reinject each read memory position its own embedding","shortMessageHtmlLink":"incorporate a lesson from block recurrent transformer - reinject each…"}},{"before":"cb42cace47809246481cf10b010f42843804a6e6","after":"d1f37abd751aa7ca3cccfcc03f6d3add599b0788","ref":"refs/heads/main","pushedAt":"2023-05-03T20:05:08.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"lucidrains","name":"Phil Wang","path":"/lucidrains","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/108653?s=80&v=4"},"commit":{"message":"default the xl memories to be on, suspect could make training more stable","shortMessageHtmlLink":"default the xl memories to be on, suspect could make training more st…"}},{"before":"578443a037c2c87348fa2e6ace0292796ec5f207","after":"cb42cace47809246481cf10b010f42843804a6e6","ref":"refs/heads/main","pushedAt":"2023-05-03T18:16:10.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"lucidrains","name":"Phil Wang","path":"/lucidrains","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/108653?s=80&v=4"},"commit":{"message":"make rotary embeddings work with xl memories, project manage","shortMessageHtmlLink":"make rotary embeddings work with xl memories, project manage"}},{"before":"c7e7392a4c2d2f6bd1e34b672bcc5619f01bba0d","after":"578443a037c2c87348fa2e6ace0292796ec5f207","ref":"refs/heads/main","pushedAt":"2023-05-03T17:14:43.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"lucidrains","name":"Phil Wang","path":"/lucidrains","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/108653?s=80&v=4"},"commit":{"message":"allow for norming the write memories being passed on to the next recurrent step","shortMessageHtmlLink":"allow for norming the write memories being passed on to the next recu…"}},{"before":"24e885577a6786e4b97eb10f61cadd01a92a9106","after":"c7e7392a4c2d2f6bd1e34b672bcc5619f01bba0d","ref":"refs/heads/main","pushedAt":"2023-05-03T15:55:57.000Z","pushType":"force_push","commitsCount":0,"pusher":{"login":"lucidrains","name":"Phil Wang","path":"/lucidrains","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/108653?s=80&v=4"},"commit":{"message":"allow one to do truncated bptt as well as memory replay backprop","shortMessageHtmlLink":"allow one to do truncated bptt as well as memory replay backprop"}},{"before":"d40006c93ff11b0ae2b4004749580c7c05e3f021","after":"24e885577a6786e4b97eb10f61cadd01a92a9106","ref":"refs/heads/main","pushedAt":"2023-05-03T14:41:38.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"lucidrains","name":"Phil Wang","path":"/lucidrains","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/108653?s=80&v=4"},"commit":{"message":"allow one to do truncated bptt as well as memory replay backprop","shortMessageHtmlLink":"allow one to do truncated bptt as well as memory replay backprop"}},{"before":"e2c5836cc96ac0e4bb47018ff2352cacca72076c","after":"d40006c93ff11b0ae2b4004749580c7c05e3f021","ref":"refs/heads/main","pushedAt":"2023-04-27T22:15:44.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"lucidrains","name":"Phil Wang","path":"/lucidrains","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/108653?s=80&v=4"},"commit":{"message":"link to Yannic","shortMessageHtmlLink":"link to Yannic"}},{"before":"1a218931b56407ccc39dd6510965b284e31fc8b2","after":"e2c5836cc96ac0e4bb47018ff2352cacca72076c","ref":"refs/heads/main","pushedAt":"2023-04-27T16:48:33.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"lucidrains","name":"Phil Wang","path":"/lucidrains","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/108653?s=80&v=4"},"commit":{"message":"note to future self","shortMessageHtmlLink":"note to future self"}},{"before":"01821308020b0b490b62c112c1a570374b72b7b9","after":"1a218931b56407ccc39dd6510965b284e31fc8b2","ref":"refs/heads/main","pushedAt":"2023-04-26T20:11:08.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"lucidrains","name":"Phil Wang","path":"/lucidrains","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/108653?s=80&v=4"},"commit":{"message":"thought","shortMessageHtmlLink":"thought"}},{"before":"181b3de0a077f084b4109b44f7c0135f5ec96628","after":"01821308020b0b490b62c112c1a570374b72b7b9","ref":"refs/heads/main","pushedAt":"2023-04-26T16:16:24.000Z","pushType":"force_push","commitsCount":0,"pusher":{"login":"lucidrains","name":"Phil Wang","path":"/lucidrains","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/108653?s=80&v=4"},"commit":{"message":"readme","shortMessageHtmlLink":"readme"}},{"before":"4fffc604f01682ac7917abb88f6aa63ba5ad5ea6","after":"181b3de0a077f084b4109b44f7c0135f5ec96628","ref":"refs/heads/main","pushedAt":"2023-04-26T16:10:11.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"lucidrains","name":"Phil Wang","path":"/lucidrains","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/108653?s=80&v=4"},"commit":{"message":"product management","shortMessageHtmlLink":"product management"}},{"before":"a44495a6cde7b0dc731b9c0c939d6c1d43ea2ccd","after":"4fffc604f01682ac7917abb88f6aa63ba5ad5ea6","ref":"refs/heads/main","pushedAt":"2023-04-26T15:59:25.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"lucidrains","name":"Phil Wang","path":"/lucidrains","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/108653?s=80&v=4"},"commit":{"message":"readme","shortMessageHtmlLink":"readme"}},{"before":"f1b4fde2410539d30c39a2f5ef48ad97aeb3fdb3","after":"a44495a6cde7b0dc731b9c0c939d6c1d43ea2ccd","ref":"refs/heads/main","pushedAt":"2023-04-26T14:44:21.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"lucidrains","name":"Phil Wang","path":"/lucidrains","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/108653?s=80&v=4"},"commit":{"message":"last commit for this repo for the week","shortMessageHtmlLink":"last commit for this repo for the week"}},{"before":"ced823b1a5362164c0c7b4868941f91dd59da659","after":"f1b4fde2410539d30c39a2f5ef48ad97aeb3fdb3","ref":"refs/heads/main","pushedAt":"2023-04-26T14:31:43.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"lucidrains","name":"Phil Wang","path":"/lucidrains","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/108653?s=80&v=4"},"commit":{"message":"add the enhanced xl recurrence trick","shortMessageHtmlLink":"add the enhanced xl recurrence trick"}},{"before":"f30f816186bc52f2818831b4cf7ca25829e8bc02","after":"ced823b1a5362164c0c7b4868941f91dd59da659","ref":"refs/heads/main","pushedAt":"2023-04-26T14:24:54.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"lucidrains","name":"Phil Wang","path":"/lucidrains","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/108653?s=80&v=4"},"commit":{"message":"add ability to use xl memories, with length customizable. original neurips paper combined txl with rmt for best wiki 103 results","shortMessageHtmlLink":"add ability to use xl memories, with length customizable. original ne…"}},{"before":"9203fb1f9f8feb51c1243b8aad8a118908cc2c85","after":"f30f816186bc52f2818831b4cf7ca25829e8bc02","ref":"refs/heads/main","pushedAt":"2023-04-26T13:44:28.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"lucidrains","name":"Phil Wang","path":"/lucidrains","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/108653?s=80&v=4"},"commit":{"message":"one more cite for normformer norm in ff","shortMessageHtmlLink":"one more cite for normformer norm in ff"}},{"before":"24285bba26da0941391c08dd4d454495726224ff","after":"9203fb1f9f8feb51c1243b8aad8a118908cc2c85","ref":"refs/heads/main","pushedAt":"2023-04-26T13:41:59.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"lucidrains","name":"Phil Wang","path":"/lucidrains","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/108653?s=80&v=4"},"commit":{"message":"cite for biasless transformer","shortMessageHtmlLink":"cite for biasless transformer"}},{"before":"cdff28aa7e9a60f26adfaf3c47187b4666ac50c2","after":"24285bba26da0941391c08dd4d454495726224ff","ref":"refs/heads/main","pushedAt":"2023-04-26T13:36:15.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"lucidrains","name":"Phil Wang","path":"/lucidrains","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/108653?s=80&v=4"},"commit":{"message":"bring in a trick from the cogview paper","shortMessageHtmlLink":"bring in a trick from the cogview paper"}}],"hasNextPage":true,"hasPreviousPage":false,"activityType":"all","actor":null,"timePeriod":"all","sort":"DESC","perPage":30,"cursor":"djE6ks8AAAAD99pREwA","startCursor":null,"endCursor":null}},"title":"Activity · lucidrains/recurrent-memory-transformer-pytorch"}