Here's my attempt at replicating the network described in https://arxiv.org/abs/1606.03073?context=cs. The network was trained on 88 training pairs and 50 epochs, so the results aren't too astounding(according to the paper, their results were achieved using 200,000 iterations, in contrast to 4400 iterations in my case). The loss did seem to drop steadily so the network will probably continue to improve for a while before stopping.