[WIP] save previous output and state of RNNs

[superbock]

This change enables recurrent neural networks to be initialised with an initial output/state (#230), to be streaming compatible (#185), and also fixes some GRU related parameter glitches (#234).

Unfortunately, this change breaks the API, since parameters are reordered and renamed.
Any thoughts?

[fdlm]

I see two issues here:

• Now I don't like the name of the parameters (prev_output) :), because from a user's perspective, I don't understand why I should give an RNN layer a "previous output" when creating it. I might want to set the "first value", or the "initial value" of the layer, but the name prev_output seems counter-intuitive for this (even though it is conceptually the same under the hood!). I would rather go with init (and set self.prev_output = init in the __init__ method).
• I didn't check this, but in non-streaming mode, calling the RNN twice for the same input would return different results, because the initial value of the units is overwritten (it would be the last output of the first call). I think this will be a general problem for all "stateful" processors (HMM, CRF, ...), and I don't have a solution except having a different process method for streaming and non-streaming mode.

[fdlm]

Is this necessary to ensure backwards compatibility with older models?

[superbock]

Yes, otherwise we can not load the old models any more, since they don't have these attributes.

[fdlm]

Shouldn't these initialisations go to the Gate class, as discussed in #233 ?

[superbock]

I thought about this as well, but they are technically not connected to the gates. The prev_output is the previous output of the unit, prev_state is the state, which is the cell weighted by the input gate plus the previous state weighted by the forget gate. Thus I thought it would be the easiest to add them to the layer directly.

[superbock]

Regarding the issues you mentioned:

• As indicated elsewhere, I also voted for init, instead of prev_output, at least in the __init__() method. I just used them for clarity right now, since this PR is WIP. I should have indicated this somewhere.
• Good point! I thought about adding a reset() method to BufferProcessor (or all classes buffering something) to clear the buffer. Although this works, this will be a problem, because people will forget to reset the buffers -- like I forgot to think about that in the first place.

I tried (and still want) to avoid having two different process methods, simply because this will introduce a lot of duplicated code.

How about the following:

We add a second buffer() method to buffer the data and then just use the old, unmodified process() method to do the actual work. This would require additional buffer() calls in processors.process_online(), but would not change the behaviour of existing code nor would require code changes (hopefully). Maybe we need two methods, one before process and one after, but you get the idea.

Processor will simply raises a NotImplementedError if these methods are called. This way we can easily control which Processors are streaming-capable or not, by simply implementing these methods.

Any thoughts?

[superbock]

The current state allows setting the initial state of recurrent layers (and cell for LSTMLayer), but does not deal with proper resetting yet.

As outlined here, it involves a lot of if/elif/else statements to get this right. Waiting for some feedback before proceeding further.

[superbock]

Closing, since #243 is merged.