back-pressure / keep alive reply

[DaemonSnake]

issue:

Currently once the Replication.Server decodes the data payload from Postgres it casts it to Replication.Publisher.

In cases where big changes occurrs (for instance a delete or insert of X * Mil records),
as the message will get stored in-memory,
WalEx can end-up consuming all of its available memory.

For temp slots, this is not that critical as when Walex will restart it will ignore the problematic transaction and start at the latest transaction instead.

But for durable replication slots, the issue becomes that Walex will retry multiple times that transaction, creating strain on the CPU of the Postgres database and in the mid-time the WAL files will start to pile on.
After a long while, as replies to the keep alive increases the restart_lns field of the durable slot, the trasaction will get dropped but it can take a very long time.

proposal:

To fix this we propose to make the way Server communicates with Publisher configurable,
the idea would be to add a message_middleware option that would take the current decoded server message and app_name.
This callback would then become responsible to publish the message to Publisher.

This would allow for user-define back-pressure.
For instance the callback could write the messages to the local disk, an SQS FIFO queue, etc.
It would then become the responsibility of the end-user to read these messages and to send them to WalEx.Replication.Publisher
to continue the processing (using for instance Broadway).

Not fixed by this proposal:

Walex will still require that the full transaction be held in memory by Publisher before its sent to Destination,
but that's a seperate issue and the size of the in-memory final transaction should be much less than
all the binary payloads with the associated Relation, Commit, Begin, etc.

[DaemonSnake]

implementation proposal done in #65

[cpursley]

Any thoughts on this @DohanKim ?

[DaemonSnake]

Addition information will be usefull here.

When a big transaction occurs, Walex will receive a big number of request in a row.
If while processing these events it takes more than wal_sender_timeout (1m default) to respond to Postgres the connection will get killed.
when Walex will restart, it will at the restart_lsn wal address (and in all likelyhood a kind of infinite loop of partial processing / restarts).

For that reason message_middleware must reply as fast as possible to avoid this case and the processing must not take too much ressources to process otherwise the above describe situation will occur.

(FYI: we are currently testing Walex integration on our side and are running in this case)

[cpursley]

Also, tests passed. What other changes (if any) are planned on this branch?

I really appreciate all the contributions!

[DaemonSnake]

When you have time, could you write up some details (example) on how to set this up with Broadway for the docs?

yes for sure ^^
this week is a bit of a rush but will have more time maybe next week

Also, tests passed. What other changes (if any) are planned on this branch?

I really appreciate all the contributions!

Thank you @cpursley 🙌
I don't expect more changes on my side for the PR.
I think adding some tests would be nice, will try to do it.

[cpursley]

More tests would be great; I'll keep this open for longer. Then I'm going to refactor the destinations stuff and cut a new release.

[cpursley]

@DaemonSnake do you mind writing some docs on this (can be in the README - and based on what you write here in the discussion). Then I will merge in 😃

[DaemonSnake]

hello @cpursley,
very sorry for the disappearing act :(
It was quite the rush and didn't manage to find time to do it earlier
I opened a new PR to add some documentation about it and durable slot in the README.
It also contains a fix for an issue we found in production.
#70