Grouping at replicator and restart case

[Manju050]

Hi,
I have three questions.
1.Do you do grouping of transactions at replicator before sending them to NATS server?
2.Assume you received few changes of a transaction let say 10 inserts to replicator and this 10 inserts are pushed to NATS server.Now before receiving commit from PostgreSQL the replicator goes down.What happens to those 10 messages in NATS server,will the worker read them or not ? or it is like messages are not published to NATS until commit is received,something like you buffer them inside replicator until commit is read?
3.Does each group have corresponding replication slot in PostgreSQL?
Thank you.

[shayonj]

Heya!

For 1&2: Because we use postgres' logical replication, combined with replication slot, pg_flo only sees committed transactions. The operations are then sent in order to NATS. From there the worker also pulls them in order, applies any transformation or rules and then syncs to the appropriate, all in the same order. If a transaction wasn't committed in postgres or rolled back, pg_flo won't see that. Also, for related reasons, it's recommended to only have one worker for now - https://github.com/pgflo/pg_flo?tab=readme-ov-file#scaling-guide. It can be 5-8k ops/s.

For 3: Thats right, one group is basically equivalent of one Publication and replication. Its basically a grouping of all the tables that are being replicated.

Let me know if you have any other questions or feedback, thanks!

[shayonj]

have you explored any other brokers outside of Kafka? also what format are you streaming in json, protobuf or something else

[Manju050]

No i did not explore any other brokers.Currently im using JSON format to stream data.

[shayonj]

got it, are you using pgoutput or wal2json?

[Manju050]

I create own plugin,simple one.

[Manju050]

Hi,
Have you ever tried this scenario..?
First create around 10GB of WAL in postgres by inserts/delets/updates.After that start decoding,in this mean time try to shutdown postgres.Will it shutdown or waits till it sends all the decoded data of 10GB?