Taxonomy is correct. But the benefit isn't "table grows indefinitely vs. vacuum-starved death spiral"
in all three approaches, if the consumer falls behind, events accumulate
The real distinction is cost per event under MVCC pressure. Under held xmin (idle-in-transaction, long-running writer, lagging logical slot, physical standby with hot_standby_feedback=on):
1. SKIP LOCKED systems: every DELETE or UPDATE creates a dead tuple that autovacuum can't reclaim (xmin is frozen). Indexes bloat. Each subsequent FOR UPDATE SKIP LOCKED scans don't help.
2. Partition + DROP (some SKIP LOCKED systems already support it, e.g. PGMQ): old partitions drop cleanly, but the active partition is still DELETE-based and accumulates dead tuples — same pathology within the active window, just bounded by retention. Another thing is that DROPping and attaching/detaching partitions is more painful than working with a few existing ones and using TRUNCATE.
3. PgQue / PgQ: active event table is INSERT-only. Each consumer remembers its own pointer (ID of last event processed) independently. CPU stays flat under xmin pressure.
I posted a few more benchmark charts on my LinkedIn and Twitter, and plan to post an article explaining all this with examples. Among them was a demo where 30-min-held-xmin bench at 2000 ev/s: PgQue sustains full producer rate at ~14% CPU; SKIP LOCKED queues pinned at 55-87% CPU with throughput dropping 20-80% and what's even worse, after xmin horizon gets unblocked, not all of them recovered / caught up consuming withing next 30 min.
I think there are two kinds of partition based approach which may cause some confusion if lumped together in this kind of comparison.
Insert and delete with old partition drop vs insert only with old partition drop.
The semantics of the two approaches differ by default but you can achieve the same semantics from either with some higher order changes (partitioning the event space, tracking a cursor per consumer etc).
How does PgQue compare to the insert only partition based approach?
1. partitions are never dropped – they got TRUNCATEd (gracefully) during rotation
2. INSERT-only. Each consumer remembers its position – ID of the last event consumed. This pointer shifts independently for each consumer. It's much closer to Kafka than to task queue systems like ActiveMQ or RabbitMQ.
When you run long-running tx with real XID or read-only in REPEATABLE READ (e.g., pg_dump for long time), or logical slot is unused/lagging, this affects performance badly if you have dead tuples accumulated from DELETEs/UPDATEs, but not promptly vacuumed.
PgQue event tables are append-only, and consumers know how to find next batch of events to consume – so xmin horizon block is not affecting, by design.
in all three approaches, if the consumer falls behind, events accumulate
The real distinction is cost per event under MVCC pressure. Under held xmin (idle-in-transaction, long-running writer, lagging logical slot, physical standby with hot_standby_feedback=on):
1. SKIP LOCKED systems: every DELETE or UPDATE creates a dead tuple that autovacuum can't reclaim (xmin is frozen). Indexes bloat. Each subsequent FOR UPDATE SKIP LOCKED scans don't help.
2. Partition + DROP (some SKIP LOCKED systems already support it, e.g. PGMQ): old partitions drop cleanly, but the active partition is still DELETE-based and accumulates dead tuples — same pathology within the active window, just bounded by retention. Another thing is that DROPping and attaching/detaching partitions is more painful than working with a few existing ones and using TRUNCATE.
3. PgQue / PgQ: active event table is INSERT-only. Each consumer remembers its own pointer (ID of last event processed) independently. CPU stays flat under xmin pressure.
I posted a few more benchmark charts on my LinkedIn and Twitter, and plan to post an article explaining all this with examples. Among them was a demo where 30-min-held-xmin bench at 2000 ev/s: PgQue sustains full producer rate at ~14% CPU; SKIP LOCKED queues pinned at 55-87% CPU with throughput dropping 20-80% and what's even worse, after xmin horizon gets unblocked, not all of them recovered / caught up consuming withing next 30 min.