Does Apache Pulsar Remember Between Runs? Topics Keep Messages Until Every Subscription Acknowledges Them. The Next Run Still Starts With Only Its Message.

Apache Pulsar is a distributed messaging and streaming platform built around topics. A producer publishes a message to a topic, brokers serve that message to subscriptions, and Apache BookKeeper stores the message persistently in ledgers. For a team whose scheduled agents and workers consume those topics, that raises a practical question: when the next run picks up a new message, does it know anything about what the last run concluded? No. Pulsar holds each message exactly as it was published, for exactly as long as its retention rules allow. The next run still starts with only the message in front of it. What the last run found is not passed along.

A topic keeps the message, not the meaning

Pulsar separates serving from storage. Brokers handle delivery, subscriptions, and acknowledgements, while BookKeeper handles the persistent copy. That design makes the system durable and scalable, and it also defines what gets remembered. What is written to a ledger is the published payload, with its properties and schema, in topic order. What a consumer later decides about that payload is never written back into the ledger.

By default, that storage window is narrower than many teams expect. For each topic, brokers retain only messages that are still in at least one backlog. A backlog is the set of messages for a subscription that have not yet been acknowledged. Once a message has been acknowledged on every existing subscription, it is marked for deletion, even if a later reader might have wanted to see it again. A topic with no subscriptions created yet retains nothing by default, because there is no backlog to protect.

Two controls widen that window. A retention policy, set with namespace granularity, keeps acknowledged messages for a configured size limit and time limit, which is what makes replay through the Reader interface useful. A time to live setting does the opposite job: a message not acknowledged within the configured timeframe is automatically acknowledged, so stale work does not sit in a backlog forever. Tiered storage can offload older ledgers to cheaper storage such as S3, so retained messages stay reachable for longer.

Every one of those settings answers a delivery question: how long should published messages remain available, and when is it safe to delete them? None of them stores an outcome. Two consumers can read the same retained message and reach opposite conclusions about it, and the topic records neither conclusion. A retained message is the producer's original payload, unchanged. The topic organizes what was sent. It does not organize what any reader learned from it.

A subscription cursor remembers a position, not a result

Progress in Pulsar lives in subscriptions. A subscription is a named consumption position on a topic, and its cursor tracks which messages have been acknowledged and which are still outstanding. Consumers attached to the same subscription share that position according to the subscription type: exclusive allows one consumer at a time, failover keeps standbys ready to take over, shared distributes messages round-robin across several consumers, and key shared distributes them by key so messages for the same key keep their order.

Acknowledgement is the mechanism that moves the cursor. A consumer can acknowledge a message individually, acknowledge cumulatively up to a position, or negatively acknowledge a message so it is redelivered after a delay. If a worker crashes after receiving a message but before acknowledging it, the message stays in the backlog and is delivered again. That is exactly the protection a reliable pipeline needs, and it is also the whole of what Pulsar hands back. A resumed subscription knows where reading stopped. It does not know what the messages meant, which ones mattered, which checks were already run against them, or what decision the previous run made.

A redelivered message makes the gap concrete. It is the original message delivered a second time. It carries no marker that part of the work was already done, no partial result, and no note from the run that handled it first. The worker has to work all of that out from the payload alone, or repeat the work in full and hope the conclusion comes out the same way twice.

Consider a scheduled AI agent that consumes supplier invoice events from a topic each night and checks them before payment. On Monday it reviews a batch, spots that one supplier submitted the same invoice twice under two different numbers, and marks that supplier as already verified for a bank detail change. On Tuesday, new invoices from the same supplier arrive on the same topic. Without an outside memory, the agent scores each invoice on its own contents and can treat the known duplicate pattern and the completed verification as brand new. The topic kept every invoice that was still inside its retention rules, replayable in order. The investigation still restarted, because a stored event and a stored finding are different things.

What a scheduled Pulsar consumer actually needs

The missing piece is not a longer retention period, a larger backlog quota, or a second topic for notes. Pulsar already does the messaging job well: persistent topics backed by BookKeeper, subscription cursors that resume after a failure, four subscription types for different delivery shapes, and retention and tiered storage controls for how long messages remain. The missing piece is a memory that sits outside any single message, that a consumer can read before it acts and write to when it finishes, and that every other tool in the workflow can also reach.

That memory is what Vilix AI provides as a shared memory layer. Vilix AI is cloud-hosted, so there is zero infrastructure to run or maintain: no extra store to provision and no memory service to deploy next to the brokers. The same memory is available everywhere over MCP, so the worker that consumes the topic, the chat tool used to review its output, and every other connected agent read and write one shared memory instead of separate local notes. And Vilix AI stores full conversation history, not just extracted facts, so a run can revisit what was actually said and decided, in context, rather than working from a compressed summary.

With that layer in place, the Tuesday run changes shape. Before it approves anything, it loads what Monday's run saved: which supplier was already verified, which invoice pattern was already identified as a duplicate, and which invoices were already cleared and why. It spends its time on what is genuinely new, then saves its own findings back, so Wednesday's run starts briefed as well. The topic still delivers the message. The memory delivers the briefing.

Vilix AI is free forever on the free plan, and the 7-day Pro trial needs no credit card. Everything saved can be exported or deleted at any time.

Pulsar will keep holding each message until every subscription acknowledges it or retention runs out, unchanged and replayable. The question is only whether the consumer arrives alone. With a shared memory layer beside it, it arrives knowing what the last run learned.