Does Apache Kafka Remember Between Runs? Records Stay in the Log for Days. The Next Run Still Starts With Only Its Record.

Apache Kafka is a distributed event streaming platform. Producers write records to topics, each topic is split into partitions, and consumers read those partitions at their own pace. For a team whose consumers do AI work on a schedule, that raises a practical question: when a consumer picks up a new record tomorrow, does it know anything about what today's run concluded? No. Kafka holds each record exactly as it was produced, as an entry in an ordered log. The next record still starts with only what its producer wrote into it. What the last run found is not passed along.

Reading a record does not delete it, and keeping it is not remembering it

Kafka is built on a distributed commit log. A partition is an append-only, ordered sequence of records, and every record gets the next offset in that partition: 0, 1, 2, and so on. Producers append to the end. Consumers read forward from an offset they choose. Unlike a classic queue, reading a record does not remove it. Other consumer groups can read the same record, and the same group can read it again later.

Records stay for the topic's retention period, seven days by default, or until a size limit is reached, whether or not anyone has read them. After that window the oldest segments are deleted. On a compacted topic, Kafka instead keeps only the latest record for each key. Both settings answer the same delivery question: how long should produced records remain available to read?

Neither setting stores an outcome. Two consumers can read the same record and reach opposite conclusions about it, and Kafka records neither conclusion. A retained record is the producer's original key, value, timestamp, and headers, unchanged. The log organizes what was produced. It does not organize what any reader learned from it.

An offset remembers a position, not a result

Progress in Kafka is a single number. A consumer group's position in a partition is its committed offset, the offset of the next record it will read. Committed offsets are stored in an internal topic, and the broker keeps them for a configurable period, seven days by default for a group with no active consumers. When a consumer restarts, it resumes from that number and continues forward.

That number is the whole of what Kafka hands back. It says where reading stopped. It does not say what the records meant, which ones mattered, which checks were already run, or what decision the run made. If a group stays idle past the offset retention period, even the number is deleted, and the next start falls back to the configured reset policy: read from the earliest retained record, or skip to the newest one.

Delivery timing adds the same gap from the other side. Kafka delivers at least once in normal setups, so a consumer that crashes after processing a record but before committing its offset will see that record again. The replayed record is the original record, delivered a second time. It carries no marker that part of the work was already done, no partial result, and no note from the run that handled it first. The consumer has to work that out from the record alone.

Consider a scheduled AI consumer that reads order events each night and flags orders that look fraudulent. On Monday it reviews a batch, spots a pattern across several orders from one account, and marks that account as already investigated. On Tuesday new orders from the same account arrive. Without an outside memory, the consumer scores each order on its own contents and can treat the known pattern as brand new. The log kept every order for a week. The investigation still restarted, because a stored event and a stored finding are different things.

What a scheduled Kafka consumer actually needs

The missing piece is not a longer retention period. Kafka already does the streaming job well: a durable ordered log per partition, independent consumer groups reading at their own pace, replay after a failure, and retention and compaction controls for how long records remain. The missing piece is a memory that sits outside any single record, that a consumer can read before it acts and write to when it finishes, and that every other tool in the workflow can also reach.

That memory is what Vilix AI provides as a shared memory layer. Vilix AI is cloud-hosted, so there is zero infrastructure to run or maintain: no extra store to provision and no memory service to deploy next to the brokers. The same memory is available everywhere over MCP, so the consumer that reads the topic, the chat tool used to review its output, and every other connected agent read and write one shared memory instead of separate local notes. And Vilix AI stores full conversation history, not just extracted facts, so a run can revisit what was actually said and decided, in context, rather than working from a compressed summary.

With that layer in place, the Tuesday run changes shape. Before it scores anything, it loads what Monday's run saved: which accounts were already investigated, which pattern was already identified, and which orders were already cleared and why. It spends its time on what is genuinely new, then saves its own findings back, so Wednesday's run starts briefed as well. The log still delivers the record. The memory delivers the briefing.

Vilix AI is free forever on the free plan, and the 7-day Pro trial needs no credit card. Everything saved can be exported or deleted at any time. For a scheduled consumer that redoes finished work, give the next record the last run's memory and watch repetition disappear.

Kafka will keep holding each record in the log until retention runs out, unchanged and replayable. The question is only whether the consumer arrives alone. With a shared memory layer beside the platform, it arrives knowing what the run before it learned.