Does Redpanda Remember Between Runs? Records Stay in the Log Until Retention Trims Them. The Next Run Still Starts With Only Its Record.

Redpanda remembers records for exactly as long as retention allows, and no longer. A producer writes a record to a topic, the record sits inside a partition log, and any consumer can read it, replay it, or read it again from an earlier offset. That replayable log is real memory, built carefully and on purpose. What the log never keeps is anything a consumer learned while processing a record, so an AI agent fed by Redpanda still starts every run with only the record in front of it.

A streaming log built to keep records replayable

Redpanda is a streaming platform that speaks the Kafka protocol, so the storage model will feel familiar: data lives in topics, topics split into partitions, and partitions are written as segment files on disk. A producer appends a record and moves on. The record stays in the log whether or not anyone has consumed it yet, and it stays readable after consumption too. Unlike a queue, reading does not remove anything.

That design makes the past unusually accessible. A consumer group tracks an offset for each partition, which is simply a bookmark: the position of the next record to read. Move the bookmark backwards and the same records stream past again, in the same order. A new consumer can start from the earliest offset and rebuild a picture of everything the topic still holds.

Tiered Storage stretches that window much further. Closed segments are offloaded automatically to object storage such as S3, Google Cloud Storage, or Azure Blob Storage, and the local disk becomes a cache in front of a much larger archive. A topic can hold far more history than the brokers could ever store locally, and Redpanda indexes the offloaded segments so old records can still be fetched on demand. As storage, this is generous. As memory for an agent run, its limits show up in what the log contains: records exactly as produced, and nothing else.

Retention trims the log, and an offset only marks a position

Nothing in the log is permanent. Retention is governed per topic by two settings working together. Time-based retention, set with retention.ms at the topic level and log_retention_ms at the cluster level, with a cluster default of 604800000 milliseconds, or seven days, decides how long records stay before becoming eligible for deletion. Size-based retention, set with retention.bytes, caps how large each partition may grow. Both limits apply per partition, and only closed segments can be deleted, so a record survives until its whole segment ages or grows past the limit. On Redpanda Streaming clusters, local disk retention defaults to 24 hours, and on Redpanda Cloud topics to 6 hours, with Tiered Storage holding the longer history behind it.

Compaction offers a different shape of forgetting. A topic with a compact cleanup policy keeps the latest value for each key and drops older values for that key over time. The topic becomes a current-state table rather than a history. Useful, but the surviving record is still the producer's record, not a consumer's conclusion.

Consumer offsets expire too. Redpanda supports offset retention for consumer groups, with periodic expiration that starts counting once a group empties or a standalone consumer commits for the last time, plus an API for deleting offsets deliberately. Read the whole system together and the pattern is consistent. Segments, compaction, tiered archives, and offsets all answer storage and delivery questions: how long should this record stay readable, and where did each reader stop? None of them records what a reader decided, discovered, or produced while reading.

What this means for a scheduled AI agent

Consider a common automation shape on Redpanda. An orders topic streams purchase and refund events all day. Every hour, a scheduled agent consumes the newest records, checks each case against account history, and decides whether to approve a refund, flag a repeat issue, or escalate. The agent's work is judgment work: comparing this event with earlier ones, applying a precedent, and choosing a tone for the customer reply.

The next run begins an hour later with a fresh batch of records. Replay could hand the agent the original events again, word for word, but the agent would have to redo the entire investigation to recover what it concluded last time, and retention may already have trimmed the oldest events out of the log. What the run actually needs is small and specific: this customer was refunded twice in March, the exception was approved for this reason, escalation worked better than a template reply. None of that was ever produced as a record, so none of it is in the topic, no matter how long retention runs or how deep the tiered archive goes.

Operators often close the gap by producing summary records back into a second topic. That helps one consumer, in one format, at whatever moment the summary was written. It breaks down when a second tool needs the same history, when the summary schema changes, or when the reasoning behind a decision matters more than the decision itself. A log answers whether a record exists and whether it is still within retention. Agent work depends on what has been learned so far. Those are two different jobs, and they need two different systems.

Give the agent a memory that outlives the record

Vilix AI is the shared memory layer that sits beside the log and keeps what the log never contained. Where Redpanda keeps records exactly as produced, Vilix AI keeps the full conversation history, not just extracted facts, so the reasoning behind a decision stays readable next to the decision itself. The next run does not need to replay a topic and re-derive its conclusions. It retrieves the relevant past at the start of the run.

Vilix AI is cloud-hosted with zero infrastructure to run: no cluster to size for memory, no retention window to gamble on, and no archive tier to manage. Agents connect over MCP, so the same memory follows the work everywhere, from a scheduled consumer reading a Redpanda topic to an n8n flow, a Make scenario, or a chat session the next day, across tools and devices. Every step reads and writes one shared state instead of a summary buried in a side topic.

The working pattern stays simple. Records still stream through Redpanda, which remains the right tool for durable, replayable, high-throughput delivery. At the start of a run, the agent step pulls its context from Vilix AI. At the end of the run, it saves what it did, what it decided, and what it noticed. The log trims on schedule, exactly as designed, and the agent forgets nothing, exactly as required.

Starting takes minutes. The free plan is free forever, and the 7-day Pro trial needs no credit card. Data stays under full control, with export of everything or deletion of everything available at any time. Let Redpanda remember the record until retention says otherwise. Let Vilix AI remember everything the record taught the agent.