Does Apache Beam Remember Between Runs? State Lives Per Key and Window. The Next Run Still Starts With Only Its State.

Apache Beam remembers on purpose, but only inside a narrow frame. A pipeline can keep a value, a bag of buffered elements, or a map for each key and window, and a timer can call the code back when time moves forward. That state is real, and it makes streaming aggregation possible. What it never keeps is anything an AI agent concluded while working outside the pipeline, so a scheduled agent fed by a Beam pipeline still starts every run with only the state the pipeline was coded to hold.

A model, not an engine

Beam is a unified programming model for batch and streaming pipelines, not an execution engine of its own. The same pipeline code runs on different runners, such as Apache Flink, Apache Spark, Google Cloud Dataflow, or the Direct Runner used for local testing. The runner translates the pipeline, runs it, and provides the machinery underneath, including how state is actually stored.

That split matters for memory. Beam defines what state means in the pipeline: which function can read it, which key it belongs to, and which window bounds it. The runner decides where the bytes sit while the pipeline runs. Asking whether Beam remembers is therefore really two questions. What does the model allow a pipeline to keep, and how long does that keeping last? Both answers point to the same boundary: state exists to make one pipeline compute correctly.

State belongs to a key and a window

Stateful processing in Beam lives inside a DoFn, the function applied by a ParDo transform to each element. To use state, the input must be keyed, because state is partitioned per key. Each key gets its own state, invisible to every other key, and there is no global mutable state shared across keys.

The state itself is also bounded by window. A value kept for a key in a ten-minute window belongs to that window. When the window ends, its state expires with it, unless a timer or a global window changes the picture. State is not a notebook the pipeline accumulates forever. It is a working area tied to a slice of the stream.

Inside that working area, the shapes are specific. ValueState holds a single value per key and window, such as a running count. BagState holds an unordered collection of elements, useful for buffering records before a batched call. MapState and SetState hold keyed entries and unique values. OrderedListState holds elements in timestamp order on runners that support it. Every shape answers the same kind of need: hold structured data the code declared in advance, update it as elements arrive, and read it back inside the same stateful function.

Notice what is missing from that list. There is no shape for a note, a reason, or a judgment. A state spec is declared in code, with a type, before the pipeline runs. Anything nobody declared a spec for has nowhere to land.

Timers decide when state gets used

Timers complete the model. A timer requests a callback at a future point, and when it fires, the runner calls the matching OnTimer method with the state still available. Beam supports two time domains. An event-time timer fires when the watermark passes its timestamp, which signals that the runner estimates input up to that point is complete. A processing-time timer fires when wall-clock time passes, such as a set delay after buffering began.

The classic uses show the intent. Buffer elements in BagState and flush them when a processing-time timer fires, so a batched request goes out without waiting forever. Hold a partial result and emit it when an event-time timer fires at the end of a window. Set a timeout, and if no further input arrives before it fires, clear state and emit a different output. Timers manage the lifecycle of pipeline state: when to flush it, when to emit from it, and when to let it go.

Read state and timers together and the design is clear. Beam gives fine-grained control over per-key data inside a running pipeline, portable across runners. It does not give any other tool a way to come back later and ask what happened.

What this means for a scheduled AI agent

Consider a common automation shape on Beam. A streaming pipeline reads click events all day, keeps per-user state in fixed windows, and flags sessions whose pattern looks unusual. Every hour, a scheduled agent takes the flagged sessions, checks each one against account history, prior warnings, and recent support contact, and decides whether to do nothing, send a notice, or open a review.

The pipeline memory performs exactly as designed. Counts and buffers survive inside their windows, timers flush results on schedule, and the runner keeps state consistent while the job runs. Yet when the agent run opens, the flag alone arrives. Nothing in per-key state says that two of today's flagged sessions were reviewed yesterday and cleared as a known test account, because that clearance was a judgment made in a conversation, not a value written to a state spec. The agent repeats the checks, reaches out again, and spends its run rebuilding context that already existed outside the pipeline.

The gap widens across tools. The agent's findings might live in a chat with one model, a ticket system, and a notebook. Beam state cannot hold any of it, and the next pipeline window cannot read it. Each run starts capable and uninformed at the same time.

The memory layer beside the pipeline

That second kind of memory needs its own layer. Vilix AI is the shared memory layer that sits beside the pipeline and keeps what the pipeline never contained. Where Beam keeps typed values scoped to one key and one window, Vilix AI keeps the full conversation history, not just extracted facts, so the reasoning behind a decision stays readable next to the decision itself.

Vilix AI is cloud-hosted with zero infrastructure to run, so there is no state backend, cluster, or storage job to manage for agent memory. The same memory is available everywhere over MCP, so an agent in n8n, Make, Zapier, or a custom scheduler reads the same past on every run, on any device and in any tool. A review settled on Monday is retrieved on Friday, with its evidence and outcome intact, instead of being reconstructed from a flag.

Starting takes minutes. The free plan is free forever, and the 7-day Pro trial needs no credit card. Data stays under full control, with export of everything or deletion of everything available at any time. Let Beam remember the state until its window closes. Let Vilix AI remember everything the state taught the agent.