Does Apache Spark Remember Between Runs? State Lives in Cache and Checkpoints. The Next Run Still Starts With Only Its State.

Apache Spark remembers on purpose, but only inside a narrow frame. A running application can cache a dataset in memory, a streaming query can checkpoint its offsets and its declared state, and a restarted query can pick up exactly where its checkpoint left off. That memory is real, and it makes large-scale processing practical. What it never keeps is anything an AI agent concluded while working outside the job, so a scheduled agent fed by a Spark pipeline still starts every run with only the state the job was coded to hold.

Cache lives and dies inside one application

Start with the most familiar form of Spark memory. Transformations on an RDD or DataFrame are lazy: nothing computes until an action runs, and without help, every action recomputes its result from the source by walking the lineage, the recorded chain of transformations that built the dataset. Calling cache() or persist() breaks that cycle. Spark stores the computed partitions on the executors, in memory first and on disk when a storage level allows it, and later actions reuse those partitions instead of recomputing them.

Every part of that arrangement is scoped to one application. Cached blocks belong to the running Spark application and its executors. Spark monitors cache use and evicts older partitions under memory pressure, least recently used first, so even inside the application a cached dataset is a performance hint, not a promise. When the application ends, the executors go away and the cache goes with them. The next submission of the same job starts with empty memory and recomputes from its sources, no matter how many times the job ran before.

Shared variables follow the same boundary. A broadcast variable ships a read-only value to each executor once, and accumulators collect counts from tasks back to the driver. Both exist to serve the current application. Neither survives it, and neither was shaped as a record another tool can consult later.

Checkpointing keeps data, not conclusions

Spark offers one heavier tool for batch work. An RDD can be checkpointed, which computes it fully, writes it to a reliable directory such as HDFS or object storage, and cuts its lineage so recovery no longer needs the parent datasets. Checkpointing trades recomputation for durable data: the values survive the application because they were written out as files.

Structured Streaming builds its recovery on the same idea, at larger scale. A streaming query names a checkpointLocation, and Spark writes the query's progress into that directory on durable storage. The checkpoint holds source offsets for each batch, commit records for batches that finished, and a state directory for stateful operators. Restart the query against the same checkpoint and it resumes from the recorded offsets with its state restored. Delete the checkpoint and the query starts over, because the checkpoint was the memory.

Inside that state directory, the contents stay strictly declared. Streaming aggregations, deduplication, stream-stream joins, and arbitrary stateful operations such as mapGroupsWithState, flatMapGroupsWithState, and the newer transformWithState keep their partial results in a state store, a versioned key-value store per operator and partition. Two providers ship with Spark: the default HDFS-backed provider keeps state in executor memory and writes delta files plus periodic snapshots to the checkpoint, while the RocksDB provider keeps state in local RocksDB storage and uploads it for recovery. Both answer the same need: hold typed values the query declared, update them batch by batch, and restore them after a failure.

Cleanup is deliberate, not accidental. A watermark tells Spark how late event-time data may arrive, and once the watermark passes a window, its state can be emitted and evicted. Without a watermark, aggregation and join state grows with all traffic instead of with the window. Spark memory is engineered to stay bounded and recoverable, which means it is also engineered to forget on schedule.

What this means for a scheduled AI agent

Consider a common automation shape on Spark. A Structured Streaming query reads payment events around the clock, aggregates per account inside watermarked windows, and flags accounts whose pattern spikes. A few times a day, a scheduled agent takes the flagged accounts, checks each one against prior incidents, account history, and recent support contact, and settles on an outcome: normal travel, a shared company card, or a genuine takeover attempt.

The Spark memory performs exactly as designed. The checkpoint restores offsets and aggregates after any restart, the state store holds each window until its watermark passes, and the cache keeps repeated actions fast while the application runs. Yet when the agent run opens, the flag alone arrives. Nothing in the state store says that two of today's flagged accounts were investigated last week and cleared as a known accounting integration, because that clearance was a judgment made in a conversation, not a value written to a declared state key. The agent repeats the checks, asks the same questions, and spends its run rebuilding context that already existed outside the job.

The gap widens across tools. The agent's findings might live in a chat with one model, a ticket, and a notebook page. Spark state cannot hold any of it, and the next query window cannot read it. Each run starts capable and uninformed at the same time.

The memory layer beside the job

That second kind of memory needs its own layer. Vilix AI is the shared memory layer that sits beside the job and keeps what the job never contained. Where Spark keeps typed values scoped to one application, one checkpoint, and one window, Vilix AI keeps the full conversation history, not just extracted facts, so the reasoning behind a decision stays readable next to the decision itself.

Vilix AI is cloud-hosted with zero infrastructure to run, so there is no checkpoint directory, state backend, or storage job to manage for agent memory. The same memory is available everywhere over MCP, so an agent in n8n, Make, Zapier, or a custom scheduler reads the same past on every run, on any device and in any tool. A review settled on Monday is retrieved on Friday, with its evidence and outcome intact, instead of being reconstructed from a flag.

Starting takes minutes. The free plan is free forever, and the 7-day Pro trial needs no credit card. Data stays under full control, with export of everything or deletion of everything available at any time. Let Spark remember the state until its checkpoint closes. Let Vilix AI remember everything the state taught the agent.