Does Luigi Remember Between Runs? A Task Is Done When Its Output Exists. The Next Run Learns Nothing Else.

Short answer: Luigi remembers one thing very reliably, whether a task's output already exists. If the output is there, the task counts as complete and does not run again. If the output is missing, the task runs. Everything else a previous run figured out, decided, or got corrected on is not carried into the next run.

That narrow definition of done is exactly right for batch pipelines. It becomes a problem when a pipeline gains an AI step that is supposed to build on yesterday's work instead of starting from a blank briefing every morning.

What Luigi actually is

Luigi is a Python module built at Spotify for building complex pipelines of batch jobs. A pipeline is a set of tasks with dependencies between them. Each task is a Python class with three core methods: requires() declares the tasks that must finish first, output() returns one or more Target objects, and run() does the actual work.

A Target is usually a file, a database entry, or another concrete artifact. Luigi follows a make-like rule: work only on what is needed to produce the requested output. That rule shapes everything Luigi keeps, and everything it does not.

What counts as memory inside Luigi

Inside Luigi, completion is a file check, not a recollection.

• A task is complete when its outputs exist. Unless complete() is overridden, Luigi checks whether every Target returned by output() exists. If they do, the task is treated as done. If any one is missing, the task is queued to run.
• Dependencies resolve from that same check. Luigi walks the dependency tree from the requested task, stops at tasks that are already complete, and runs only the incomplete branches. A rerun therefore skips finished upstream work without repeating it.
• The output itself is the handoff. A downstream task reads the upstream task's output through input(). The artifact is shared. The reasoning behind the artifact is not. A report file travels forward. The notes about why a source was dropped to produce that report do not, unless a task was explicitly written to save them as another output.

For deterministic data work, this is a clean and honest design. A daily extract either produced today's file or it did not. There is nothing more the next run needs to know.

What the central scheduler adds, and what it does not

Running Luigi with the central scheduler, started as luigid, adds coordination on top of that file check.

The central scheduler exists for two main purposes: making sure two instances of the same task do not run at the same time, and providing a visualization of what is going on. It does not execute tasks itself, and it does not automatically trigger jobs. An external cron entry, a systemd timer, or another caller still starts each pipeline run.

The scheduler also keeps its own state. By default its state is stored at /var/lib/luigi-server/state.pickle and reloaded on restart after a clean shutdown. That state covers scheduled and running tasks, workers, and disables after repeated failures. A separate, experimental task history feature can record executed tasks into a relational database when record_task_history is turned on, which it is not by default. That history exists for people to inspect, with listings of recent runs and their parameters.

None of that state becomes a briefing for the next pipeline run. The scheduler can show that a task failed three times yesterday. The AI step inside today's run does not receive that history, a summary of it, or a warning derived from it. The history is a record to look at, not a memory to act from.

What a new run starts with

When a Luigi pipeline is launched again, for a new date or a new batch, the new run starts with only three things:

• The parameters passed to its tasks for this run
• The task code itself, including its requires(), output(), and run() logic
• Whatever outputs from earlier runs still exist on disk or in a database, visible only as existing or missing

That third item deserves attention, because it is easy to overread. An existing output tells the new run that a task can be skipped. It does not tell an AI step what that earlier run learned while producing the output. Consider a nightly pipeline where an AI step reviews new records, flags the unusual ones, and writes a summary file. On night two, Luigi correctly sees that night one's file exists and moves on. The AI step for night two still has no access to which patterns night one already flagged, which flag a human overruled, or which source proved unreliable. It reviews with fresh eyes, and fresh eyes repeat old mistakes and contradict old calls.

Teams patch this gap by hand. A task reads yesterday's output file back in, parses it, and passes pieces of it forward. Another task maintains a side table of decisions. Each patch works for one pipeline, in one format, maintained by whoever wrote it. The next pipeline invents a different patch, and no other tool can read either one.

Where a shared memory layer fits

The fix keeps Luigi in its role. Luigi should keep deciding what is complete, resolving dependencies, and skipping work whose outputs already exist. The AI steps need a second place to store what they learn, in a form the next run can actually read before it starts.

That is the role Vilix AI is built for. Vilix AI is cloud-hosted, so there is zero infrastructure to run or maintain next to a Luigi deployment. The same memory is available everywhere over MCP, so an AI step inside a Luigi pipeline, a coding assistant, and a chat tool can all read and write the same shared state instead of keeping separate, partial notes.

Vilix AI stores full conversation history, not just extracted facts. When a later run needs to know why a call was made, the original exchange is still there to revisit, not only a one-line conclusion stripped of its context. That depth is what stops recurring runs from drifting apart in their answers week after week.

Vilix AI has a free plan forever, and Pro offers a 7-day trial with no credit card required. Data stays portable the whole time: export everything or delete it anytime.

Get Started for Free Free forever, no credit card.

The bottom line

Luigi remembers between runs in the precise sense its design promises: a task whose output Targets exist is complete, and a task whose outputs are missing will run. The central scheduler adds coordination, visualization, and an optional history for people to inspect. A new run still starts with only its parameters, its code, and a set of existence checks.

If a pipeline contains AI steps that should improve across runs, that improvement needs a memory of its own. Keep Luigi for the dependency graph. Add shared memory for the learning, and night two can finally begin where night one ended.