Netflix Conductor 4.0: Scaling Workflows to 30,000 Tasks with Asynchronous Evaluation

A workflow orchestrator can coordinate thousands of operations and still hit a scalability ceiling—not because individual tasks are too expensive, but because the engine spends too much time deciding what should happen next.

That was the problem Netflix faced with Conductor as its internal adoption grew. The platform now supports roughly 200,000 workflow definitions across about 150 applications and executes around 420 million workflows in a typical month, according to the Netflix Technology Blog. Its workloads span content and studio production, advertising, games, and other business systems. Source: Netflix Technology Blog

Netflix’s response was not simply to add more compute. It redesigned the evaluation engine and the data model underneath it.

The distinction between releases matters. Conductor 3.0 modernized the data plane; Conductor 4.0 rebuilt the workflow evaluation engine. Netflix says the 4.0 rewrite increased the supported workflow size from approximately 2,500 tasks to 30,000 and reduced p99 workflow-evaluation latency by roughly 40% in production. Those are Netflix-reported results, not universal performance guarantees.

For engineers building data pipelines, feature-engineering systems, media workflows, or multi-step AI applications, the architectural lessons are highly relevant: keep workflow metadata lightweight, avoid loading state that an evaluation does not need, and move contention-prone decision-making out of a synchronous request path.

Why it matters

A workflow engine is responsible for more than executing tasks. Whenever a task completes, fails, or reports progress, the engine must update state, evaluate dependencies, and determine which tasks can run next.

As the graph grows, that coordination can become the bottleneck.

In Netflix’s earlier engine, evaluating a workflow meant loading its entire state into memory. Workflow metadata and task payloads also shared a Cassandra partition, creating large rows and increasing the complexity of concurrent updates. Even if only one task changed, the engine could still need to process far more state than that decision required.

The second problem was synchronization. Workflow evaluation ran inline when updates arrived. Concurrent updates to the same workflow competed for a lock, and unsuccessful attempts could be deferred to a repair process. Under peak load, that caused unpredictable tail latency.

These problems have direct parallels in data and AI infrastructure:

• A feature-engineering workflow with thousands of dependent steps can spend more time coordinating than executing useful work.
• An AI agent that fans out to many tools needs a reliable way to track task status and resume after failures.
• A batch orchestration system can become slow when many workers update a shared execution record.
• Long-running workflows need recovery mechanisms that reconcile missed or delayed updates without corrupting state.

The important performance metric is therefore not just average task duration. It is how efficiently the orchestrator evaluates state, handles concurrent updates, and schedules the next piece of work when the system is under pressure.

What changed: from Conductor 3.0 to 4.0

Netflix describes the evolution as several related changes rather than one isolated rewrite.

Stage	Main architectural development
Conductor 1.0	Dynomite and DynoQueues supported metadata and queueing, with Elasticsearch for indexing
Conductor 2.x	Cassandra became the primary execution datastore, large task inputs and outputs moved to S3, and Timestone replaced DynoQueues
Conductor 3.0	The modernized data plane was consolidated; Dynomite was retired, while Cassandra, Timestone, a Kafka-fed indexer, Elasticsearch, and Iceberg served different roles
Conductor 4.0	The evaluation engine and task data model were redesigned, and evaluation became asynchronous through Timestone

Conductor 3.0 also introduced automated workflow repair and multi-cluster deployment, allowing workflows to run in isolated contexts. These were important foundations, but the engine still carried an older evaluation design.

The 4.0 work addressed that remaining bottleneck. Netflix’s engineering post explains the rewrite in three connected areas: a partitioned task model, safer reconciliation of competing updates, and asynchronous workflow evaluation.

1. Separate workflow metadata from individual task records

The first change was to stop treating an entire workflow execution as one large object that had to be loaded whenever the engine made a decision.

In the redesigned model, workflow metadata is separated from task and user data, and each task has its own record. The engine evaluates a lightweight workflow blueprint and retrieves only the task information needed for the decision at hand.

The principle is familiar from data modelling: separate information with different access patterns instead of storing everything in one oversized record.

Consider a workflow with several thousand tasks. If one task completes and unlocks two downstream tasks, the evaluator generally needs the workflow structure and relevant dependency state—not the payloads for every unrelated task. Keeping the metadata and individual task records separate reduces the amount of state that must be loaded for this decision.

Netflix reports that this redesign reduced the evaluation engine’s memory burden and helped raise the workflow-size ceiling from roughly 2,500 tasks to 30,000.

The improvement does not mean that a 30,000-task workflow is automatically fast. Individual task duration, dependency structure, payload sizes, storage latency, and scheduling pressure still matter. The gain is that the evaluator no longer needs to pay the same full-state loading cost for every decision.

The broader engineering lesson

The same design principle can help workflow engines, event-processing services, and orchestration layers built on Cassandra or other distributed datastores.

Ask three questions:

1. Which fields must be read to make a scheduling decision?
2. Which payloads can be stored separately and fetched only when needed?
3. Does a single record or partition grow with the entire workflow, even when most of its contents are irrelevant to an individual update?

Large records are not inherently wrong, but they become expensive when every small update or decision forces the system to revisit the whole object.

2. Make concurrent task updates safe without relying on conflicting write paths

Separating records solves only part of the problem. A distributed workflow engine must also handle updates that arrive out of order.

Netflix describes a case where a long-running worker sends periodic IN_PROGRESS updates before reporting that the task has completed. Due to network timing, a delayed progress update might arrive after the completion event. Without a rule for reconciling those updates, the task could incorrectly move backward from a terminal state to an in-progress state.

The first approach used Cassandra lightweight transactions to serialize writes. Netflix explains that mixing these with ordinary writes creates consistency challenges because the two paths use different timestamping mechanisms.

The redesigned model separates active task state from terminal task state using two partitions:

• A pending partition stores tasks that are still in flight.
• A terminal partition records a task when it reaches a terminal state, such as COMPLETED or FAILED.

During evaluation, the application reconciles the two. The rule is deliberately simple: a terminal state takes precedence over a non-terminal state. If two writes are of the same category, ordinary last-write-wins behaviour applies.

In the delayed-update example, a late IN_PROGRESS write cannot override the terminal completion record when the engine reconciles state.

This is an application-level consistency rule backed by a data model designed for the lifecycle of a task. It avoids depending on a single lock or a mixed set of write mechanisms to establish the final state.

It is also worth understanding the boundary of the guarantee. This rule resolves the conflict described by Netflix; it is not a blanket guarantee that every failure scenario disappears. Terminal-state writes, storage failures, duplicate events, and application-level side effects still require appropriate handling and testing.

3. Move workflow evaluation off the synchronous request path

The second major bottleneck existed even after the broader data plane had improved.

Previously, a request that started a workflow or updated a task could trigger synchronous workflow evaluation. The evaluator needed a lock for the workflow, so concurrent updates to the same execution competed to acquire it. Under heavy load, unsuccessful lock attempts were deferred to a background repair process that could take minutes to catch up in extreme cases.

That created a gap between receiving an update and making progress on the workflow. Tail latency rose precisely when the system was busiest.

Conductor 4.0 decouples those steps.

When a workflow event arrives, the request no longer has to evaluate the entire workflow inline. Instead, the update enqueues the workflow onto a Timestone exclusive queue. A central asynchronous evaluator processes pending work in order, with the queue ensuring that only one worker evaluates a particular workflow at a time.

This reduces contention on the workflow-level lock because competing updates are serialized through the queue rather than independently attempting to evaluate the same workflow.

The design also separates the responsibilities:

1. Accept an event or state update.
2. Enqueue the workflow for evaluation.
3. Process the workflow’s pending work asynchronously.
4. Schedule the next eligible tasks.
5. Use recovery mechanisms to detect and repair work that does not complete as expected.

The request path becomes less coupled to the amount of evaluation work that follows. It does not mean that all downstream work completes immediately: queue depth and evaluation throughput still affect when the next task is scheduled. The benefit is a more controlled path through contention and burst traffic.

Netflix reports that unsuccessful lock-acquisition attempts—which had reached roughly 2,700 per interval under contention—fell to essentially zero with asynchronous evaluation. It also reports around a 40% reduction in p99 workflow-evaluation latency for the rewritten engine. Source: Netflix Technology Blog

What the results mean

The headline figures describe two related but different improvements.

Metric	Earlier engine	Rewritten engine
Workflow size	Around 2,500 tasks	Up to 30,000 tasks
p99 workflow-evaluation latency	Previous baseline	Approximately 40% lower
Lock contention under reported workload	Significant under peak contention	Unsuccessful acquisitions near zero in the cited observation

These are the results Netflix reports for its own production environment. They show what the redesign achieved at Netflix, but they do not predict a 40% reduction for every deployment or guarantee that a workflow with 30,000 tasks will meet a particular service-level objective.

The result to focus on is the combination: a smaller unit of evaluation, safer task-state reconciliation, and serialized asynchronous evaluation. Optimizing only one of those layers would leave the others as potential bottlenecks.

4. Scale task polling as well as workflow evaluation

A workflow engine can evaluate quickly and still make slow progress if workers cannot pick up scheduled tasks efficiently.

Netflix describes another problem in its task-polling layer. A fixed allocation of worker threads can leave some threads idle on lightly loaded queues while a busy queue accumulates work. CPU may remain underutilized even as queue depth grows.

The updated allocation strategy uses real-time queue depth to rebalance threads across task queues. Every queue receives a minimum allocation so it cannot be completely starved. A floating pool of remaining threads is distributed according to relative workload, while a configurable multiplier limits how quickly the allocation can increase.

This creates a controlled response to uneven demand. Busy queues can gain threads without every queue being granted an unbounded share of the worker pool.

The lesson is that orchestration throughput depends on more than the evaluator. Storage, task scheduling, worker polling, downstream compute, and external services can each become the limiting factor.

A queue-depth-aware thread strategy helps align capacity with work, but it does not remove the need to monitor worker saturation, task latency, and downstream backpressure.

5. Improve concurrency controls and task execution options

The Netflix post also describes changes that affect how workflows are authored and executed.

Run strategies

Conductor previously allowed workflow instances to run in parallel by default. That was suitable for most cases but required teams to implement their own coordination when concurrent runs operated on the same entity.

The Run Strategies feature adds a built-in way to choose how concurrent workflow instances are handled:

• Parallel: the default behaviour, with no additional restriction.
• Sequential: runs at most one workflow at a time for a given sequence key, queuing subsequent requests until the current run finishes.

The sequence key allows related work to be serialized without preventing unrelated entities from being processed concurrently. For example, workflows operating on the same asset can run sequentially, while workflows for different assets continue in parallel.

This is more than a performance feature. It can help prevent duplicate processing and conflicting updates in workloads where ordering matters.

Managed tasks

Conductor also supports task types where the platform drives compute directly, rather than requiring an application to host a long-lived polling worker for every kind of task.

Netflix describes integrations with Titus for container jobs and Stratum for serverless media-processing functions. A workflow can mix regular polled workers, containerized jobs, and serverless functions depending on its needs.

The advantage is that bursty work does not necessarily require a worker fleet to remain running just to poll for tasks. The trade-off is that the orchestration platform still needs to track task lifecycle, timeouts, and results reliably.

Workflow SDK

Netflix is also investing in a type-safe Java DSL for authoring workflows. Rather than maintaining a large JSON graph with string-based references, developers can express control flow in Java while the SDK translates supported constructs into Conductor workflow definitions.

This preserves the workflow graph’s visibility while improving authoring, compile-time checks, testing, and reuse across applications. Netflix says integrated testing and safer schema evolution remain areas of continued development.

Together, these changes show that Conductor 4.0 is not only an engine-performance story. It is also part of a broader effort to make workflow execution and authoring easier to control as the platform grows.

What data and AI engineers should do

The architecture provides several practical patterns for teams building large orchestration systems—whether for data pipelines, ML inference, model evaluation, or multi-step AI agents.

1. Keep the scheduling state lightweight

Avoid loading entire workflow payloads when the scheduler needs only metadata and dependency state. Store large input and output objects separately, and fetch them when a task actually requires them.

Measure the data read for each scheduling decision, not just the total volume stored by the system.

2. Move expensive evaluation out of the request path

Trace the path from a task update to the scheduling of its dependents. If the request is doing substantial graph evaluation while holding a lock, consider an event-queue and evaluator design.

Asynchronous evaluation introduces its own queue and ordering requirements. Define what happens when events are delayed, duplicated, or reordered, and monitor the time from an accepted update to the next scheduling decision.

3. Define task-state transitions explicitly

A retry or delayed heartbeat should not move a completed task back into an active state. Write down the legal transitions, define conflict resolution rules, and test them under concurrent writes and delayed messages.

For critical data workflows, also design idempotent side effects. A workflow engine can track a task’s state while a downstream API still receives duplicate requests if an operation is retried after an uncertain response.

4. Monitor tail latency and queue depth together

Average evaluation time may look healthy even while a small portion of workflows experiences severe delays.

Measure p95, p99, and p99.9 evaluation latency, lock contention where applicable, queue depth, task pickup delay, worker utilization, and recovery time. These signals help distinguish a slow evaluator from a saturated queue or an under-provisioned worker pool.

5. Serialize only where correctness requires it

Sequential execution can prevent races when multiple runs affect the same business entity. But serializing all workflows would unnecessarily reduce parallelism.

Choose a stable sequence key that represents the entity whose operations must not overlap. Keep unrelated entities eligible for concurrent execution, and monitor whether a small number of hot keys create long queues.

6. Treat workflow generation and schema changes like software delivery

If your team generates workflows from code or metadata, include validation of the generated graph in CI. Check task references, input/output schemas, dependency cycles, retries, and backward compatibility before deploying changes.

For AI-agent graphs, validate tool contracts and authorization boundaries as well. The ability to add more tasks is useful only if the resulting workflow remains observable, testable, and safe to recover.

Limitations and open questions

Netflix’s post provides an internal production account, but several details should be treated carefully when applying the lessons elsewhere.

• The performance figures are workload-specific. The 30,000-task capacity and 40% p99 improvement are Netflix-reported results. Different workloads, storage configurations, graph shapes, and concurrency patterns can behave differently.
• Asynchronous work still creates a queueing problem. Moving evaluation off the request path reduces lock contention, but queue depth and evaluator throughput still determine how quickly downstream tasks are scheduled.
• The partitioning strategy has trade-offs. Splitting task state into pending and terminal partitions helps reconcile conflicting updates, but adds reads and a reconciliation rule. Teams need to test the consequences of partial failures and delayed writes.
• Automated repair is not a substitute for application correctness. Recovery can identify and resume stalled workflow work, but it cannot guarantee that external side effects are idempotent or that partial business operations are harmless.
• The open-source story is different from Netflix’s internal implementation. Netflix discontinued maintenance of its public Conductor repository in December 2023 to focus on its internal fork. The 4.0 architecture described in the blog should not be assumed to be available as a drop-in upgrade to the archived open-source project. Source: Netflix Conductor repository
• Some roadmap items are not current guarantees. Netflix says it is considering richer execution policies, deeper agent tooling, integrated testing, and schema-registry safeguards. These are stated directions, not necessarily features already available to all users.

The central design question is whether an orchestrator can make a scheduling decision with a small, well-defined amount of state and without blocking on competing updates. Conductor’s redesign shows how much that question matters as workflows become larger and more concurrent.

Key takeaways

1. Conductor 4.0, not 3.0, is the major evaluation-engine rewrite. Netflix reports raising the workflow-size ceiling from about 2,500 to 30,000 tasks.
2. Separate metadata from task payloads. Evaluating against a lightweight workflow blueprint and retrieving only relevant task state reduces unnecessary memory and data access.
3. Move evaluation off the synchronous request path. Timestone’s exclusive queue serializes evaluation for each workflow and reduces contention between concurrent updates.
4. Make task-state conflict rules explicit. Separating pending and terminal state lets the engine prevent a delayed progress update from overriding a terminal state during reconciliation.
5. Scale worker scheduling as well as evaluation. Queue-depth-aware thread allocation helps direct capacity toward queues with actual demand.
6. Measure the full path to progress. Tail latency, queue depth, task pickup time, and repair behaviour are necessary to understand real orchestration performance.
7. Do not confuse Netflix’s internal engine with the archived open-source version. Check the actual implementation and support model before planning an adoption.

Netflix’s Conductor story is a useful reminder that orchestration can become its own distributed-systems problem. More workers and more storage do not fix an engine that loads too much state or makes every update compete for the same lock. By partitioning task state, making reconciliation explicit, and scheduling evaluation asynchronously, Netflix reports that Conductor 4.0 can handle much larger workflows with lower tail latency.

For data and AI engineers, the broader lesson is to design the control plane as carefully as the compute plane: keep decisions small, make state transitions reliable, and measure how quickly the system turns updates into useful work.

Sources

1. Netflix Technology Blog, Netflix Conductor: The Next Chapter, August 21, 2026.
2. Netflix, Conductor repository and maintenance announcement, December 13, 2023.