How IFCO Cut a 7-Hour dbt Job to 2 Hours 20 Minutes on Databricks

A data pipeline does not need a radical rewrite to become dramatically faster. Sometimes, the biggest gains come from making an incremental model truly incremental—and finding out what the query engine actually executed rather than trusting what the transformation code appears to say.

That is the central lesson from IFCO’s work with Databricks Forward Deployed Engineering. IFCO, whose reusable packaging network operates across more than 50 countries, processes large volumes of tracking data to calculate business-critical metrics such as cycle time, loss, breakage, washing costs, and pool size. Late-arriving events complicate the job: historical records may need reconciliation even after an earlier run has reported its results.

According to Databricks, IFCO reduced the runtime of its core semantic-layer job from approximately seven hours to 2 hours 20 minutes—a reduction of about 66%. Daily compute costs fell by 58%, and the team retired the nightly full refresh it had previously needed to maintain KPI accuracy. The consolidation model also scanned about 75% fewer rows, while only around 3–5% of the relevant assets needed to be recomputed on each run. Source: Databricks Engineering

The changes were not one magic setting. They combined more deliberate incremental writes, execution-plan-based diagnosis, and a different way of operating the dbt dependency graph on Databricks.

For data engineers maintaining large transformation projects, the lesson is practical: optimize the rows and files a model touches, then make every model independently observable and recoverable.

Why it matters

At large scale, the cost of a transformation depends not only on the SQL it expresses but also on how the database reads and writes data to execute it.

An incremental dbt model is meant to avoid rebuilding a whole table when only a small part has changed. Yet a configuration that looks incremental can still trigger broad scans or writes. A changing timestamp, an unbounded time window, unnecessary columns, or a poor match between the model’s grain and its write strategy can quietly turn a small update into a large operation.

This becomes particularly important when source events can arrive late. A pipeline needs to reconcile recent history without rescanning or rewriting far more data than correctness requires. Too narrow a window risks missing corrections; too broad a window wastes compute.

The operational problem grows when hundreds of models run inside a single dbt command. If one model fails near the end of a long job, diagnosing the failure and deciding what to rerun can be harder than fixing the SQL itself. Aggregate logs also make it difficult to see which model is responsible for a runtime or cost regression.

IFCO’s approach addresses both dimensions: reduce unnecessary data work inside each model, and reduce the operational opacity of the overall graph.

1. Diagnose the query that actually ran

One of the first lessons from IFCO’s optimization work is deceptively simple: compiled dbt SQL is not necessarily the complete story of what Databricks executes.

dbt compiles model code and macros into SQL, but the runtime execution path can add behaviour associated with materialization and incremental strategy. Reviewing the compiled statement alone may therefore fail to explain the scans and writes visible in production.

Databricks and IFCO used actual execution information from query history to investigate the problem. That allowed them to examine the executed query plan and look for evidence such as rows scanned, write volume, spill, and the operators responsible for expensive work.

The distinction is important. A model may look incremental in the repository but still touch too much of its target table. An optimization should be grounded in observable runtime behaviour rather than a guess based on the SQL’s appearance.

A useful diagnostic workflow is:

1. Find the model consuming the most time or compute.
2. Inspect its executed query and physical plan, not just its compiled SQL.
3. Identify which operators scan or rewrite more data than the model’s business logic requires.
4. Make one targeted change and remeasure the same metrics.
5. Confirm that the resulting table remains correct, including when late data arrives.

This is more repeatable than changing cluster size or tuning multiple configurations simultaneously. It also creates evidence that can be reviewed by the engineers responsible for the model.

2. Make incremental writes genuinely incremental

The core performance work focused on how dbt incremental models interact with Delta Lake table layout and write operations.

Three considerations stand out.

Match the write strategy to the data changes

A MERGE can be appropriate when incoming records need to update matched rows and insert new ones. But matching source rows against a large target can be expensive if the operation scans a broad portion of the table.

A delete-plus-insert strategy can be a better fit for certain replacement patterns—for example, when the affected logical slice can be identified and replaced as a unit. It is not universally faster or interchangeable with MERGE: the right choice depends on the model’s grain, update semantics, uniqueness assumptions, and the need to preserve records outside the replacement scope.

The important question is not which write operation is fashionable. It is which operation rewrites the smallest correct portion of the target.

Align liquid clustering with access patterns

Liquid clustering can help Delta tables organize data around useful query dimensions without relying on a permanently fixed partition layout. But clustering is not a performance switch that automatically makes every write efficient.

Clustering columns should reflect how the workload filters and accesses data. If an incremental model can identify the relevant time or business-key range and the layout supports pruning, Databricks may need to scan fewer files to locate the rows affected by a write.

Clustering, incremental filters, and the write strategy should therefore be designed as a single decision. An inappropriate clustering key—or an operation that cannot take advantage of it—may deliver little benefit.

Bound the work required to reconcile history

Late-arriving data requires a clear reconciliation policy. Time-bounded processing can limit how much recent history a model revisits, but the boundary must be wide enough to capture expected lateness and corrections.

If data can arrive outside that window, the pipeline needs a separate mechanism to identify and repair older records. Otherwise, a faster incremental run can become a faster way to preserve incorrect KPIs.

For IFCO, Databricks reports that the team’s changes substantially reduced the rows scanned by its consolidation model and brought the share of assets recomputed per run down to approximately 3–5%. The nightly full refresh could then be retired. These are results from IFCO’s workload, not a guarantee that the same configuration will produce equivalent savings for every dbt project.

3. Measure the result, not the configuration change

Performance work is easy to overstate when teams report that a new setting was enabled but do not measure the resulting workload.

IFCO’s reported results provide a better frame of reference:

Metric	Before	After
Core semantic-layer job runtime	Approximately 7 hours	2 hours 20 minutes
Runtime improvement	—	Approximately 66%
Daily compute cost	Baseline	58% lower
Rows scanned by the consolidation model	Baseline	Approximately 75% fewer
Assets recomputed per run	Almost the full pool	Approximately 3–5%
Nightly full refresh	Required	Retired

These figures come from Databricks’ account of the engagement. The article does not isolate how much of the total saving came from each individual change, so it would be misleading to attribute the entire 58% cost reduction to liquid clustering, a particular write strategy, or job decomposition alone.

For your own workloads, collect before-and-after measurements for the same date range and comparable data volume. Include query duration, rows scanned, bytes read and written, spill, task duration, and compute cost where available. Otherwise, normal changes in input volume can be mistaken for an optimization gain.

4. Break the dbt graph into individual Databricks tasks

Incremental-write optimization addresses the cost of each model. IFCO also changed how the project is orchestrated.

A conventional Databricks dbt task can execute a whole project through commands such as dbt run and dbt test. This is a useful way to begin, but the job can behave like a black box: individual models have their own dependencies and runtime profiles, yet operators may need to inspect the overall run to understand what happened.

IFCO used the open-source databricks-dbt-factory project to generate a granular Databricks workflow from the dbt manifest. Each dbt object—models, tests, seeds, and snapshots—can become its own task, with dependencies derived from the project graph.

This creates a different operating model:

• Per-model visibility: Engineers can see which model is slow or failing without treating the whole project as a single execution unit.
• Targeted recovery: A failed task can be rerun with its required dependencies rather than blindly restarting every model.
• More precise testing: Tests can be associated with the resources they validate and integrated into the graph.
• Better cost attribution: Model-level runtimes can be connected with query tags and billing information to help identify regressions.
• Clearer ownership: Teams can assign owners and alerting expectations to the models that produce important business metrics.

The distinction is not that dbt suddenly understands dependencies differently. It is that the orchestration layer exposes those dependencies as separately observable and schedulable units.

How the factory builds the graph

The factory reads the project’s dbt manifest and a job template, then generates a Databricks workflow definition. Its documentation describes support for native Databricks dbt_task entries as well as a notebook-runner approach. The implementation used should be chosen deliberately because task types have different execution and configuration characteristics.

IFCO’s engineering write-up describes a notebook runner, a prebuilt serverless environment containing dbt-databricks, and manifest injection to reduce repeated setup and parsing overhead. The runner triggers dbt, while the generated SQL executes against the selected Databricks SQL warehouse.

This distinction matters: the job orchestration and the SQL transformation compute are not necessarily doing the same work. In the notebook-runner pattern, the notebook coordinates dbt, and the SQL warehouse executes the transformations.

A prebuilt environment avoids repeating package installation for each small task. Passing a prepared manifest to the runner can also reduce the parsing overhead that would otherwise be repeated across hundreds of tasks. The factory repository documents the use of a prebuilt partial-parse artifact for the notebook-runner path.

Without these optimizations, creating one task per model could introduce enough startup and parsing overhead to undermine the benefits of better visibility and parallelism.

5. Use AI-assisted diagnostics as an evidence-backed review process

The Databricks article also describes a repeatable diagnostic skill that examines executed query information to find common causes of inefficient models.

Its value is not that an AI system automatically rewrites production transformations. The documented approach produces findings supported by execution evidence—for example, excessive rows scanned, spill, or a problematic plan node—along with a proposed change. An engineer reviews the recommendation before it is applied, then checks whether the next run improves the same metrics.

That is a useful pattern for applying AI to platform operations:

1. Gather the executed plan and measured runtime signals.
2. Identify a known performance anti-pattern.
3. Produce a finding linked to the evidence.
4. Suggest a targeted remediation.
5. Require approval before changing production logic.
6. Re-measure after deployment to determine whether the change worked.

A diagnostic assistant is only as trustworthy as the evidence it can inspect and the checks applied to its recommendation. It should not infer that a query is correct simply because it runs faster. Tests and data-quality constraints remain part of the change.

For teams managing many similar models, repeating the same diagnostic checks can make performance work more systematic and help find recurring problems before they become large production incidents.

What data and AI engineers should do

Start with the most expensive model

Do not begin by decomposing every job or rewriting the entire project. Use query history and job metrics to find the model with the clearest performance problem. Inspect its actual plan, then make the smallest change likely to address the cause.

Define the model’s correctness boundary

Before bounding the incremental window or changing the write strategy, document the model’s grain and how late or corrected data is handled. Determine which rows can change, how far back those changes can arrive, and how older corrections are repaired.

Treat clustering and write semantics together

Choose clustering columns based on real access patterns and verify whether the executed plan benefits from pruning. Compare merge and replacement strategies only after confirming they preserve the same intended data semantics.

Automate the graph, but control generated changes

The dbt manifest is an input to job generation, so manifest and runtime behaviour must stay aligned. Pin the factory version, review regenerated workflow definitions, and test the generated task graph when models or dependencies change.

The factory documentation explicitly cautions that manifest schema v12 does not contain enough parse context for the factory to prove that dbt reconstructs exactly the same graph at runtime. Treat the manifest as a versioned build artifact, not an infallible contract.

Account for task overhead

A model-per-task graph creates more scheduling units and more opportunities for targeted retries, but it can also increase coordination overhead. Shared environments and partial parsing help, yet the right granularity depends on project size, model duration, concurrency limits, and dependency shape.

Benchmark the whole job, including orchestration overhead, rather than summing only SQL execution time.

Connect model ownership to cost monitoring

Use query tags and available system tables to build per-model runtime and cost views. Assign each model an owner and alert on material regressions. Track data volume with the costs, so the team can distinguish an inefficient query from a legitimate increase in input data.

Limitations and open questions

IFCO’s results show what is possible when incremental semantics and orchestration are tuned together. Several considerations still apply when adopting the approach elsewhere.

• Savings are workload-specific. Databricks reports a 58% reduction in daily compute costs, but the public write-up does not isolate the contribution of each optimization. Reproduce the measurement on your own data before estimating a business case.
• Incremental correctness is a business rule. Time-bounded recomputation can miss late corrections if the boundary is not designed for the source’s lateness characteristics.
• More tasks are not automatically faster. Large graphs can spend substantial time parsing, scheduling, initializing environments, and coordinating dependencies. Prebuilt environments and manifest reuse reduce some overhead but do not make it disappear.
• Manifest and runtime graphs can diverge. The factory’s documentation notes limitations in what manifest schema v12 records about parse context. Pin versions and validate graph regeneration as part of CI.
• Generated job definitions need reliable deployment. Treat the manifest, runner, and job specification as related release artifacts. Add validation so an interrupted deployment cannot leave an incomplete or inconsistent configuration unnoticed.
• AI diagnostics need review. A faster query plan is not sufficient evidence that a model’s semantics are preserved. Validate the output with tests and compare before-and-after results.

The broader question is how much of a large transformation platform should be generated from metadata. The factory approach can improve visibility and make recovery more targeted, but it also makes the build process, manifest, runner, and job specification part of the production system. Those components deserve tests and controlled releases of their own.

Key takeaways

1. Inspect what actually executes. The compiled dbt SQL may not explain all runtime write behaviour; use query history and the executed plan to locate expensive work.
2. Make incremental writes match the data-change pattern. Choose clustering, merge or replacement strategy, and reconciliation boundaries based on the rows that really need to change.
3. Measure the result. IFCO reported a reduction from seven hours to 2 hours 20 minutes, 58% lower daily compute costs, and roughly 75% fewer rows scanned in the consolidation model.
4. Expose the dbt graph operationally. Per-model Databricks tasks make failures, runtime regressions, tests, and targeted reruns easier to manage.
5. Reduce orchestration overhead deliberately. Prebuilt environments and manifest reuse can help keep a model-per-task strategy practical.
6. Keep AI diagnostics evidence-based. Use execution metrics to propose changes, require review, and validate the result after deployment.

IFCO’s experience is a reminder that large-scale data engineering is often a matter of precision rather than wholesale replacement. Understanding the real write behaviour of incremental models reduced wasted work, while per-model orchestration made the pipeline easier to observe and recover. That combination—not any single tuning knob—is what made the reported improvement possible.

Sources

1. Databricks Blog, Scaling and Operating a Large dbt Project on Databricks: IFCO’s Data Team on Performance, Visibility, and Debugging, October 6, 2026.
2. GitHub, mwojtyczka/databricks-dbt-factory.
3. Databricks Documentation, Use dbt transformations in Lakeflow Jobs.