The Thin Orchestrator: How Meta Made Its Data Pipeline Agents Reliable by Giving the Router Less to Do
Data Engineering

The Thin Orchestrator: How Meta Made Its Data Pipeline Agents Reliable by Giving the Router Less to Do

Meta replaced one all-knowing agent with a 1,300-line deterministic router and 14 specialist subagents. Column config prep fell from three to four days to under an hour, and about two-thirds of changes land without significant human edits.

Nov 20, 2026 · 5 min read
Jev in SQL: Classify Text Columns in DuckDB and MotherDuck With Typed Answers
Data Engineering

Jev in SQL: Classify Text Columns in DuckDB and MotherDuck With Typed Answers

TypeSafe's Jev scores a fixed list of answers instead of writing text. MotherDuck classified 100,000 complaints in 82 seconds for $0.50, and DuckDB extensions turn plain-English conditions into WHERE clauses.

Nov 17, 2026 · 5 min read
Real-Time Energy Trading on Databricks: One Governed Data Copy for the Desk, Risk and ML
Data Engineering

Real-Time Energy Trading on Databricks: One Governed Data Copy for the Desk, Risk and ML

Databricks streams market data with Real-Time Mode into Lakebase Postgres, serves the trading desk from it, and trains models and runs analytics on the same governed copy. Here is how the pieces fit, and how to wire an app to Lakebase.

Nov 13, 2026 · 5 min read
Loading MongoDB into DuckDB from Java: Why a Table Function Beat JDBC and the Appender
Data Engineering

Loading MongoDB into DuckDB from Java: Why a Table Function Beat JDBC and the Appender

On a 485,168-record sample, a pure-Java table function loaded data in 1m 27s, against 2m 02s for the Appender and 3m 37s for JDBC batches. On production data it matched the Mongo community extension.

Nov 12, 2026 · 5 min read
ClickHouse on NVIDIA Vera: How a 352-Thread CPU Scales Filters, Aggregations and Joins
Data Engineering

ClickHouse on NVIDIA Vera: How a 352-Thread CPU Scales Filters, Aggregations and Joins

An unmodified ClickHouse binary scales almost linearly to Vera's 176 physical cores on CPU-heavy queries. Past that, SMT adds little, and a simple LIKE filter runs out of memory bandwidth at 64 threads.

Nov 11, 2026 · 5 min read
Fivetran Managed Data Lake Write Credentials: Letting Engines Write to Iceberg and Delta Safely
Data Engineering

Fivetran Managed Data Lake Write Credentials: Letting Engines Write to Iceberg and Delta Safely

By issuing short-lived, directory-scoped credentials against a Polaris-backed REST catalog, MDLS lets engines alter Iceberg and Delta tables without sharing Fivetran's long-lived keys or breaking the write-once metadata model.

Nov 10, 2026 · 6 min read
Netflix Conductor 4.0: Scaling Workflows to 30,000 Tasks with Asynchronous Evaluation
Data Engineering

Netflix Conductor 4.0: Scaling Workflows to 30,000 Tasks with Asynchronous Evaluation

Netflix redesigned Conductor’s workflow engine by separating task records from workflow metadata, moving evaluation off the request path, and using Timestone for ordered asynchronous processing. The company reports support for workflows of up to 30,000 tasks and roughly 40% lower p99 evaluation latency.

Oct 28, 2026 · 16 min read
How IFCO Cut a 7-Hour dbt Job to 2 Hours 20 Minutes on Databricks
Data Engineering

How IFCO Cut a 7-Hour dbt Job to 2 Hours 20 Minutes on Databricks

IFCO reduced its core dbt job runtime by about 66% and daily compute costs by 58% by fixing incremental-write behaviour, inspecting actual execution plans, and splitting a large dbt graph into individual Databricks tasks—with model-level diagnostics and targeted recovery.

Oct 16, 2026 · 14 min read
ClickHouse Cloud Executable UDFs GA: Native Code, Network Access, and Query-Level Metrics
Data Engineering

ClickHouse Cloud Executable UDFs GA: Native Code, Network Access, and Query-Level Metrics

ClickHouse Cloud now supports compiled-language UDFs, outbound network access for eligible UDF runtimes, and per-query execution metrics. For LLM observability, that means token accounting and sampled quality evaluation can live closer to the data—with important resource, security, and runtime trade-offs.

Oct 14, 2026 · 14 min read