Fivetran Managed Data Lake Write Credentials: Letting Engines Write to Iceberg and Delta Safely
Data Engineering

Fivetran Managed Data Lake Write Credentials: Letting Engines Write to Iceberg and Delta Safely

By issuing short-lived, directory-scoped credentials against a Polaris-backed REST catalog, MDLS lets engines alter Iceberg and Delta tables without sharing Fivetran's long-lived keys or breaking the write-once metadata model.

Nov 10, 2026 · 6 min read
Netflix Conductor 4.0: Scaling Workflows to 30,000 Tasks with Asynchronous Evaluation
Data Engineering

Netflix Conductor 4.0: Scaling Workflows to 30,000 Tasks with Asynchronous Evaluation

Netflix redesigned Conductor’s workflow engine by separating task records from workflow metadata, moving evaluation off the request path, and using Timestone for ordered asynchronous processing. The company reports support for workflows of up to 30,000 tasks and roughly 40% lower p99 evaluation latency.

Oct 28, 2026 · 16 min read
OpenAI Dots: A Persistent Cloud Agent for 4,000+ Apps
AI Engineering

OpenAI Dots: A Persistent Cloud Agent for 4,000+ Apps

Dots gives Pro and Business Premium users a dedicated cloud computer powered by GPT-6 Astra, but its high default cost makes the cheaper GPT-6.1 Sol a critical consideration for engineers.

Oct 26, 2026 · 4 min read
How IFCO Cut a 7-Hour dbt Job to 2 Hours 20 Minutes on Databricks
Data Engineering

How IFCO Cut a 7-Hour dbt Job to 2 Hours 20 Minutes on Databricks

IFCO reduced its core dbt job runtime by about 66% and daily compute costs by 58% by fixing incremental-write behaviour, inspecting actual execution plans, and splitting a large dbt graph into individual Databricks tasks—with model-level diagnostics and targeted recovery.

Oct 16, 2026 · 14 min read
ClickHouse Cloud Executable UDFs GA: Native Code, Network Access, and Query-Level Metrics
Data Engineering

ClickHouse Cloud Executable UDFs GA: Native Code, Network Access, and Query-Level Metrics

ClickHouse Cloud now supports compiled-language UDFs, outbound network access for eligible UDF runtimes, and per-query execution metrics. For LLM observability, that means token accounting and sampled quality evaluation can live closer to the data—with important resource, security, and runtime trade-offs.

Oct 14, 2026 · 14 min read
Meta Open-Sources Stateless NTS: Authenticated Time Without Server-Side Cookie State
Data Engineering

Meta Open-Sources Stateless NTS: Authenticated Time Without Server-Side Cookie State

Meta has open-sourced its Network Time Security implementation and launched nts.meta.com. Here is how its stateless cookie design derives sealing keys from a master secret and the epoch day, how to try it with chrony, and what NTS still cannot fix.

Oct 12, 2026 · 12 min read
n8n Bridges Workflow Automation and API Management, With Important Trade-offs
Data Engineering

n8n Bridges Workflow Automation and API Management, With Important Trade-offs

n8n combines self-hosted workflow orchestration, webhook-driven endpoints, custom code and AI integrations. It can reduce the need for separate integration infrastructure, but API security, traffic governance and production reliability still require deliberate design.

Oct 11, 2026 · 12 min read
Databricks SQL Serverless Cost Governance: Budgets, Alerts, and Query-Level Attribution
Data Engineering

Databricks SQL Serverless Cost Governance: Budgets, Alerts, and Query-Level Attribution

Databricks adds budgets, billing system tables and usage dashboards to make SQL Serverless spend easier to explain. But budgets are not spending caps, estimates are not invoices, and useful attribution depends on consistent tagging and operational controls.

Oct 10, 2026 · 13 min read
Databricks Runtime Introduces NEAREST BY: Native SQL Vector Search at Scale
Data Engineering

Databricks Runtime Introduces NEAREST BY: Native SQL Vector Search at Scale

Databricks brings batch vector search into SQL with NEAREST BY, a fused Photon GEMM operator and IVF indexes stored in liquid-clustered Delta tables, reducing the need for separate vector-serving infrastructure in large-scale analytical workloads.

Oct 09, 2026 · 12 min read