Real-Time Energy Trading on Databricks: One Governed Data Copy for the Desk, Risk and ML
Data Engineering

Real-Time Energy Trading on Databricks: One Governed Data Copy for the Desk, Risk and ML

Databricks streams market data with Real-Time Mode into Lakebase Postgres, serves the trading desk from it, and trains models and runs analytics on the same governed copy. Here is how the pieces fit, and how to wire an app to Lakebase.

Nov 13, 2026 · 5 min read
ClickHouse on NVIDIA Vera: How a 352-Thread CPU Scales Filters, Aggregations and Joins
Data Engineering

ClickHouse on NVIDIA Vera: How a 352-Thread CPU Scales Filters, Aggregations and Joins

An unmodified ClickHouse binary scales almost linearly to Vera's 176 physical cores on CPU-heavy queries. Past that, SMT adds little, and a simple LIKE filter runs out of memory bandwidth at 64 threads.

Nov 11, 2026 · 5 min read
Fivetran Managed Data Lake Write Credentials: Letting Engines Write to Iceberg and Delta Safely
Data Engineering

Fivetran Managed Data Lake Write Credentials: Letting Engines Write to Iceberg and Delta Safely

By issuing short-lived, directory-scoped credentials against a Polaris-backed REST catalog, MDLS lets engines alter Iceberg and Delta tables without sharing Fivetran's long-lived keys or breaking the write-once metadata model.

Nov 10, 2026 · 6 min read
Netflix Conductor 4.0: Scaling Workflows to 30,000 Tasks with Asynchronous Evaluation
Data Engineering

Netflix Conductor 4.0: Scaling Workflows to 30,000 Tasks with Asynchronous Evaluation

Netflix redesigned Conductor’s workflow engine by separating task records from workflow metadata, moving evaluation off the request path, and using Timestone for ordered asynchronous processing. The company reports support for workflows of up to 30,000 tasks and roughly 40% lower p99 evaluation latency.

Oct 28, 2026 · 16 min read
How IFCO Cut a 7-Hour dbt Job to 2 Hours 20 Minutes on Databricks
Data Engineering

How IFCO Cut a 7-Hour dbt Job to 2 Hours 20 Minutes on Databricks

IFCO reduced its core dbt job runtime by about 66% and daily compute costs by 58% by fixing incremental-write behaviour, inspecting actual execution plans, and splitting a large dbt graph into individual Databricks tasks—with model-level diagnostics and targeted recovery.

Oct 16, 2026 · 14 min read
ClickHouse Cloud Executable UDFs GA: Native Code, Network Access, and Query-Level Metrics
Data Engineering

ClickHouse Cloud Executable UDFs GA: Native Code, Network Access, and Query-Level Metrics

ClickHouse Cloud now supports compiled-language UDFs, outbound network access for eligible UDF runtimes, and per-query execution metrics. For LLM observability, that means token accounting and sampled quality evaluation can live closer to the data—with important resource, security, and runtime trade-offs.

Oct 14, 2026 · 14 min read
n8n Bridges Workflow Automation and API Management, With Important Trade-offs
Data Engineering

n8n Bridges Workflow Automation and API Management, With Important Trade-offs

n8n combines self-hosted workflow orchestration, webhook-driven endpoints, custom code and AI integrations. It can reduce the need for separate integration infrastructure, but API security, traffic governance and production reliability still require deliberate design.

Oct 11, 2026 · 12 min read
Databricks SQL Serverless Cost Governance: Budgets, Alerts, and Query-Level Attribution
Data Engineering

Databricks SQL Serverless Cost Governance: Budgets, Alerts, and Query-Level Attribution

Databricks adds budgets, billing system tables and usage dashboards to make SQL Serverless spend easier to explain. But budgets are not spending caps, estimates are not invoices, and useful attribution depends on consistent tagging and operational controls.

Oct 10, 2026 · 14 min read
Databricks Runtime Introduces NEAREST BY: Native SQL Vector Search at Scale
Data Engineering

Databricks Runtime Introduces NEAREST BY: Native SQL Vector Search at Scale

Databricks brings batch vector search into SQL with NEAREST BY, a fused Photon GEMM operator and IVF indexes stored in liquid-clustered Delta tables, reducing the need for separate vector-serving infrastructure in large-scale analytical workloads.

Oct 09, 2026 · 13 min read