Fivetran Managed Data Lake Write Credentials: Letting Engines Write to Iceberg and Delta Safely

Fivetran says Managed Data Lake Write Credentials are now generally available, letting query engines run DDL and DML against lake tables ^[1]. Engines authenticate via OAuth2 client credentials—one Client ID and Secret per destination—so each gets a distinct identity against the hosted catalog ^[2]. For teams running a dual-format Iceberg and Delta lake, the real question is how to scope write access tightly enough that an engine can delete rows or drop a table without broad bucket permissions, while Fivetran retains sole authority over metadata and schema evolution. This article explains how Fivetran strikes that balance, and what to set up before you rely on it.

Why it matters

For data engineers, the practical consequence is that permission management collapses into a single policy layer. Fivetran’s catalog returns per-user access rules through the Iceberg REST protocol, so an organization defines governance once rather than re-encoding it for every engine ^[3]. In Fivetran’s post, Snowflake’s Spitzer describes this as one policy definition and one catalog access point feeding all connected engines simultaneously ^[3:1]. Engineers no longer maintain separate ACLs per query tool.

For AI and data engineers who provision write paths, the writable catalog credentials are visible only to Destination or Account Admins in the Fivetran dashboard ^[1:1]. That means a data scientist spinning up an ad-hoc Spark session receives read-only access by default; only an admin can mint the short-lived, directory-scoped S3 credentials that unlock DDL and DML against a specific table’s storage ^[2:1]. The engine never holds standing broad bucket permissions, and because Fivetran retains sole authority over metadata publication, a stray schema edit in one engine cannot desynchronize what another engine reads.

What happened

Fivetran announced its Managed Data Lake service, which pulls data through more than 750 connectors and lands it in commodity object-storage buckets such as S3, ADLS, and GCS ^[1:2]. The pipeline normalizes, compacts, and deduplicates records before writing them in both Apache Iceberg and Delta Lake open table formats, granting warehouse-grade ACID transactions on top of a lake. Alongside the core ingestion path, Fivetran introduced a write-credentials mechanism: short-lived tokens that let external compute engines execute DDL and DML directly against the lake’s data files while Fivetran retains sole authority over metadata updates. According to Fivetran, this decoupling of storage from compute lets customers query and modify data through engines such as Databricks, Snowflake, or Starburst without Fivetran mediating every write.

The managed data lake write credentials feature supports destinations on S3, Azure Data Lake Storage, OneLake, and Google Cloud Storage ^[4]. It includes native catalog integrations for AWS Glue, Databricks Unity Catalog, and Apache Polaris ^[4:1]. Fivetran manages schema evolution and performance optimization on the backend ^[4:2], allowing engine-side writers to use vended credentials for standard SQL or Python operations ^[2:2].

How it works

Fivetran provisions a dedicated REST catalog per destination, built on Apache Polaris ^[2:3]. Each engine receives short-lived credentials scoped to a specific directory within the lake ^[2:4], so it can only touch data files inside that boundary. The writable catalog the engine receives points to the same underlying files, metadata, and Polaris namespace as Fivetran’s primary catalog; any change the engine commits is immediately visible in the original catalog ^[1:3]. Fivetran restricts metadata writes to itself, while engines access the layer in a read-only manner ^[2:5].

Users can query and modify data using compatible compute engines such as Databricks, Snowflake, Starburst, or Redshift ^[4:3]. The platform also manages schema evolution and performance optimization on the lake ^[4:4].

The trade-off is lifecycle rigidity. Fivetran retains the current metadata snapshot plus three previous versions and auto-deletes older ones ^[2:6]. Orphan file cleanup runs every other Saturday, sweeping up Parquet leftovers from unsuccessful pipeline operations that no table metadata references ^[2:7]. Fivetran also cautions that the service is a poor fit for organisations relying on sub-second streaming or preferring a non-open table format, since the dual-format write-once design assumes batch-oriented, open-format workloads ^[4:5].

What data and AI engineers should do

If your governance model includes Athena or Redshift alongside REST-protocol engines, Fivetran says the AWS Glue catalog toggle must be enabled during initial destination setup and cannot be reversed once saved, making it a one-way configuration decision ^[2:8]. Map every engine you intend to deploy against the lake before your first sync so that toggle is set correctly from day one, and assign distinct schema names per data lake to prevent the naming collisions that Glue enforces at the Region level. For environments where network isolation is a compliance requirement, Fivetran notes that PrivateLink routing through your VPC without public-internet exposure is available only on the Business Critical plan ^[2:9]. Factor that tier upgrade into your budget before committing to a regulated-industry deployment rather than discovering the plan restriction after architecture is locked.

Before wiring production pipelines, Fivetran offers a fourteen-day free trial on the Managed Data Lake ^[4:6]. Use that window to validate two things: first, confirm your S3 lifecycle rules will not expire object files before the snapshot retention period lapses, which would leave snapshots pointing at missing data; second, verify that your least-privilege IAM and KMS policies allow every engine to read the dual-format lake while Fivetran remains the sole writer of metadata. Treat the trial as a dry run of your full governance checklist, not a smoke test.

Limitations and open questions

Operators must pause the relevant connector before executing any DDL or DML commands, as unpaused alterations risk permanent data loss ^[1:4]. The feature primarily targets Apache Iceberg tables, while teams amending Delta Lake tables must contact support first ^[1:5]. For security reasons, the client secret is displayed only at generation and cannot be retrieved later ^[1:6].

Fivetran’s posts do not define atomicity guarantees when a connector is paused mid-batch, nor do they explain how Fivetran’s sole authority over metadata reconciles with schema changes made from an engine.

Key takeaways

• Fivetran retains sole authority over metadata, preventing stray schema edits in one engine from desynchronizing others. ^[2:10]
• Operators must pause connectors before issuing DDL or DML to avoid permanent data loss during active syncs. ^[1:7]
• Only Destination or Account Admins can mint write credentials, and each one is scoped to a single directory, so engines never hold broad bucket permissions. ^{[1:8] [2:11]}
• The AWS Glue catalog toggle cannot be reversed after setup: decide which engines you need before the first sync. ^[2:12]
• Fivetran keeps the current metadata snapshot plus three previous versions, so align S3 lifecycle rules with that retention. ^[2:13]

Sources