Loading MongoDB into DuckDB from Java: Why a Table Function Beat JDBC and the Appender

On a 485,168-record sample, a pure-Java table function loaded data in 1m 27s, against 2m 02s for the Appender and 3m 37s for JDBC batches. On production data it matched the Mongo community extension.

Loading MongoDB into DuckDB from Java: Why a Table Function Beat JDBC and the Appender

A team storing vulnerability data in MongoDB moved a year of records into DuckDB, on the same server, to make its analytics screen and Excel exports fast. Getting the data in turned out to be the hard part. A guest post on the DuckDB blog walks through every Java loading method they tried, with timings on a public sample dataset, and explains why the one they shipped is a table function written in pure Java [1]. If you load data into DuckDB from a JVM application, it is a useful map of what each API really costs.

Why it matters

The obvious answers were not the right ones. JDBC batch inserts are the standard Java way to bulk load, and the Appender is the API DuckDB’s documentation recommends for bulk inserts [2]. Both were too slow here. The fastest route, a community extension, was ruled out for maintenance reasons. The table function approach gives Java teams a way to stream data from any source straight into DuckDB’s query engine, without staging files and without maintaining C++ code.

The setup

The targets were demanding: every analytical query in one second or less, and a 50,000-row Excel export in under ten seconds. DuckDB had to run on the same server as MongoDB and work well with limited resources [1:1].

The published sample has 485,168 records with 24 fields. Two text columns are large, with median lengths of 5,000 and 2,500 characters and some values up to 32,000 characters, which makes every copy and sort expensive [1:2]. The Java examples split the import into day-sized batches fetched from MongoDB in parallel, then sorted the result into the final table, because sorting drastically reduced the database file size [1:3].

What happened

Method Sample import time
Mongo community extension 55 s
JDBC executeBatch, 8 workers 3 min 37 s
Java Appender, parallel workers 2 min 02 s
Java table function, 8 threads 1 min 27 s

All three Java times include the final sorted CREATE TABLE ... AS ... ORDER BY, which took about 30 seconds in the JDBC run [1:4].

The extension was fastest, and rejected. The mongo community extension imported the sample in 55 seconds, and the team called it the easiest and most performant option. They passed because community extensions carry no future-proof guarantee: if the extension is not updated, it can block DuckDB upgrades. Building it themselves would have meant maintaining a C++ fork [1:5].

JDBC batches pay per row. executeBatch is implemented in Java: the statement is prepared once, but each row still runs as a separate database-level INSERT. That made it 3.5x slower than the extension [1:6].

The Appender batches, but not enough. The Appender does real batching, building a 2,048-row data chunk in Java and flushing it in one call. It was still 2x slower than the extension, or 1.5x without the final sort, which was not acceptable on production data [1:7].

The table function won among the Java options. After reading the extension’s source, the author found it used table functions, and rebuilt the same idea in Java [1:8].

How it works

A Java table function is registered with DuckDB and then queried like a table: CREATE TABLE staging.vuln1 AS FROM vuln_import(from_date, to_date). DuckDB calls the function’s apply method repeatedly, and each call writes up to 2,048 rows directly into DuckDB’s column vectors. The import ends when the function returns zero rows [1:9].

Two settings make it run in parallel [1:10]:

  • info.setMaxThreads(8) tells DuckDB the function supports concurrent apply calls.
  • SET preserve_insertion_order = FALSE lets DuckDB use those threads. Without it, DuckDB keeps calls on one thread to preserve order.

On production data, disk space was tight, so the team used a non-parallel version and put the ORDER BY in the same CREATE TABLE query, avoiding an intermediate copy. That matched the extension’s import times [1:11]. The result is an ingestion pipeline in pure Java that they can adapt to other sources.

What data and AI engineers should do

  • Skip executeBatch for large loads. It runs one INSERT per row under the hood.
  • Use a table function when the source is a Java API. It feeds DuckDB’s vectors directly and lets DuckDB parallelize the work. The sample code is in the staticlibs/duckdb_java_data_import repository [1:12].
  • Decide where sorting happens. A separate sort step is simple and parallel but needs disk space for a second copy; sorting inside the CREATE TABLE saves space at the cost of parallelism.
  • Weigh extensions by maintenance, not only speed. A community extension can be the fastest option and still be the wrong dependency.

Limitations and open questions

This is one team’s experience on one server. The comparison is not apples to apples: the extension imported a CSV file in one statement, while the Java methods used day-sized queries against MongoDB plus a sort step, and the author notes that day-sized batches can be uneven [1:13]. A file-based route (export to NDJSON, CSV or Parquet, then load) was abandoned for lack of disk space, not because it was slow, so it may still suit teams with more storage [1:14].

Key takeaways

  • On the sample, a Java table function loaded data in 1 min 27 s, against 2 min 02 s for the Appender and 3 min 37 s for JDBC batches. [1:15]
  • JDBC executeBatch still runs one INSERT per row, which makes it the slowest option. [1:16]
  • Parallel table functions need setMaxThreads and preserve_insertion_order = FALSE. [1:17]
  • The Mongo community extension was fastest (55 s) but was rejected because it could block DuckDB upgrades. [1:18]

Sources


  1. DuckDB Blog, “Importing Data Using Java Table Functions”, 2026-10-05. https://duckdb.org/2026/10/05/import-data-with-java.html ↩︎ ↩︎ ↩︎ ↩︎ ↩︎ ↩︎ ↩︎ ↩︎ ↩︎ ↩︎ ↩︎ ↩︎ ↩︎ ↩︎ ↩︎ ↩︎ ↩︎ ↩︎ ↩︎

  2. DuckDB Documentation, “Java client: Data import”. https://duckdb.org/docs/current/clients/java/data_import ↩︎

Discussion

Join the discussion: create a free account or sign in to comment.

No comments yet.

Keep reading

Jev in SQL: Classify Text Columns in DuckDB and MotherDuck With Typed Answers
Data Engineering

Jev in SQL: Classify Text Columns in DuckDB and MotherDuck With Typed Answers

TypeSafe's Jev scores a fixed list of answers instead of writing text. MotherDuck classified 100,000 complaints in 82 seconds for $0.50, and DuckDB extensions turn plain-English conditions into WHERE clauses.

Nov 17, 2026 · 5 min read
ClickHouse on NVIDIA Vera: How a 352-Thread CPU Scales Filters, Aggregations and Joins
Data Engineering

ClickHouse on NVIDIA Vera: How a 352-Thread CPU Scales Filters, Aggregations and Joins

An unmodified ClickHouse binary scales almost linearly to Vera's 176 physical cores on CPU-heavy queries. Past that, SMT adds little, and a simple LIKE filter runs out of memory bandwidth at 64 threads.

Nov 11, 2026 · 5 min read
ClickHouse Cloud Executable UDFs GA: Native Code, Network Access, and Query-Level Metrics
Data Engineering

ClickHouse Cloud Executable UDFs GA: Native Code, Network Access, and Query-Level Metrics

ClickHouse Cloud now supports compiled-language UDFs, outbound network access for eligible UDF runtimes, and per-query execution metrics. For LLM observability, that means token accounting and sampled quality evaluation can live closer to the data—with important resource, security, and runtime trade-offs.

Oct 14, 2026 · 14 min read