We Ran the Same Iceberg MERGE on EMR and on DuckDB

This is a benchmark report, not an argument. I took one workload, ran it on two engines under conditions made as identical as I could make them, verified afterwards that both engines had actually done the same work, and wrote down what came out. The conclusion is yours to draw.

The full write-up is also available on Medium: We ran the same Iceberg MERGE on EMR and on DuckDB.

TL;DR

• Spark on a 32-vCPU EMR cluster: 30.777 s median. DuckDB on a 32-vCPU single box: 37.511 s median. Spark faster by ~18%, with non-overlapping ranges.
• The cluster costs 34% more per hour but is only 18% faster, so the faster engine is ~10% more expensive per merge - 3.06 cents vs 2.78 cents at us-east-1 on-demand rates.
• Both merges cost about three cents, which is the more useful observation: at this scale the merge window is not where the money goes. An idle hour on that cluster costs 117 merges.
• The comparison was verified rather than assumed. Both engines started from the same Iceberg snapshot and produced identical signatures: 27 delete files, 1,000,000 positional deletes, 41,000,000 records.
• The trap worth stealing: Iceberg defaults write.merge.mode to copy-on-write, and DuckDB's DDL never sets it. My first attempt merged copy-on-write under Spark and took 209.6 s instead of 30.8 s. The table name described the intent, not the configuration.

The pipeline

The shape being tested is deliberately ordinary, because it is the one most data teams are already running:

• Batches of JSON land on S3. Some upstream system drops files on a prefix, daily or hourly.
• The target is an Apache Iceberg table, also on S3. Regular S3, Glue catalog, format-version 2, unpartitioned.
• The job is an upsert. New records insert, existing records update in place. In Iceberg terms, a single MERGE INTO.

MERGE INTO <target> t
USING merge_source s ON t.pid = s.pid
WHEN MATCHED THEN UPDATE SET *
WHEN NOT MATCHED THEN INSERT *

There are two obvious ways to run that job today, and they represent genuinely different bets.

Amazon EMR with Spark is the default answer: a managed cluster, a distributed execution engine, mature Iceberg support, and the ability to scale past one machine when the table outgrows one.

DuckDB on a single EC2 instance is the newer option. As of 1.5.5, DuckDB writes Iceberg tables through an attached REST catalog, which makes it a candidate for this job rather than only a reader of it. One process, one machine, no cluster.

The question this benchmark answers is narrow and specific: for one all-updates merge of a 1M-row batch into a 40M-row table, how far apart are they in time, and how far apart are they in cost?

Making it a fair fight

Benchmarks comparing a cluster to a laptop are easy to write and worth nothing. The design constraint here was comparability, so the two arms differ in as close to one variable as I could manage.

Same compute

	EMR / Spark	DuckDB
Instances	1× m5.xlarge primary + 4× r5dn.2xlarge core	1× r5dn.8xlarge
Worker vCPU	32	32
Worker memory	256 GiB	256 GiB
Network	4× "up to 25 Gbps"	1× 25 Gbps sustained

Four r5dn.2xlarge nodes are 8 vCPU and 64 GiB each; one r5dn.8xlarge is 32 vCPU and 256 GiB. The worker fleets are the same size to the core and to the gigabyte. This is core-for-core, not a hardware-handicap story.

Two asymmetries are worth naming rather than hiding. EMR additionally gets an m5.xlarge primary node for coordination, which DuckDB has no equivalent of. And the cluster's four network interfaces each burst to 25 Gbps independently, against the single instance's one sustained 25 Gbps - an advantage to the cluster on I/O fan-out, and one that a single box structurally cannot match.

Same table

DuckDB built the 40M-row target table itself: regular S3, Iceberg v2, unpartitioned, zstd level 3, 256 MB target file size. Then both engines merged into that same table - not into two tables built to the same recipe, but into the identical set of files in the identical bucket. Any difference in file sizes or row distribution is therefore common to both arms and cannot leak into the result as an engine effect.

The data is the Spotify Million Playlist Dataset. The 40M-row target is the raw playlists cloned with pid offsets; the incoming batch is 1M playlists whose pids each match exactly one target row. That makes this an all-updates merge - the common batch-ingestion pattern, and the one that puts the most pressure on the write path.

Same protocol

• 1 warmup merge, then 3 timed merges, per engine.
• The table is rolled back to the same seed snapshot between every run via Iceberg's rollback_to_snapshot, so run 3 starts from exactly the table run 1 started from.
• The merge source is materialized before any timer starts. The JSON parse happens once, up front; no timed merge pays for reading its own input.

One constraint that shaped the whole design

DuckDB's Iceberg writer is merge-on-read only. It writes positional delete files; it has no copy-on-write mode. So the only comparable arm is merge-on-read, and Spark was configured to match with 'write.merge.mode'='merge-on-read'.

Everything below is MOR. Nothing here says anything about copy-on-write, which is a substantially heavier operation on the same data.

Results

Wall-clock duration of the MERGE, with Spark's figures taken from SparkListenerSQLExecutionStart/End in the event log:

Engine	Hardware	Timed runs (s)	Min	Median	Max
Spark 3.5 / EMR 7.12	1× m5.xlarge + 4× r5dn.2xlarge	33.773 / 30.777 / 30.612	30.612	30.777s	33.773
DuckDB 1.5.5	1× r5dn.8xlarge	39.009 / 37.511 / 37.315	37.315	37.511s	39.009

Spark is faster by about 18% - 30.777 s against 37.511 s, a gap of 6.7 seconds on a half-minute job.

The ranges do not overlap: Spark's slowest run (33.773 s) is still faster than DuckDB's fastest (37.315 s). Whatever else is uncertain here, the direction is not.

Both engines show the same run-order pattern - the first timed run is the slowest and the next two are close together - which is the shape you expect from page cache warming rather than from any property of either engine. Spark's separate warmup merge took 43.5 s.

For scale, the target-table scan inside Spark's merge reads 45.5 MB and takes about 2 seconds, because a MERGE needs only the join key and each row's coordinates - every wide column is pruned at the Parquet level and never leaves S3. The remaining ~28 seconds are shuffle and delete-file writes. This is not an I/O-bound job on either engine.

An aside on table size

DuckDB ran this same merge against a 20M-row table earlier in the same session, on the same box and the same 1M-row batch. Its timed runs there were 39.264 / 39.817 / 39.009 - median 39.264 s, marginally slower than the 37.511 s it posted against the table twice that size.

The two tables were built separately, so treat that as directional rather than exact. But it points at the mechanism: in merge-on-read the cost tracks the size of the incoming batch, not the size of the target, because the target is never rewritten. Doubling the table did not cost DuckDB anything measurable.

Did both engines actually do the same work?

This is the question that decides whether the 18% means anything, and it is not answerable by looking at table names. I checked it three ways.

Snapshot signatures match. After the merge, both engines produced snapshots with 27 delete files, exactly 1,000,000 positional deletes, and 41,000,000 total records. That triple is the merge-on-read signature of one million rows updated in place, with no rows lost and none duplicated. Two engines arriving at the same three numbers on the same input were performing the same logical operation.

Spark's physical plan confirms the write mode. The plan contains WriteDelta and zero ReplaceData nodes. In Iceberg's Spark implementation those two nodes are the tell: WriteDelta is merge-on-read, ReplaceData is copy-on-write. Seeing WriteDelta means Spark wrote delete files rather than rewriting data files, matching what DuckDB has no choice but to do.

Both arms started from the same snapshot, and I can name it. The table's recorded build state was:

{ "table": "glue_cat.bench.t_duckdb_mor_40m",
  "row_count": 40000000, "distinct_pids": 40000000,
  "file_count": 299, "total_bytes": 102387554583,
  "delete_files": 0, "snapshot_id": 7922868912395733029 }

Both benchmark runs rolled back to snapshot 7922868912395733029 before every timed merge - the same ID appears in each engine's run log. Every merge on both sides therefore began from the identical 299 files, 40,000,000 rows, 102,387,554,583 bytes, and zero delete files.

Each run also asserted 40,000,000 rows before the merge and 40,000,000 rows after it, which is the correct result for an all-updates batch: one million rows updated, none inserted, none lost. (That is the live row count; the snapshot's 41,000,000 records counts the newly written rows before the positional deletes are applied. Both numbers describe the same table.)

Why this check exists: the write.merge.mode trap

My first attempt at this comparison measured the wrong thing, and the failure generalizes well enough to be worth reporting.

Iceberg defaults write.merge.mode to copy-on-write. DuckDB's table DDL never sets it. So a table created by DuckDB and named t_duckdb_mor_40m was, as far as Spark was concerned, a copy-on-write table - the name described the intent, not the configuration. Spark dutifully rewrote data files while DuckDB wrote delete files, and the two arms were not measuring the same operation at all.

The magnitude of that mistake: the copy-on-write warmup merge on the 40M table took 209.6 s, against 30.777 s for merge-on-read on the same table with the same batch - roughly 6.8×. A benchmark that missed this would have reported DuckDB beating Spark by a factor of five, and would in fact have been reporting the default value of a table property.

There are two ways to catch it. Before the run, read the plan: ReplaceData versus WriteDelta. After the run, read the snapshot: total-delete-files of zero means files were rewritten, whatever the table is called.

What the 18% costs

Time is not the number that decides this for most teams; the invoice is. Here is the arithmetic, with the inputs shown so you can substitute your own.

Rates are us-east-1, on-demand, Linux, shared tenancy, pulled from the AWS Pricing API on 2026-07-27:

Component	EC2	EMR uplift	Total
m5.xlarge ×1	$0.1920/hr	$0.0480/hr	$0.2400/hr
r5dn.2xlarge ×4	$2.6720/hr	$0.6680/hr	$3.3400/hr
EMR cluster			$3.5800/hr
r5dn.8xlarge ×1	$2.6720/hr	-	$2.6720/hr

The EMR cluster costs 34% more per hour than the single box: the EMR service uplift is 25% on top of every instance, and the cluster carries a primary node the single box does not need.

Counting only the merge window:

Engine	Rate	Merge time	Cost per merge
EMR / Spark	$3.5800/hr	30.777 s	$0.0306
DuckDB	$2.6720/hr	37.511 s	$0.0278

So the faster engine is also the more expensive one, by about 10% per merge: EMR's 18% time advantage does not cover its 34% rate premium.

Both figures are around three cents, which is the more important observation - at this scale the merge window is not where the money is. A 30-second job on a $3.58/hr cluster costs three cents; the same cluster left running for an hour costs $3.58, or roughly 117 merges' worth. Whichever engine you choose, the bill is dominated by how long the machine is up and not by how long the query runs. Cluster provisioning time, idle time between batches, and teardown discipline all matter more than the 6.7-second gap measured above - and none of them were measured here.

Caveats

The boundaries of this result, stated plainly:

• One workload shape. All-updates, 1M rows into 40M, unpartitioned table, ~256 MB files, zstd level 3. A sparser merge, a partitioned table, or a different file-size distribution would move the number in ways this run cannot predict.
• Merge-on-read only. DuckDB has no copy-on-write mode, so that arm does not exist. Since COW on the same table and batch took 209.6 s against 30.777 s, this benchmark covers the cheaper of the two write modes and says nothing about the expensive one.
• Three timed runs, one day, one cluster. The ranges do not overlap, which supports the direction, but three runs per arm cannot pin the gap to a precise 18%.
• One writer, no concurrency. Each engine had the table to itself. Nothing here speaks to concurrent writers, commit contention, or retry behaviour under conflict - an area where a distributed engine and a single process may differ considerably.
• 32 cores is the only scale tested. DuckDB has no shuffle and one machine's network; Spark has both, plus four independent NICs. A merge that fanned out across many more files, or a cluster scaled past 32 cores, changes the terms of the comparison. A single box also has a ceiling that a cluster does not.
• Provisioning and teardown were not measured. The cost table covers the merge window only. For a job this short, the time to bring each environment up and take it down plausibly dominates the total bill.
• Prices are a snapshot. On-demand, us-east-1, retrieved 2026-07-27. Spot, Savings Plans, or a different region change the cost conclusion - and the cost conclusion is the one closest to flipping, since 18% and 34% are not far apart.
• DuckDB's Iceberg write path is young. 1.5.5 writes Iceberg only through an attached REST catalog; I used Glue. That path works, but it is newer than Spark's and carries correspondingly less production mileage.

What I measured, in one paragraph

On a 40M-row unpartitioned Iceberg table on regular S3, merging a 1M-row all-updates batch in merge-on-read mode, Spark on a 32-vCPU EMR cluster took a median 30.777 s and DuckDB 1.5.5 on a 32-vCPU single instance took 37.511 s - Spark faster by 18%, with non-overlapping ranges. Both engines started from snapshot 7922868912395733029 and produced identical snapshot signatures: 27 delete files, 1,000,000 positional deletes, 41,000,000 records. At us-east-1 on-demand rates the EMR configuration costs 34% more per hour, making the individual merge about 10% more expensive on the faster engine. Both merges cost approximately three cents.