HIVE-29816: Iceberg: MERGE INTO with WHEN MATCHED THEN DELETE incorrectly duplicates rows under Copy-On-Write delete mode by Aggarwal-Raghav · Pull Request #6701 · apache/hive

Aggarwal-Raghav · 2026-08-14T18:59:43Z

What changes were proposed in this pull request?

When write.delete.mode=copy-on-write and write.merge.mode=merge-on-read (default), running a MERGE INTO ... WHEN MATCHED THEN DELETE query incorrectly causes the writer to use COW instead of MOR causing duplicate rows instead of deleting the rows.

Added an isMergeStatement boolean flag to FileSinkDesc and propagated it to the Hadoop JobConf during FileSinkOperator initialization via HiveCustomStorageHandlerUtils.setIsMergeStatement

Why are the changes needed?

Check HIVE-29816

Does this PR introduce any user-facing change?

Yes, executing MERGE INTO... DELETE query on tables utilizing Copy-On-Write delete modes will no longer experience data duplication and sequence number increments.

How was this patch tested?

Using, q files and on spark

Aggarwal-Raghav · 2026-08-14T19:04:27Z

BEFORE FIX HIVE:

AFTER FIX HIVE:

SPARK:

Aggarwal-Raghav · 2026-08-17T17:55:45Z

@ayushtkn , can you help with review?

Aggarwal-Raghav · 2026-08-24T18:57:43Z

@deniskuzZ , can you please help with review?

deniskuzZ · 2026-08-26T11:12:44Z

@Aggarwal-Raghav, sure, please give me some time

Aggarwal-Raghav · 2026-08-26T11:21:17Z

@Aggarwal-Raghav, sure, please give me some time

understood

kasakrisz · 2026-08-31T12:42:03Z

      setWriteOperation(jc, getConf().getTableInfo().getTableName(), getConf().getWriteOperation());
+      setStatementOperation(jc, getConf().getTableInfo().getTableName(), getConf().getStatementOperation());


Why are both WriteOperation and StatementOperation needed? Aren't they the same?

Hey @kasakrisz ,

StatementOperation represents the top-level intent during query compilation. For example: If a MERGE query is fired from Beeline, then StatementOperation = MERGE.

During Query execution, if the table is configured with write.merge.mode=COW and write.delete.mode=MOR, then MERGE query is splitted into INSERT and DELETE Branch (writeOperation = DELETE) and its incorrectly picking write.delete.mode instead of write.merge.mode.

The aim of this PR is to pass the top-level query operation (StatementOperation) from the compilation layer down to the execution layer so it honor correct iceberg table property.

@kasakrisz do you have in mind alternative solution?

ctx.getOperation() and queryState.getHiveOperation() will not match because for MERGE, Update or SELECT queryState.getHiveOperation() will return HiveOperation.QUERY.

queryState.getHiveOperation() is used extensively in LineageLogger (I've worked on that previously).

To test it I added this in SemanticAnalyzer#createFileSinkDesc() and re-ran the iceberg_mixed_write_modes.q file

LOG.info("TESTING_OPERATIONS: ctx.getOperation() = " + ctx.getOperation() + ", queryState.getHiveOperation() = " + queryState.getHiveOperation());

Output:

deniskuzZ · 2026-09-02T18:55:21Z

-    boolean isCOW = IcebergTableUtil.isCopyOnWriteMode(operation, table.properties()::getOrDefault);
+    boolean isCOW =
+        IcebergTableUtil.isCopyOnWriteMode(
+            statementOperation != null ? statementOperation : writeOperation,


ObjectUtils.defaultIfNull(statementOperation, writeOperation) ?

deniskuzZ · 2026-09-02T18:56:35Z

    }
-    boolean isCOW = IcebergTableUtil.isCopyOnWriteMode(operation, conf::get);
+    boolean isCOW = IcebergTableUtil.isCopyOnWriteMode(
+        statementOperation != null ? statementOperation : writeOperation, conf::get);


deniskuzZ

LGTM, some minor things

- Revert `writeOperation` back to `operation` in WriterBuilder and HiveIcebergSerDe - Use `ObjectUtils.defaultIfNull` instead of ternary operators. - Extract `operation` fallback checks into local variables to improve readability and avoid spotless line wrapping. - Update imports to use `Operation` directly instead of `Context.Operation`.

Aggarwal-Raghav · 2026-09-09T15:14:36Z

rebased with master due to merge conflict becuase of HIVE-29815

kasakrisz · 2026-09-10T12:30:55Z

  private TaskAttemptID attemptID;
  private String queryId;
  private Operation operation;
+  private final boolean isCopyOnWrite;


nit: AFAIK the boolean field name doesn't need the is prefix. The getter has it.

field: boolean copyOnWrite; getter: isCopyOnWrite() setter: setCopyOnWrite(boolean)

kasakrisz · 2026-09-10T12:32:00Z

  private Path destPath;
  private boolean isHiveServerQuery;
  private boolean isMerge;
+  private boolean isCopyOnWrite = false;


nit.

field: boolean copyOnWrite; getter: isCopyOnWrite() setter: setCopyOnWrite(boolean)

See other boolean fields

hive/ql/src/java/org/apache/hadoop/hive/ql/plan/FileSinkDesc.java

Lines 65 to 67 in 2be30b7

private boolean multiFileSpray;

private boolean temporary;

private boolean materialization;

hive/ql/src/java/org/apache/hadoop/hive/ql/plan/FileSinkDesc.java

Lines 352 to 372 in 2be30b7

public boolean isMultiFileSpray() {

return multiFileSpray;

}

/**

* @param multiFileSpray the multiFileSpray to set

*/

public void setMultiFileSpray(boolean multiFileSpray) {

this.multiFileSpray = multiFileSpray;

}

/**

* @return destination is temporary

*/

public boolean isTemporary() {

return temporary;

}

public void setTemporary(boolean temporary) {

this.temporary = temporary;

}

kasakrisz · 2026-09-10T12:32:37Z

    return Boolean.parseBoolean(operation);
  }
+
+  public static void setIsCopyOnWrite(Configuration conf, String tableName, boolean isCopyOnWrite) {


nit.: setCopyOnWrite

kasakrisz

+1, pending tests

Aggarwal-Raghav · 2026-09-10T18:54:44Z

2 un-related UT failure. re-triggering CI

sonarqubecloud · 2026-09-11T03:54:23Z

Quality Gate passed

Issues
1 New issue
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
1.2% Duplication on New Code

See analysis details on SonarQube Cloud

asf-ci-hive added the tests pending label Aug 14, 2026

asf-ci-hive added tests passed and removed tests pending labels Aug 14, 2026

deniskuzZ self-requested a review August 26, 2026 11:12

deniskuzZ reviewed Aug 31, 2026

View reviewed changes

Comment thread ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java Outdated

Aggarwal-Raghav force-pushed the HIVE-29816 branch from 4e6163e to fc5b28a Compare August 31, 2026 10:17

asf-ci-hive added tests pending and removed tests passed labels Aug 31, 2026

deniskuzZ reviewed Aug 31, 2026

View reviewed changes

Comment thread iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/writer/WriterBuilder.java

kasakrisz reviewed Aug 31, 2026

View reviewed changes

asf-ci-hive added tests passed tests pending and removed tests pending tests passed labels Aug 31, 2026

deniskuzZ reviewed Sep 2, 2026

View reviewed changes

Comment thread iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/writer/WriterBuilder.java Outdated

deniskuzZ approved these changes Sep 2, 2026

View reviewed changes

asf-ci-hive added tests pending and removed tests passed labels Sep 2, 2026

Aggarwal-Raghav force-pushed the HIVE-29816 branch from 07a1f23 to 0d2a7f2 Compare September 2, 2026 19:54

asf-ci-hive added tests failed and removed tests pending labels Sep 2, 2026

Aggarwal-Raghav added 4 commits September 9, 2026 20:36

Address review comments 2 writeOperation

f7b056b

Address review comments, pass isCopyOnWrite instead of 2nd operation

562c755

resolve merge conflicts

bd256ff

Aggarwal-Raghav force-pushed the HIVE-29816 branch from 0af100f to bd256ff Compare September 9, 2026 15:13

asf-ci-hive added tests pending and removed tests passed labels Sep 9, 2026

asf-ci-hive added tests passed and removed tests pending labels Sep 9, 2026

kasakrisz reviewed Sep 10, 2026

View reviewed changes

asf-ci-hive added tests pending and removed tests passed labels Sep 10, 2026

kasakrisz approved these changes Sep 10, 2026

View reviewed changes

Aggarwal-Raghav force-pushed the HIVE-29816 branch from b561e60 to 2caf84e Compare September 10, 2026 18:54

asf-ci-hive added tests unstable tests pending and removed tests pending tests unstable labels Sep 10, 2026

style: rename isCopyOnWrite boolean field to copyOnWrite per review

f171f3e

Aggarwal-Raghav force-pushed the HIVE-29816 branch from 2caf84e to f171f3e Compare September 11, 2026 02:55

asf-ci-hive added tests pending and removed tests unstable labels Sep 11, 2026

asf-ci-hive added tests passed and removed tests pending labels Sep 11, 2026

deniskuzZ merged commit db55599 into apache:master Sep 13, 2026
5 checks passed

		setWriteOperation(jc, getConf().getTableInfo().getTableName(), getConf().getWriteOperation());
		setStatementOperation(jc, getConf().getTableInfo().getTableName(), getConf().getStatementOperation());

	private boolean multiFileSpray;
	private boolean temporary;
	private boolean materialization;

	public boolean isMultiFileSpray() {
	return multiFileSpray;
	}

	/**
	* @param multiFileSpray the multiFileSpray to set
	*/
	public void setMultiFileSpray(boolean multiFileSpray) {
	this.multiFileSpray = multiFileSpray;
	}

	/**
	* @return destination is temporary
	*/
	public boolean isTemporary() {
	return temporary;
	}

	public void setTemporary(boolean temporary) {
	this.temporary = temporary;
	}

Uh oh!

Conversation

Aggarwal-Raghav commented Aug 14, 2026

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

Aggarwal-Raghav commented Aug 14, 2026

Uh oh!

Aggarwal-Raghav commented Aug 17, 2026

Uh oh!

Aggarwal-Raghav commented Aug 24, 2026

Uh oh!

deniskuzZ commented Aug 26, 2026

Uh oh!

Aggarwal-Raghav commented Aug 26, 2026

Uh oh!

Uh oh!

Uh oh!

kasakrisz Aug 31, 2026

Choose a reason for hiding this comment

Uh oh!

Aggarwal-Raghav Aug 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

deniskuzZ Sep 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Aggarwal-Raghav Sep 2, 2026

Choose a reason for hiding this comment

Uh oh!

deniskuzZ Sep 2, 2026

Choose a reason for hiding this comment

Uh oh!

deniskuzZ Sep 2, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

deniskuzZ left a comment

Choose a reason for hiding this comment

Uh oh!

Aggarwal-Raghav commented Sep 9, 2026

Uh oh!

kasakrisz Sep 10, 2026

Choose a reason for hiding this comment

Uh oh!

kasakrisz Sep 10, 2026

Choose a reason for hiding this comment

Uh oh!

kasakrisz Sep 10, 2026

Choose a reason for hiding this comment

Uh oh!

kasakrisz left a comment

Choose a reason for hiding this comment

Uh oh!

Aggarwal-Raghav commented Sep 10, 2026

Uh oh!

sonarqubecloud Bot commented Sep 11, 2026

Quality Gate passed

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Aggarwal-Raghav Aug 31, 2026 •

edited

Loading

deniskuzZ Sep 2, 2026 •

edited

Loading