Skip to content

Data Diff Troubleshooting

Common problems when creating and running a Data Diff check, their causes, and how to resolve them.

Validation Fails When Creating the Check

Clicking Validate returns an error instead of a success message.

Cause: The rule could not run against the selected data. The most common reasons are a field that does not exist on the reference container, an invalid filter clause on either side, or a container that failed to load.

Resolution:

  1. Review the error message: it explains what needs attention.
  2. Confirm every compared field, Row Identifier, and Passthrough Field exists on both containers with a comparable type.
  3. Check both Filter Clause fields for typos: each must be a valid SQL WHERE expression for its own container.
  4. If the message says a container is marked as Unloadable, it was skipped after repeated operation failures. Follow the steps in Unloadable Container Error.

Every Row Is Reported as Added and Removed

The anomaly reports the whole dataset twice: once as added, once as removed.

Cause: The two sides are not pairing. Either no Row Identifiers are set, or the identifier does not match across the containers: a surrogate key regenerated at load time, a different type on each side, or a value that is formatted differently (padded, trimmed, cased).

Resolution: Set Row Identifiers that are stable and identical on both sides. When the identifier is stored differently, normalize it upstream (or with a Computed Field) so the two sides pair on the same value.

Differences That Do Not Matter Keep Firing

The check reports rows that differ only in trailing decimals, milliseconds, casing, or whitespace.

Cause: Without a tolerance, the comparison is exact for every compared field.

Resolution: Set the Comparators for the affected types: numeric tolerance for amounts and metrics, duration tolerance for timestamps, string normalization for casing and whitespace. Comparators exist exactly for the noise a pipeline legitimately introduces.

The Anomaly Reports Rows That Only Exist on One Side by Design

A one-way flow reports added rows that are expected, because the target keeps receiving new data.

Cause: By default all three change types are reported: added, removed, and changed.

Resolution: Restrict Diff Change Types to the statuses that represent a problem for your flow (often removed and changed). This requires Row Identifiers, since classification depends on pairing.

The Two Sides Cover Different Slices

Row counts in the message differ far more than expected, with large added and removed groups.

Cause: The two filters do not scope the same slice. Filtering only the target (or only the reference) makes the extra rows on the unfiltered side look added or removed.

Resolution: Keep the two filters aligned, and remember the target filter is applied under Target while the reference filter lives on the Right Reference panel. The anomaly message does not echo either clause, so open the check itself to confirm the scope each side was read with.

No Side-by-Side Comparison in the Anomaly

The anomaly opens with the standard Source Records list instead of the paired comparison view.

Cause: The Comparison Source Records view requires Row Identifiers. Without them the check can only report a set-level difference, with no way to align a target row with a reference row.

Resolution: Add Row Identifiers and re-run the scan. See Comparison Source Records for the full view reference.

An Edited Check Keeps Behaving the Old Way

You changed the fields, identifiers, or comparators but the anomaly list did not change.

Cause: Edits take effect on the next Scan. Saving the check does not re-evaluate the data, and anomalies raised under the previous configuration are not modified.

Resolution: Run a Scan on the container (or wait for the scheduled one). Old anomalies stay open until you triage them or a Full scan with Auto Resolve clears them. See What Happens to Existing Anomalies.

The Check Is Slow on Large Tables

The scan takes much longer than other checks on the same container.

Cause: Data Diff reads both containers and pairs them row by row, so it is the most expensive comparison rule. Wide composite identifiers and long field lists add to that cost.

Resolution: Pair on the narrowest stable identifier, compare only the fields that carry business meaning (move the rest to Passthrough Fields), and scope both sides with aligned filters. When per-row detail is not needed, a Volumetric or Aggregation Comparison check answers "did anything diverge?" at a fraction of the cost.