Data Diff Best Practices

Guidelines for getting reliable signal from Data Diff checks while keeping the noise (and the cost) low.

Always set Row Identifiers when the data has a key

Without Row Identifiers the check can only tell you that the two sides differ, as a set. With them, each target row is paired to its reference row, differences are classified as added, removed, or changed, and the per-row detail is rendered side by side in the Comparison Source Records view. Identifiers turn "something is off" into "these five orders changed, in this column".

Pick the cheapest identifier that is stable on both sides

Pairing on a single integer key is the cheapest case; pairing on a wide composite of string fields costs noticeably more. The identifier must exist and mean the same thing on both containers: an identifier that is regenerated on the reference side (a surrogate key created at load time) pairs nothing and turns every row into an added plus a removed.

Compare only the fields that matter

Every field you list is read and compared on both sides. Listing the whole table makes the check slower and floods the anomaly with columns nobody is validating. Start from the fields that carry business meaning, and add Passthrough Fields for the extra columns you want to see in the comparison view without them triggering differences.

Use Comparators instead of accepting known noise

Timestamps that differ by milliseconds, amounts that differ in the last decimal, or strings that differ only in case or whitespace produce differences that are technically correct and practically useless. Set the numeric, duration, and string Comparators so the check tolerates the noise your pipeline legitimately introduces, and keeps firing on the differences that matter.

Narrow the diff to the slice you actually reconcile

Both sides support a filter clause: filter on the target and the Filter Clause on the Right Reference panel. A full-table diff on every scan is the most expensive shape this rule can take. Scoping both sides to the current day, the current tenant, or the migrated batch cuts the work proportionally and makes the anomaly easier to act on. Keep both filters aligned: filtering one side only manufactures added and removed rows.

Restrict the change types when only part of the diff is a problem

A one-way migration usually cares about removed (rows that failed to arrive) and changed (rows that arrived wrong), while added is expected because the target keeps receiving new data. Setting Diff Change Types to just the statuses you care about keeps the check quiet without weakening it. This requires Row Identifiers, since classification depends on pairing.

Choose the right rule for the job

Use Data Diff when you need to know which rows differ and how.
Use Volumetric when the row count alone answers the question. Much cheaper on wide tables.
Use Aggregation Comparison when a summary statistic (a sum, a count) is enough to detect divergence.
Use Exists In or Not Exists In for referential integrity, when the values themselves do not need comparing.
Use Equal To Field when the two sides are two columns on the same row, not two containers.
Do not build new checks on Is Replica Of: it is deprecated and Data Diff replaces it.

Route the anomalies to the right people

A diff failure usually points at a pipeline between the two containers, not at either container by itself. Set an Anomaly Assignee from the team that owns the replication or ETL step, tag the check (migration, replication), and note in the description which side is the source of truth so whoever triages knows which one to fix.