Create a Data Diff Check
Step-by-step tutorial for creating a Data Diff check between a target container and a reference container. For what each property means and how the rule evaluates the data, see the How It Works page.
Permission Required
You need the Author team permission on the datastore to create an Active check, or Drafter to create it as a Draft. See the Permissions page for the full matrix.
Show me how
The app can walk you through this. Click Show me how in the check creation form's header, or press H while it is open, and the Add a Check walkthrough highlights each step while you fill in the real form.
Field reference
The Authored Check Details form is organized in the sections below. Fill them in as you follow the Steps.
Target
| Field | Required | Type | Description |
|---|---|---|---|
| Associate with a Check Template | Toggle | Links the check to a Check Template. Enabling it replaces the Rule Type dropdown with a Template dropdown where you pick an existing template, which then controls the check's properties. When off, you configure everything on this form. | |
| Rule Type | Option | The validation logic to apply. Select Data Diff for this check. Shown only while Associate with a Check Template is off. | |
| Table / File | Option | The target container the check runs against. Locked after the check is saved. | |
| Fields | Option | The fields compared between the two sides, read on both the target and the reference container. Every field type is accepted. | |
| Filter Clause | Text | A SQL WHERE expression that scopes the target container before the comparison. Keep it aligned with the reference filter so the two sides cover the same slice. |
|
| Custom Anomaly Description | Toggle | Has no effect on this rule. The option replaces the message on Record Anomalies, and Data Diff emits a Shape Anomaly only. |
Properties
| Field | Required | Type | Description |
|---|---|---|---|
| Row Identifiers | Option | The field (or combination of fields) used to pair each target row with its reference row. Strongly recommended: without identifiers the check reports a set-level difference and cannot classify rows or show them side by side. | |
| Passthrough Fields | Option | Extra fields shown in the comparison view for context. They are displayed but never compared, so they cannot cause a difference. | |
| Diff Change Types | Option | Which change types are reported: added, removed, changed. All three are selected by default. The field stays hidden until Row Identifiers holds at least one field, and clearing the identifiers clears the selection again, because classification depends on pairing rows. |
Right Reference
| Field | Required | Type | Description |
|---|---|---|---|
| Datastore | Option | The datastore holding the reference container. Leave it empty to use the target's own datastore. | |
| Container | Option | The table or file compared against the target. | |
| Filter Clause | SQL | A SQL WHERE expression that scopes the reference container before the comparison. Accepts Check Variables. |
Comparators
| Field | Required | Type | Description |
|---|---|---|---|
| Numeric | Toggle | Enables a tolerance for numeric fields, so values within the configured margin are not reported as different. Turning it on displays Comparison and Threshold. | |
| Comparison | Option | How the tolerance is measured. Absolute Value uses a fixed margin; Percentage Value uses a percentage of the compared value. | |
| Threshold | Number or Percentage | The allowed margin. The field is labeled Percentage Threshold when Percentage Value is selected. | |
| Duration | Toggle | Enables a tolerance for date and timestamp fields. Turning it on displays Unit and Value. | |
| Unit | Option | The unit the tolerance is expressed in: Millis, Seconds, or Days. | |
| Value | Number | The allowed margin, in the selected Unit. | |
| String | Toggle | Enables tolerant text comparison. Turning it on displays Ignore Whitespace. | |
| Ignore Whitespace | Toggle | Ignores spacing differences between the two values. Shown only while String is enabled. |
Ownership
| Field | Required | Type | Description |
|---|---|---|---|
| Owner | Option | The user responsible for the check. Already filled in with the check creator. | |
| Anomaly Assignee | Option | The user automatically assigned to anomalies the check produces. |
Metadata
| Field | Required | Type | Description |
|---|---|---|---|
| Description | Text | A plain-language description of what the check enforces. Supports Markdown formatting: click the field to open the Markdown editor. Click the Apply suggested description button for a suggestion based on the rule type. | |
| Tags | Option | Tags applied to the check for filtering and organization. | |
| Additional Metadata | Key-value | Custom key-value pairs, typically links to catalog entries, tickets, or governance records. |
No Pass Criteria section
Data Diff does not use coverage, so the form shows no Coverage slider. Any difference between the two sides that is not tolerated by a Comparator is reported.
Let AgentQ fill in the form
When AgentQ is configured for your deployment, the Check Assistant panel in the check form can author the check for you: describe the comparison in plain language (for example, "the warehouse copy of orders must match the source system") and it proposes a configuration you can apply to the form. Review the filled-in fields, then Validate and Save as usual.
Steps
Each field is described in the Field reference above.
Step 1: Select the source datastore from the left menu, then click the Checks tab.
Step 2: Click Add in the top-right corner and select Check from the dropdown. The Authored Check Details form opens.
Step 3: Select Data Diff in the Rule Type dropdown.
Step 4: Fill in the remaining Target fields: the target table or file, the Fields to compare, and, optionally, a Filter Clause.
Step 5: Under Properties, set the Row Identifiers that pair the two sides, add any Passthrough Fields for context, and restrict the Diff Change Types when only some change types matter.
Step 6: Fill in the Right Reference panel: the reference Container (and its Datastore when it lives elsewhere) and an optional Filter Clause for that side.
Step 7: Optionally, set the Comparators so differences within a known tolerance are not reported.
Step 8: Fill in the Ownership and Metadata sections. Only the Description is required.
Step 9: Click Validate to test the rule against the selected data without saving it. A success message confirms the rule can run on that data. If validation fails, see Troubleshooting.
Step 10: Click Save. A success message confirms that the check was created.
Common Variations
| Goal | Fields | Row Identifiers | Diff Change Types |
|---|---|---|---|
| Validate a full replica against its source | every business field | the primary key | all three |
| Confirm a migration lost nothing | the migrated fields | the stable business key | removed, changed |
| Reconcile a daily delivery with the system of record | the delivered fields | the delivery line key | all three, with both filters scoped to the delivery date |
| Detect drift on a reporting copy, tolerating rounding | the metric fields | the report key | changed, with a numeric Comparator |