How the Data Reconciliation Recipe Works

The Data Reconciliation recipe walks you from two assets that should agree to a written keep set of corrections. Its side panel sums the job up in one sentence: "Investigate an asset against a reference, confirm every record lines up, and surface the fields where their values disagree." This page covers the mechanics that span the whole flow. The steps themselves are detailed on Build and Validate and Scan and Reconcile.

Left and Right

Every reconciliation has two sides, and the recipe keeps them apart throughout:

Left is the asset to inspect. Mismatches are reported here, the check is created on it, and the keep and remove sets describe what it should end up with.
Right is the reference asset, treated as correct.

Rows are paired by their row identifiers, the shared fields that identify the same row on both sides. The scan then reports three kinds of difference, always from the left asset's point of view:

Change type	Meaning
Added	Rows only the left asset has.
Removed	Rows the left asset is missing.
Changed	Rows whose compared values differ.

Fields are paired by name (case-insensitively) and must share a compatible type: text with text, numbers with numbers, booleans, dates and timestamps, and so on. The Select Assets step shows this overlap before you go further.

Phases and Steps

The recipe runs in two phases. The side panel lists them, and the header above the step content reads Step 1 of 2 or Step 2 of 2 (it counts phases, not steps) followed by the numbered steps of the current phase.

Phase	What it does	Steps
Build & Validate	Configure the reconciliation check and validate it.	Select Assets, Row Identifiers, Compare Fields, Tolerances, Review, Validate
Scan & Reconcile	Full scan, resolve every mismatch, materialize the keep set.	Scan, Reconcile, Materialize, Complete

Back is available on every step after the first, and every step you have already reached stays clickable. Steps ahead of you lock while the current step's requirement is unmet: two assets with at least one compatible shared field, at least one row identifier, at least one compare field, at least one change type, a validated check, a completed scan with its mismatches loaded, every sampled mismatch resolved, and a written keep set.

Changing the configuration after validation (either asset, the identifiers, the compare or passthrough fields, the change types, or the tolerances) clears the validation and everything downstream of it. The check has to be validated again before you can move on to the second phase.

You can collapse the side panel with Hide steps and bring it back with Show steps. The choice is remembered on your device.

Prerequisites

Two conditions block the flow until they are met.

The two assets must share compatible fields. Reconciliation pairs fields by name and type. When no field name appears on both assets, the Select Assets step shows No fields in common and you cannot continue. Fields that share a name but not a type are listed with their two types and simply excluded from the identifier and compare choices.

The left datastore must have a linked enrichment destination before the Scan step. The scan writes every mismatched row there, with both sides' values, which is what the mismatch explorer reads, and the keep and remove sets are written there too. When the left datastore has none, the Scan step blocks in place with the message Choose a datastore to receive the mismatched rows and a Link Enrichment button that opens the same Enrichment Destination dialog as the datastore page. See Link an Enrichment Destination from the Scan Step.

Reads and Writes

Neither asset is changed. The recipe reads both sides, compares them row by row, and writes its results elsewhere: the check, scan, and anomaly live in Qualytics, and the keep and remove sets are new outputs in the left datastore's enrichment destination. The How It Works panel on the Select Assets step shows this as a two-column summary, with both assets under Reads, the keep and remove sets under Writes, and the reminder "Both tables stay exactly as they are".

What the Run Produces

Output	Where it lives	Notes
Data Diff check	The checks of the left datastore	Created as a Draft when you leave Review, set to Active by Validate, and left Active afterwards. Its default description is Data Reconciliation check on followed by the two asset names.
Scan operation	The left datastore's Activity	A full scan of every record, not only the records changed since the last scan.
Anomaly	The left datastore's Anomalies	The mismatch report: every flagged row with both sides' values. Acknowledged automatically when the keep set is written, or archived as Discarded when every mismatch was resolved in favour of the left asset.
Keep set	The enrichment destination	The values the left asset should end up with, and which asset each one came from. Named `<prefix>_mat_dr_<table>`.
Remove set	The enrichment destination	The rows the left asset holds that the reference does not, listed on their own to delete. Named like the keep set with a `_removals` suffix. Written only when it has rows.

Output names in the enrichment destination

The keep set is named after the enrichment prefix of the left datastore and the left table, in the form <prefix>_mat_dr_<table>. With the prefix _qualytics and a left table called ORDERS, the keep set is _qualytics_mat_dr_ORDERS and the remove set is _qualytics_mat_dr_ORDERS_removals. When the source table name contains anything other than letters, digits, and underscores, or is too long, the recipe falls back to an id-based form, <prefix>_materialize_dr_<id>, with the same _removals companion. The Materialize step shows both output names before you write anything. A JDBC enrichment destination stores them as tables. A DFS enrichment destination lowercases the names and stores them as Delta folders.

Inside the enrichment datastore, Qualytics gives the outputs a readable display name, <TABLE> / Data Reconciliation (<left datastore>) for the keep set and <TABLE> / Data Reconciliation / Removals (<left datastore>) for the companion. Each run replaces the contents of the keep set. The remove set is written only when it has rows, so a run with nothing to remove leaves the previous run's remove set in place; check its contents against the Complete step's counts. See Enrichment Introduction.

What Goes into the Keep and Remove Sets

The outputs are corrections only: nothing ever asks the reference asset to change. For each mismatch you pick a side, and the pick decides where the row goes.

Change type	Pick the left side	Pick the right side
Changed	Use left: nothing is written.	Use right: the row goes to the keep set with the right side's values.
Added	Keep row: nothing is written.	Ignore row: the row goes to the remove set.
Removed	Ignore row: nothing is written.	Keep row: the row goes to the keep set with the right side's values.

Both outputs carry the row identifiers, then each compared field under its own name with the value of the side that has the row, plus two fields: the change type and the name of the asset the values came from. When every pick keeps the left asset as it is, there is nothing to write, and the recipe completes without a materialize.

AgentQ's Role

When an AI provider is configured for AgentQ, the recipe offers AI help at defined points. Suggestions appear in a banner at the top of the step with their reasoning and an Apply all action, and you can run them again or hide them. Every step also works without AI.

Where	What AgentQ does	Data sharing level needed
Row Identifiers	Suggests the shared fields that identify a row on both sides, weighing uniqueness and completeness. Runs automatically.	Metadata Shared
Compare Fields	Splits the shared columns into comparisons, context (passthrough), and tolerances, and pre-fills the Tolerances step. Runs automatically.	Metadata Shared
Scan	Interprets the scan results with a headline, an analysis, and recommended next steps.	Metadata Shared
Reconcile	Recommends which side is right for each mismatch. Opt-in: nothing is sent until you click Get AI suggestions, because this step sends both sides' row values to the model.	Source Data Shared

The first three send field names and statistics, never values from your data. The resolution recommendations are the exception, which is why they sit at the higher level and wait for your consent. When AgentQ's data sharing level is below what a step needs, or no AI provider answers, the step reports that its suggestions could not load and lets you continue manually. When no AI provider is configured at all, the AI banners are simply absent. See AgentQ Access Controls. The global AgentQ chat button is hidden while a recipe is open.

Relationship with the Rule Type

The recipe does not change how Data Diff works. It only helps you build the check and act on its results. How rows are paired, how each tolerance is applied, and how the change types are computed are documented on How Data Diff Checks Work. A check created in the recipe can be edited, cloned, archived, or scheduled like any other, and an active Data Diff check can be taken into the recipe with Scan in Recipe. See Entry Points and Resuming.

The How It Works Panel and Working Assets

How It Works is an explainer panel available on the first four steps, opened from the button in the footer. It illustrates each concept with a worked example of two rows: how the row identifier finds a row's counterpart, how a disagreement in a compared field becomes a mismatch while passthrough fields ride along, and how tolerances decide how close two values must be to count as equal. On the Tolerances step the example reacts live to the numeric tolerance and the change types you select.

Working Assets, in the side panel, lists everything the run touches as badges: the Left and Right datastores and assets, the Row Identifiers, the Compare Fields, and then the Check, the Scan, and the Anomaly as the recipe creates them. Each badge opens the asset in a new tab. When you enter the recipe from a check or an anomaly, the matching badge flashes to show where you came in.