Scan and Reconcile

Scan & Reconcile is the second phase of the Data Reconciliation recipe: "Full scan, resolve every mismatch, materialize the keep set." It starts from the active check that Build and Validate produced, scans every record, lets you decide which side is right for each mismatch, and writes the corrections to your enrichment destination. This page describes each of its four steps.

Scan

Activate the check and reconcile every record.

Before anything runs, the step checks that the left datastore has a linked enrichment destination, because the scan writes every mismatched row there, with both sides' values, and that is what the mismatch explorer reads. Without one, the step blocks in place with the message Choose a datastore to receive the mismatched rows and a Link Enrichment button, which opens the same Enrichment Destination dialog as the datastore page. The step unblocks as soon as the link is saved. See Link an Enrichment Destination from the Scan Step.

With the enrichment destination in place, the Run Scan view reads "Ready to scan all records" and explains that running the check across every record of the left asset against the right asset activates the check, so future scheduled scans keep reporting drift. Clicking Scan starts a full scan of the left asset, the same operation you would run from the datastore page. It may take a few minutes depending on data volume.

The Review Results view opens when the scan finishes:

• Rows analyzed across both sides of the comparison, and Mismatched rows broken down into added, removed, and changed. When the number of rows only the right asset has is projected from a sample, the figure is marked as estimated.
• A composition bar with the share of rows without differences.
• A Scanned with row of chips recording the identifiers, the compared fields, the tolerances, and the change types the scan used.

The recipe then loads the mismatched rows from the scan's anomaly, up to 10,000 of them, for you to resolve. Larger mismatch sets are sampled, and the Reconcile step says so. If the rows are not loaded yet, a Mismatch records not yet loaded card offers Load Mismatch Records.

When AgentQ is available, it interprets the results in a banner with a headline and two tabs, Analysis and Actions.

Run scan again repeats the scan with the same settings and discards the previous results and resolutions. When the scan finds no mismatches, there is nothing to reconcile, and the footer offers Complete. Otherwise Next becomes available once the mismatched rows are loaded.

Reconcile

Review every mismatch and decide which side is right, like resolving merge conflicts.

The step lists the mismatched rows in a table, with tabs for All, Changed, Added, and Removed. Each row shows where it was found, its status, the row identifiers, and one column per compared field. Where the two sides disagree, the cell shows both values with an inline diff, and clicking it opens the full diff. Columns that match can be hidden. Fifty rows are shown per page.

For each mismatch, pick a side in the Choose column. The labels depend on the change type:

The Mismatches resolved tile counts your progress. Two pills project the outcome as you go: Keep set, the corrections the keep set will carry, and Remove set, the added rows listed for deletion. When the scan found more mismatches than were loaded, the tile says the loaded rows are a sample, and defaults you set per change type cover the rest of the mismatches when the keep set is written.

AI help on this step is opt-in, because it sends both sides' row values to the model. A banner reads "AI suggestions are available. Allow your configured AI model to review these mismatched rows and pick the correct side for each." Nothing is sent until you click Get AI suggestions. AgentQ then scores the mismatches a few at a time, and you can accept them one by one or with Accept AI suggested resolutions. When some mismatches get no suggestion, the step says how many and lets you retry them or resolve them yourself.

When every pick keeps the left asset as it is, the step notes that no keep or remove set will be created. Next requires every loaded mismatch to be resolved.

Materialize

Write the keep and remove sets to your enrichment destination.

Under What Gets Written, the step shows where the outputs will land (Written to and the enrichment destination) and the outputs it will write:

• Keep Set: "The values the left asset should end up with, and which asset each one came from."
• Remove Set: "The rows the left asset holds that the reference does not, listed on their own to delete." Shown only when some added rows were set to be ignored.

Both output names are shown before you write anything (see How It Works for the naming scheme). The Preview tabs list the rows each output will receive, built from the loaded mismatches; when they were sampled, the caption notes that the operation reconciles all mismatched rows.

Clicking Materialize runs a materialize operation on the left datastore that applies your picks, and the defaults for any mismatch that was not loaded, to every mismatched row. When it completes, the recipe acknowledges the anomaly the scan raised and moves to the Complete step.

When every mismatch was resolved in favour of the left asset, the step reads Nothing to materialize: the left asset's values already stand, so no keep or remove set will be created. Complete then records the run without writing anything and archives the anomaly as Discarded.

Complete

The reconciliation report is ready. Review everything the run produced.

The final step opens with Data Reconciliation Completed and one of three captions: the keep set is ready, with each row carrying the value the left asset should hold and which asset it came from; every mismatch was resolved in favour of the left asset, so no keep or remove set was needed; or every record lines up between the two tables. Three tiles summarize the run: rows analyzed, records reconciled, and mismatched rows.

Everything this run produced lists the outputs as cards that open each one:

Start Over clears your progress and returns to the first step. Nothing the run produced is deleted. See Entry Points and Resuming.

See Also