Data Reconciliation Recipe Examples
Real scenarios showing two assets compared, their mismatches resolved, and the keep set written. Pick a tab to see how the choice of sides, identifiers, and tolerances differs with the situation.
Reconciling a warehouse table with its source system
Context. A finance team loads an orders table from the order management system into a warehouse every night. Analysts complain that totals in the warehouse do not match the source. The team opens Recipes, chooses Data Reconciliation, selects the warehouse table on the Left as the asset to inspect and the source system's table on the Right as the reference.
What happens. The Schema overlap panel shows twelve shared fields and one type mismatch on a date column stored as text in the warehouse. AgentQ suggests order_id as the row identifier, and on Compare Fields suggests comparing status, amount, and currency while passing through region for context. On Tolerances they keep all three change types and set the numeric comparator to Absolute with a tolerance of 0.01 for rounding. Validate accepts the check, and the full scan reports a few hundred changed rows and a handful of rows only the warehouse has. On Reconcile they pick Use right for the changed rows, since the source system is the reference, and Ignore row for the added rows, which were test orders. Materialize writes a keep set of corrected amounts and a remove set of the test orders.
Why it works. Putting the copy on the left and the system of record on the right means every pick is a correction to the copy, and the reference is never asked to change. The absolute tolerance turns harmless rounding into agreement, so only real differences reach the reconcile step.
Confirming two systems agree before a cutover
Context. A team is migrating customer records from a legacy CRM to a new platform and must prove the two agree before switching over. Both tables are profiled, and the new platform's datastore has an enrichment destination linked.
What happens. They put the new platform's table on the left and the legacy table on the right, pick customer_id as the identifier, and compare email, phone, and updated_at. On Tolerances they set the date and time comparator to Within 5 minutes, since the migration job stamps each record on write, and turn on Ignore whitespace for the text fields. The scan finds rows the new platform is missing, reported as Removed. On Reconcile they pick Keep row for each, so the legacy rows land in the keep set for the migration team to load, and Use left for the few changed rows where the new platform holds the corrected value. The keep set becomes the punch list for the final load.
Why it works. The duration tolerance and whitespace rule remove the differences the migration itself introduces, and the change types separate "missing on the new side" from "different values", so each kind gets the right decision. Because the check stays active, the same scan is rerun after the final load to confirm the two agree.
Watching two assets drift after the recipe finishes
Context. The finance team from the first scenario wants to know when the warehouse and the source disagree again, without rebuilding the configuration.
What happens. The Data Diff check the recipe created is still Active on the warehouse table, so they add it to the nightly scan schedule. When a scan raises a Data Diff anomaly, a team member opens it on the datastore page and clicks Reconcile in Recipe. The recipe opens directly at the Reconcile step with that anomaly's mismatched rows loaded, they resolve them, and materialize a fresh keep set. Between loads, Scan in Recipe on the check reruns the comparison on demand.
Why it works. The recipe is a guided flow around an ordinary check, not a separate object. Scheduled scans keep reporting drift, and the entry points take the team straight to the phase that needs a human.
Reconciling more mismatches than fit on screen
Context. After a bad load, a comparison between a product table and its reference finds tens of thousands of changed rows.
What happens. The Scan step reports the full count, and the Reconcile step loads a sample of 10,000 mismatched rows to work with, saying so in its progress tile. The team resolves the sample by change type: Use right for the changed rows, since the reference is correct, and Ignore row for the few added rows. Their picks by change type become the defaults that cover the mismatches that were not loaded. Materialize applies the picks and the defaults to every mismatched row and writes the keep set.
Why it works. A human cannot review tens of thousands of rows, but a decision per change type is exactly what a bad load needs. The sample lets the team confirm the pattern on real rows before the defaults are applied everywhere, and the Complete step's counts confirm how many rows were reconciled.
See Also
-
Best Practices
Guidelines for choosing sides, identifiers, tolerances, and resolutions that hold up.
-
Permissions
The roles and team permissions behind each step of the recipe.
-
How It Works
Left and right, the phases, the prerequisites, what the run reads and writes, and AgentQ's role.
-
Build and Validate
Select Assets, Row Identifiers, Compare Fields, Tolerances, Review, and Validate, step by step.
-
Scan and Reconcile
The full scan, resolving each mismatch, writing the keep and remove sets, and the completion summary.
-
Entry Points and Resuming
Scan in Recipe, Reconcile in Recipe, and how progress is kept between visits.