Scan and Remediate

Scan & Remediate is the second phase of the Entity Resolution recipe: "Full scan, pick golden records, and write the golden set." It starts from the active check that Build and Validate produced, scans every record, groups the duplicates it finds, and writes the records you keep to your enrichment destination. This page describes each of its four steps.

Scan

Before anything runs, the step checks that the source datastore has a linked enrichment destination, because the scan needs somewhere to store the source records of the duplicates it finds, and the golden set will be written there too. Without one, the step blocks in place with the message Choose a datastore to receive the scan's records, explaining that the datastore isn't linked to an enrichment destination yet and that linking one lets the scan retrieve the source records and remediation write the golden set. The Link Enrichment button opens the same Enrichment Destination dialog as the datastore page, and the step unblocks as soon as the link is saved. See Link an Enrichment Destination from the Scan Step.

With the enrichment destination in place, the Run Scan view reads "Ready to scan all records. Run the entity-resolution check across every record to find all duplicate entities." Clicking Scan starts a full, non-incremental scan of the selected table, the same operation you would run from the datastore page. The scan appears in the datastore's Activity and in the Working Assets panel while it runs. It may take a few minutes depending on data volume.

The Review Results view opens when the scan finishes. It shows Entities identified and Duplicate records out of the records scanned, a composition bar with the share of the table that has duplicates, and a Scanned with row of chips recording the distinction field, the blocking fields when any were set, the compare fields, and the match threshold the scan used. The recipe then loads the anomaly's source records, which are the duplicates you will remediate. If they are not loaded yet, a Source records not yet loaded card offers Load Source Records.

When AgentQ is available, it interprets the results in a banner with a headline and two tabs, Analysis and Actions, the latter listing recommended next steps.

Run scan again repeats the scan with the same settings and discards the previous results, golden-record selections, and anything else downstream. When the scan finds no duplicates, there is nothing to remediate, and the recipe moves straight to the Complete step. Otherwise Next becomes available once the source records are loaded.

Remediate

The step lists every entity the scan found as a card headed Entity N with its record count, four entities per page. Open an entity to see its records in a table with a Keep as golden column. Tick the record to keep. You can keep more than one record in an entity when the scan grouped records that you want to preserve independently: every ticked record is retained in the golden set, and every unticked record in that entity is excluded as a duplicate. Each entity with more than one record needs at least one record kept; an entity with a single record counts as resolved on its own.

The Entities resolved tile counts your progress, and a progress bar fills as entities are settled. Next entity moves forward, and Next unresolved jumps to the next entity that still needs a decision, wrapping around at the end.

AI help on this step is opt-in, because it sends the records themselves to the model. A banner reads "AI suggestions are available. Allow your configured AI model to review these source records and select the best golden record." Nothing is sent until you click Get AI suggestions. AgentQ then scores the entities a few at a time, highlighting its pick in each table with the reasoning behind it, and you can Accept it per entity or use Accept AI suggested golden records to accept them all at once. Accepting a suggestion adds the recommended record without clearing others you have ticked. AgentQ recommends one record per entity; keeping more is your call. When a recommendation fails for some entities, the step says so and you pick those records yourself.

Next requires every entity with more than one record to have at least one record kept.

Materialize

Under Select What to Write, shown once the recipe has resolved the enrichment destination, the step lists where the outputs will land (Written to and the destination) and the two outputs it can write:

• Deduplicated Records, "The records retained from each entity." This is the golden set and is always written.
• Excluded Records, "The duplicates removed from the golden set." This optional companion is written when you tick it, with the same name plus an _inverse suffix.

Both output names are shown before you write anything (see How It Works for the naming scheme). The Preview tabs, Golden records and Excluded, list the records each output will receive.

The excluded records travel to the materialize operation as a list of distinction-field values to leave out, so two limits apply per run: at most 25,000 excluded records, and a bounded total size for their values. When either is exceeded the step explains, for example "This selection excludes N records, above the limit of 25,000. Narrow the scan or resolve these duplicates in batches.", and Materialize stays disabled until the selection fits.

Clicking Materialize runs a materialize operation on the source datastore ("Writing the golden set…", or "Writing the golden set and the excluded records…"). When it completes, the recipe acknowledges the anomaly the scan raised, since its duplicates have been remediated, confirms it with a message, and moves to the Complete step.

Complete

The final step opens with Entity Resolution Completed and the caption "Your golden set is ready. Use it to fix the source data and keep duplicates from coming back." Three tiles summarize the run: records analyzed, golden records kept, and duplicates excluded. After a scan that found nothing, the caption reads "No duplicate entities were found. Your data is already resolved." instead.

Everything this run produced lists the outputs as cards that open each one:

Start Over clears your progress and returns to the first step. Nothing the run produced is deleted. See Entry Points and Resuming.

See Also