How Entity Resolution Checks Work

This page explains how field roles and comparison types control candidate selection, scoring, clustering, and the resulting Shape Anomaly. It also covers cluster review and golden-record selection in the Entity Resolution Recipe.

If you only need a quick reference, the Introduction covers the formal definition, field scope, and supported comparisons.

How the Check Evaluates Entity Resolution

Every Entity Resolution check follows the same five-step flow:

1. Apply the filter clause. Only rows that match the filter continue. Rows outside the filter cannot cause a violation.
2. Apply block fields. Records must have the same value on every field whose role is Block. A disagreement prevents the records from becoming a candidate pair.
3. Score comparison fields. Each field whose role is Compare produces a score. The selected comparison type controls how the score is calculated, and the field's weight controls its influence on the composite score.
4. Build clusters from matching pairs. A pair matches when its composite score is greater than or equal to the composite match threshold. Matches are grouped transitively: if A matches B and B matches C, all three records belong to one cluster even when A and C do not match directly.
5. Enforce the distinction field. A cluster is non-compliant when its records contain more than one value of the distinction field.

Block fields are evaluated before comparison fields. A pair that disagrees on a block field is never scored, regardless of its other similarities. An exact comparison field behaves differently: disagreement scores 0.0, but evidence from other comparison fields can still lift the composite score above the threshold.

Field Role and Comparison Type

Every target field has four common settings:

• Field name: the source field to evaluate.
• Type: String, Numeric, or DateTime.
• Role: Block or Compare.
• Comparison type: the type-specific method used to compare values.

Compare fields also have a weight. Block fields do not contribute to the composite score, so their weight is not used.

Block Fields

A block field is an exact hard boundary. Records with different values are not considered as a pair. Good block fields represent rules with no reasonable exceptions, such as tenant, legal entity, or another partition where cross-boundary matches would always be wrong.

Blocking can also reduce the number of pairs that must be scored. However, a field such as Country should not be a blocker when cross-country matches are rare but still possible. Configure it as an exact comparison field instead.

String Comparisons

Optional fuzzy string settings:

• pair_substrings promotes the field score to 1.0 when one value contains the other, such as "ACME" and "ACME Inc.".
• pair_homophones promotes the field score to 1.0 when the values sound alike, such as "Catherine" and "Katherine".
• consider_term_frequency gives uncommon tokens more influence than common tokens.

Numeric Comparisons

Datetime Comparisons

Weights and the Composite Score

For each candidate pair, the platform multiplies every comparison field's score by its weight, adds the weighted scores, and divides by the total comparison weight:

composite = sum(score_i * weight_i) / sum(weight_i)

Increasing a field's weight gives it more influence. When comparison fields are configured, at least one must have a positive weight.

For example, consider these comparison fields:

If the name matches perfectly but industry and country both differ, the composite score is 4 / 7, or about 0.57. At a threshold of 0.85, the pair does not match. If industry and country agree, a slightly different name can still clear the threshold. This lets categorical fields provide positive and negative evidence without creating an absolute boundary.

The Composite Match Threshold

The composite_match_threshold is a value between 0.0 and 1.0 (default 0.7):

• Lower threshold, such as 0.6: tolerates more variation. More pairs match, clusters grow larger, and unrelated records are more likely to be grouped.
• Higher threshold, such as 0.9: requires stronger evidence. Clusters stay smaller, but real variations may be missed.

Start at the default, inspect the first scan's Source Records, and adjust the threshold or weights until the clusters reflect your definition of the same entity.

NULL Values

• A record with NULL in any block field cannot be paired.
• A comparison field with NULL on either side contributes 0.0, while its weight remains part of the composite score.

A pair with a NULL comparison value can still match when its other comparison fields provide enough evidence.

How Clusters Become Entities

Pairs that meet the threshold form connected clusters:

• A and B belong to the same cluster when they match directly.
• A, B, and C belong to the same cluster when A matches B and B matches C, even if A and C do not match directly.

Each cluster receives a _qualytics_entity_id. This system-managed column appears in Source Records so records from the same cluster can be identified.

The Resulting Shape Anomaly

Entity Resolution produces one Shape Anomaly for the check. The violation belongs to a cluster of records rather than one individual field value, so the check does not produce Record Anomalies.

Anomaly Message Format

N records were resolved to D distinct entities (composite threshold T: field_a (w=W), field_b (w=W) ...). K of those entities are assigned more than one value of <distinction_field>

When block fields exist, the message includes blocked on [...].

• N: distinct records analyzed after filtering, block-field NULL handling, and de-duplication.
• D: entity clusters produced.
• T: configured composite match threshold.
• K: clusters with more than one distinction-field value.

Source Records

Source Records include only non-compliant clusters. Within each cluster, the anomaly shows one example row for each distinct value of the distinction field. Every row includes _qualytics_entity_id so the cluster boundaries are visible.

Using the Entity Resolution Recipe

The Recipe presents comparison fields as the primary matching configuration. Select weighted evidence and choose a comparison type for each field in Compare Fields.

Blocking fields are an advanced, optional setting alongside the distinction field. Expand Fields that segment possible duplicates with high confidence. to select hard boundaries. Records with differing values in any selected blocking field will never be considered duplicates. Leave this section empty when the data has no absolute boundary.

When AgentQ is available, it recommends comparison fields when you reach Compare Fields. It recommends blocking fields separately only when you expand the advanced setting. A blocking suggestion does not choose comparison fields, and a comparison suggestion does not add blockers. You can edit either recommendation manually.

After a full scan, the Recipe groups source records by entity. Select one or more rows in the Keep as golden column for each cluster. Every selected row is retained in the golden set, and every unselected row in that cluster is excluded as a duplicate.

Retaining Multiple Records from a Cluster

Select multiple rows when the check grouped records that you want to preserve independently. For example, if A and B are duplicates but C is a different entity, select the preferred record from A or B and also select C. The golden set retains both selected rows.

AgentQ recommends one best-fit record from each cluster. Accepting that recommendation selects the suggested row without clearing other rows you selected. AgentQ does not select additional rows because the clusters were created by the deterministic check.

You can clear any selection before creating the golden set. The Recipe requires at least one selected row for each cluster.

Performance Considerations

• Use block fields for true boundaries. They reduce candidate pairs and can improve performance on large containers.
• Do not use a block field when exceptions should remain possible. Use an exact comparison helper with an appropriate weight instead.
• Filter the check to a meaningful scope when resolution is needed only for part of the data.

Entity Resolution skips evaluation when there are more than 5,000,000 distinct combinations across the target fields. The operation logs explain when this safeguard is reached.

If more than 90% of candidate records remain in single-record clusters, the operation logs suggest reviewing the threshold or adding a suitable fuzzy string comparison.

Relationship with Other Rule Types

Related

• Introduction: formal definition, field roles, comparison types, and field scope.
• Examples: production scenarios with sample data and resulting anomalies.
• API: current public payload and field reference.
• FAQ: short answers to common configuration and remediation questions.