Unique Anomaly Reporting¶

How the Unique check reports duplicates: the anomaly message it produces, what the counts mean, how duplicate groups appear in Source Records, and why per-row reporting does not apply.

Anomaly Messages¶

When the Unique check fires, it produces a single Shape Anomaly describing the dataset-level violation. The Unique check does not produce Record Anomalies: uniqueness is a property of the dataset as a whole (a duplicate group), not of any individual row's value, so the platform reports the violation at the shape level only.

Single-field check:

order_id values are repeated across N of M asserted records

Composite-key check, where the field names are joined with and:

order_id and line_number values are repeated across N of M asserted records

When a filter is set, the Shape Anomaly message names it at the end of its first sentence, as (applied to records where <expression>).

What the Numbers Mean¶

• M: the number of records the scan read from the container, before the filter. On a filtered check, rows outside the filter still count toward M.
• N: the share of evaluated rows that belong to a duplicate group, applied to M. Every row in a duplicate group counts, not only the extras.

With no filter and no row whose selected fields are all NULL, the evaluated rows are the scanned rows, so N is the number of rows in duplicate groups: for 2 rows sharing the same tuple, N = 2, not 1. N is computed from a ratio and rounded down, so it can occasionally come out one lower than the actual count.

To count the rows in duplicate groups yourself, group by the selected fields and add up the groups that have more than one row. GROUP BY puts NULLs in the same group, so repeated partial-NULL tuples such as (1, NULL) are counted, as they are by the check. A plain IN or = comparison would miss them, because NULL never equals NULL there.

SELECT SUM(cnt) AS rows_in_duplicate_groups
FROM (
    SELECT a, b, COUNT(*) AS cnt
    FROM my_table
    WHERE NOT (a IS NULL AND b IS NULL)  -- rows with every selected field NULL are skipped
    GROUP BY a, b
    HAVING COUNT(*) > 1
) dup;

When a filter is set, or some rows have NULL in every selected field (those rows are skipped, see How NULLs Are Handled), the share is computed on the evaluated rows only and then scaled to M. N can then differ from the number of rows in duplicate groups. To count those rows exactly, run the query above with the check's filter added to its WHERE clause.

For a Shape Anomaly, the failing share appears in the opening line of the anomaly's description, for example 2 of 4 records (50%) in ACCOUNTS failed 1 check during full scan #4821. That line counts against every record the scan processed, so on a filtered check its denominator is no smaller than the check's own asserted count. See Anomaly Description.

Source Records Behavior¶

Because the violation belongs to a duplicate group rather than to one row, Source Records show every row in each duplicate group, not just the extra copies. That makes the group readable end to end: you can see which record should survive and which are the duplicates to remove.

Custom Anomaly Description¶

Custom Anomaly Description does not apply to this rule. The option replaces the message on Record Anomalies, and Unique emits a Shape Anomaly only: the violation belongs to a duplicate group, not to a single row.

See Also¶