Anomaly Fingerprints¶

Anomaly fingerprints are unique identifiers generated for each detected anomaly to help the system recognize and manage duplicates effectively. By comparing these fingerprints, Qualytics can determine whether a newly detected anomaly matches a previously identified one. This mechanism reduces redundancy, ensures consistency in anomaly tracking, and simplifies decision-making during data quality operations.

Info

Record anomalies are always fingerprinted. Shape anomalies are fingerprinted only when the data asset has an incremental identifier or a last-modified date, and Freshness and Volumetric anomalies are never fingerprinted. See Shape Anomalies.

Duplicate Handling Configuration¶

DO Running scans with duplicate handling

Once anomalies are fingerprinted, Qualytics can use these unique identifiers to determine whether a newly detected anomaly matches any existing one. This fingerprint-based recognition powers the duplicate handling configuration during scan setup.

When configuring a scan operation, you can define how the system should respond to anomalies that share fingerprints with previously detected ones:

Duplicate Status: You can instruct the system to automatically mark newly detected anomalies as “Duplicate” if their fingerprints match those of any open anomalies. These duplicates are then archived, ensuring focus remains on the original issue without creating redundant records.
Re-opening Option: If a new anomaly matches an archived one, you can configure the system to automatically re-open the earlier anomaly. This ensures that reoccurring issues are not overlooked simply because they were resolved or dismissed in a prior scan.

Fingerprinting Criteria¶

To determine whether anomalies are identical, Qualytics generates unique fingerprints based on specific criteria. These criteria differ depending on the type of anomaly being evaluated. This approach ensures that anomalies are only considered duplicates when they are truly the same in both nature and context.

Record Anomalies¶

For record-level anomalies, fingerprinting is based on the checks behind the anomaly and the complete source data of the anomalous record. This ensures that every unique row is evaluated precisely:

Identifying check(s): The check or checks the anomaly lists as failed (e.g., a null value or out-of-range check).
Source record data: All field values within the affected row.

Note

Identical records across multiple scans will generate the same fingerprint and therefore be flagged as duplicates.

Which checks go into the fingerprint depends on the One anomaly per check scan default of the record's container:

One anomaly per check	Anomalies per failing record	Checks in the fingerprint
Off (default)	One, listing every check the record failed	The full set of checks the record failed
On	One per failed check	The one check behind that anomaly

A record that fails a single check gets the same fingerprint in both modes. Only records that fail several checks are fingerprinted differently.

Each mode fingerprints the unit you remediate

Each mode fingerprints the anomaly the way your team remediates it, so pick the mode that matches your workflow:

Off: the record is the unit of remediation. The fingerprint covers the full set of checks the record fails. When that set changes, because the record starts failing another check or one of its issues is fixed, the next scan opens a new anomaly instead of marking it Duplicate. The record's owner sees the record's current set of issues in one anomaly.
On: the check is the unit of remediation. The fingerprint covers one check. Whether the record also fails other checks does not affect it, so the anomaly for that check is matched as a duplicate until the record passes that check.

When you change One anomaly per check

Turning the setting on or off, on the container itself or on a datastore or Platform Default the container inherits from, changes how records that fail several checks are fingerprinted. On the first scan after each change, their anomalies do not match the anomalies created under the previous value, so:

They are not marked Duplicate of an anomaly that is already open. New anomalies are opened instead.
With Reactivate Recurring Anomalies turned on, an archived anomaly created under the previous value is not reopened.
With Auto Resolve Anomalies turned on (the default for Full scans), an open anomaly those records got under the previous value is resolved by the first Full scan that runs every check it lists, even though its record still fails. The new anomalies replace it.

Records that fail a single check keep their fingerprint and are matched as usual. From the next scan on, duplicate handling works normally with the new fingerprints. After each change, review the new anomalies from that first scan. Archive the earlier anomalies they replace yourself when auto-resolve can't: a check they list no longer runs (for example, it was archived or left out of the scan), the container only gets Incremental scans, or Auto Resolve Anomalies is off.

Shape Anomalies¶

For shape anomalies, which refer to patterns or distributions of data rather than individual records, the fingerprint is derived from a broader set of attributes:

Identifying check: The check that triggered the anomaly. Every shape anomaly has exactly one failed check.
Failure message: The full text of the failed check's message. Depending on the check, it may include the share and count of records that failed, thresholds, or observed values. A change in any value present in the message produces a different fingerprint.
Scanned range: The highest value of the incremental field and/or the most recent last-modified date in the scanned data.

Tip

Shape anomalies can only be fingerprinted if the data asset includes an incremental identifier. This field anchors the fingerprint to a specific range of data, ensuring accurate comparisons across different scans. Without an incremental field or a last-modified date, a shape anomaly gets no fingerprint and is never marked Duplicate. Freshness and Volumetric anomalies never get a fingerprint, whatever the data asset includes, so every failing scan opens a new one.

This fingerprinting mechanism ensures consistent anomaly tracking by minimizing false duplicates and keeping historical issues relevant when they reoccur.

Anomalies that share a fingerprint are linked to the original, the earliest anomaly with that fingerprint, whatever its status:

On a later anomaly, the Status card in the Summary shows an Original badge with the original's ID. Click it to open the original.
On the original, the Timeline records each later anomaly with the same fingerprint, with its ID as a link.

This lets you follow a recurring issue from any of its occurrences back to the first one, and see from the first one how often it came back.

Recurring Anomalies on Reloaded Tables¶

A table that is truncated and reloaded on a schedule brings the same bad rows back on every load. With Archive Duplicate Anomalies and Reactivate Recurring Anomalies turned on, their fingerprints let each issue appear once instead of on every Scan. See A Table Reloaded Every Night for the full scenario.

If the same rows tend to fail several checks and your team remediates check by check, turn on One anomaly per check for the container as well, so each check's anomaly is matched on its own, whichever other checks those rows fail.