Shape Anomalies¶

Reviewing shape anomalies

A Shape anomaly describes a problem that belongs to many rows or to the dataset as a whole, rather than to a single row. The simplest Shape anomaly is a table that no longer has a field it is expected to have: no row is wrong on its own, the structure is. Shape anomalies also cover problems that only show up when rows are compared with each other, such as duplicate keys, and failures too widespread to report one row at a time.

This page explains what creates a Shape anomaly, what one contains, and how to read it. For failures tied to one row, see Record Anomalies.

What Creates a Shape Anomaly¶

A Scan reports a Shape anomaly in three situations.

A Rule That Looks at the Dataset¶

Some rules can only be answered by looking at the data as a whole: whether a field exists, whether values repeat, whether the row count or the latest update is within its limits, whether two tables agree. These rules always produce Shape anomalies.

See Rule Types for every rule type that produces only Shape anomalies.

A Coverage Below 100%¶

A rule that evaluates each row, such as Not Null or Min Value, can be set to a coverage below 100%. The check then tolerates some failing rows and asks a different question: does the share of passing rows reach the coverage? When it does not, the Scan reports one Shape anomaly for the dataset, with the number of failing rows, instead of one Record anomaly per row.

More Failing Rows Than the Limit¶

When a check at 100% coverage fails more rows than the Scan's Maximum Record Anomalies per Check (10 by default), all of that check's failures are reported as one Shape anomaly, which keeps the total count. No Record anomaly is created for that check, so a widespread failure does not flood the anomaly list. The other checks on the same rows still report their own anomalies. See Maximum Record Anomalies per Check.

What a Shape Anomaly Shows¶

Every Shape anomaly holds exactly one failed check. When several checks fail at the shape level in the same Scan, even on the same fields, each one produces its own Shape anomaly.

The Custom Anomaly Description option of a check does not change Shape anomaly messages. It applies to Record anomalies only.

Impact and Fingerprints¶

Shape anomalies differ from Record anomalies in two ways that affect triage:

• Impact follows the scale of the failure. A coverage or rolled-up failure is rated by the share of checked records that failed: less than 1% is Low, 1% to less than 10% is Medium, and 10% or more is High. A check that tests the table as a whole, such as Expected Schema, Freshness, or Volumetric, is rated High when it fails. The check's Importance then raises or lowers that level. See Anomaly Impact.
• A fingerprint needs a scanned range. The fingerprint of a Shape anomaly is built from its check, the full text of its failure message, and the range of data the Scan read. That range comes from the table's or file's incremental field or last-modified date. Without one, the Shape anomaly has no fingerprint and is never marked Duplicate. Volumetric and Freshness anomalies never have one, so every failing Scan opens a new anomaly.

Because the message is part of the fingerprint, a Shape anomaly whose counts change between Scans, such as 14 of 200 becoming 15 of 210, is a new anomaly, not a duplicate.

Rule Types¶

These rule types look at the data as a whole, so they produce only Shape anomalies. The rule types that evaluate each row produce Shape anomalies too, when the check's coverage is below 100% or when it fails more rows than the limit. See Record Anomalies for those.

Examples¶

The Sample Data tables below follow the Source Records view: one row per record, with the failing cells highlighted. A structural failure has no failing cell, so nothing is highlighted.

See Also¶