How Matches Pattern Checks Work

Definition

Asserts that every value in a text field matches a configured regular expression.

Overview

The Matches Pattern rule tests a single text field against a regular expression. The row passes when the value matches the expression and fails when it does not. Because the expression describes a shape rather than a set of values, the rule scales to vocabularies that are too large or too open-ended to enumerate: order references, license plates, postal codes, internal identifiers.

Typical use cases:

• Enforce the shape of an identifier, such as SKU-1234 or an invoice reference.
• Validate a formatted value that has no dedicated rule type, such as a postal code or a phone extension.
• Reject placeholder text in a column that must carry a structured value.

Field Scope

Single: The rule evaluates exactly one field per check.

Accepted Types

General Properties

The filter allows you to define a subset of data upon which the rule will operate.

It requires a valid Spark SQL expression that determines the criteria rows in the DataFrame should meet. This means the expression specifies which rows the DataFrame should include based on those criteria. Since it's applied directly to the Spark DataFrame, traditional SQL constructs like WHERE clauses are not supported.

O_TOTALPRICE > 1000

C_MKTSEGMENT = 'BUILDING'

WHERE O_TOTALPRICE > 1000

WHERE C_MKTSEGMENT = 'BUILDING'

O_ORDERPRIORITY = '1-URGENT' AND O_ORDERSTATUS = 'O'

(L_SHIPDATE = '1998-09-02' OR L_RECEIPTDATE = '1998-09-01') AND L_RETURNFLAG = 'R'

WHERE O_ORDERPRIORITY = '1-URGENT' AND O_ORDERSTATUS = 'O'

O_TOTALPRICE > 1000, O_ORDERSTATUS = 'O'

RIGHT(
    O_ORDERPRIORITY,
    LENGTH(O_ORDERPRIORITY) - INSTR('-', O_ORDERPRIORITY)
) = 'URGENT'

LEVENSHTEIN(C_NAME, 'Supplier#000000001') < 7

RIGHT(
    O_ORDERPRIORITY,
    LENGTH(O_ORDERPRIORITY) - CHARINDEX('-', O_ORDERPRIORITY)
) = 'URGENT'

EDITDISTANCE(C_NAME, 'Supplier#000000001') < 7

O_ORDERSTATUS IN (
    SELECT DISTINCT O_ORDERSTATUS
    FROM {{_qualytics_self}}
    WHERE O_TOTALPRICE > 1000
)

O_ORDERSTATUS IN (
    SELECT DISTINCT O_ORDERSTATUS
    FROM ORDERS
    WHERE O_TOTALPRICE > 1000
)

Specific Properties

Matches Pattern has one rule-specific property:

Anomaly Types

Evaluation Flow

Every Matches Pattern check follows the same four-step evaluation flow:

1. Apply the filter clause. If the check has a filter set, only rows matching the filter expression continue to the next step. Rows outside the filter are ignored and cannot contribute to the violation count.
2. Read the field value as text. The platform reads the row's value for the selected field.
3. Match it against the expression. The row passes when the value matches the configured Pattern, and fails when it does not.
4. Apply coverage. At 100% coverage, any failing row causes the check to fail. Below 100% coverage, the check fails only when the passing fraction drops below the threshold (see Coverage and Tolerance).

Anchor the Expression When the Whole Value Matters

An unanchored expression matches anywhere inside the value, so [0-9]{4} accepts SKU-1234, note 1234 here, and 1234567. Anchoring with ^ and $ requires the whole value to match: ^[0-9]{4}$ accepts only a bare four-digit string.

Decide which one you want before saving the check. Containment is useful when the field legitimately carries surrounding text; anchoring is what you want for identifiers and codes, and it is the more common intent.

Case and Whitespace Are Part of the Pattern

Matching is literal: ^[A-Z]{3}$ rejects abc, and a trailing space makes "ABC " fail an anchored three-letter pattern. Build case-insensitivity into the expression (^[A-Za-z]{3}$) or normalize the column upstream. Note that the check form does not trim the values it tests.

NULL Handling

The check passes NULL values: a row with NULL in the evaluated field is not counted as a violation. Matches Pattern only asserts that present values have the right shape.

An empty string, by contrast, is a present value: it fails any pattern that requires at least one character. If the field must also be populated, pair the check with Not Null.

Array Fields

On an array field, every element is tested against the expression and the row counts as failing as soon as one element does not match. Empty arrays and NULL arrays pass, since there is nothing to evaluate. A field nested under an array is evaluated the same way: it is enough that one ancestor on the path is an array.

On arrays the check runs as a column-level assertion, so the result is always reported as a single Shape Anomaly for the dataset, at any coverage. Per-row Record Anomalies, and the Custom Anomaly Description that rewrites them, apply only to scalar text fields.

The Filter Clause

The filter clause is a SQL WHERE expression applied before the evaluation. Filtered-out rows are ignored entirely (they cannot trigger a violation and are not counted in the totals).

When a filter is set, both the Record Anomaly and the Shape Anomaly messages end with [filter: <expression>] so the evaluated scope is visible in the alert.

Coverage and Tolerance

Coverage is a fractional value between 0 and 1 that defines the minimum fraction of evaluated rows that must pass:

• 1.0 (100%, default): every row in the filtered set must pass. Any failing row causes the check to fail. This is the strictest setting.
• < 1.0: the check tolerates a fraction of rows failing. The check fails only when the fraction of passing rows drops below the threshold.

Lower coverage values are useful when a small, known fraction of failures is expected. Use coverage carefully: a 0.5% tolerance can mask a real regression that happens to fall just under the threshold.

See Also