Any Not Null Check

Definition

Asserts that at least one of the selected fields must hold a value.

Overview

The Any Not Null rule evaluates a set of fields together and requires that, on every row, at least one of those fields is populated. A row passes as soon as any selected field carries a non-NULL value, and fails only when every selected field is NULL. The rule is useful when several fields are individually optional but the row is considered incomplete if none of them are present.

Typical use cases:

• Require at least one contact method (phone, email, or mobile).
• Ensure that alternative identifier fields such as national_id or passport_number are not all missing.
• Guarantee that at least one fallback value column is populated when a primary source is optional.

Field Scope

Multiple: The rule is designed for two or more fields per check. A row passes if at least one of them holds a non-NULL value.

Accepted Types

Any Not Null accepts every field type. NULL is treated the same across types, so the check does not restrict which columns can be included.

General Properties

The filter allows you to define a subset of data upon which the rule will operate.

It requires a valid Spark SQL expression that determines the criteria rows in the DataFrame should meet. This means the expression specifies which rows the DataFrame should include based on those criteria. Since it's applied directly to the Spark DataFrame, traditional SQL constructs like WHERE clauses are not supported.

O_TOTALPRICE > 1000

C_MKTSEGMENT = 'BUILDING'

WHERE O_TOTALPRICE > 1000

WHERE C_MKTSEGMENT = 'BUILDING'

O_ORDERPRIORITY = '1-URGENT' AND O_ORDERSTATUS = 'O'

(L_SHIPDATE = '1998-09-02' OR L_RECEIPTDATE = '1998-09-01') AND L_RETURNFLAG = 'R'

WHERE O_ORDERPRIORITY = '1-URGENT' AND O_ORDERSTATUS = 'O'

O_TOTALPRICE > 1000, O_ORDERSTATUS = 'O'

RIGHT(
    O_ORDERPRIORITY,
    LENGTH(O_ORDERPRIORITY) - INSTR('-', O_ORDERPRIORITY)
) = 'URGENT'

LEVENSHTEIN(C_NAME, 'Supplier#000000001') < 7

RIGHT(
    O_ORDERPRIORITY,
    LENGTH(O_ORDERPRIORITY) - CHARINDEX('-', O_ORDERPRIORITY)
) = 'URGENT'

EDITDISTANCE(C_NAME, 'Supplier#000000001') < 7

O_ORDERSTATUS IN (
    SELECT DISTINCT O_ORDERSTATUS
    FROM {{_qualytics_self}}
    WHERE O_TOTALPRICE > 1000
)

O_ORDERSTATUS IN (
    SELECT DISTINCT O_ORDERSTATUS
    FROM ORDERS
    WHERE O_TOTALPRICE > 1000
)

Anomaly Types

Next Steps