How Not Null Checks Work

Definition

Asserts that every selected field holds a value on every row.

Overview

The Not Null rule is a completeness assertion. Each selected field must be populated on every row; a row where any one of them is NULL is flagged. When several fields are selected, they are evaluated together as one rule: the row must have all of them populated, so the check states "these columns are mandatory" in a single place rather than one check per column.

Typical use cases:

• Enforce mandatory columns on a table, such as an identifier, a status, or a created-at timestamp.
• Catch an upstream job that started writing rows with a column omitted.
• Pair with a format or range rule so a field is both present and valid.

Field Scope

Multiple: The rule accepts one or more fields, evaluated together. A row passes only when every selected field is populated.

General Properties

The filter allows you to define a subset of data upon which the rule will operate.

It requires a valid Spark SQL expression that determines the criteria rows in the DataFrame should meet. This means the expression specifies which rows the DataFrame should include based on those criteria. Since it's applied directly to the Spark DataFrame, traditional SQL constructs like WHERE clauses are not supported.

O_TOTALPRICE > 1000

C_MKTSEGMENT = 'BUILDING'

WHERE O_TOTALPRICE > 1000

WHERE C_MKTSEGMENT = 'BUILDING'

O_ORDERPRIORITY = '1-URGENT' AND O_ORDERSTATUS = 'O'

(L_SHIPDATE = '1998-09-02' OR L_RECEIPTDATE = '1998-09-01') AND L_RETURNFLAG = 'R'

WHERE O_ORDERPRIORITY = '1-URGENT' AND O_ORDERSTATUS = 'O'

O_TOTALPRICE > 1000, O_ORDERSTATUS = 'O'

RIGHT(
    O_ORDERPRIORITY,
    LENGTH(O_ORDERPRIORITY) - INSTR('-', O_ORDERPRIORITY)
) = 'URGENT'

LEVENSHTEIN(C_NAME, 'Supplier#000000001') < 7

RIGHT(
    O_ORDERPRIORITY,
    LENGTH(O_ORDERPRIORITY) - CHARINDEX('-', O_ORDERPRIORITY)
) = 'URGENT'

EDITDISTANCE(C_NAME, 'Supplier#000000001') < 7

O_ORDERSTATUS IN (
    SELECT DISTINCT O_ORDERSTATUS
    FROM {{_qualytics_self}}
    WHERE O_TOTALPRICE > 1000
)

O_ORDERSTATUS IN (
    SELECT DISTINCT O_ORDERSTATUS
    FROM ORDERS
    WHERE O_TOTALPRICE > 1000
)

Specific Properties

Not Null has one rule-specific property, which only applies to array fields:

Anomaly Types

Evaluation Flow

Every Not Null check follows the same four-step evaluation flow:

1. Apply the filter clause. If the check has a filter set, only rows matching the filter expression continue to the next step. Rows outside the filter are ignored and cannot contribute to the violation count.
2. Read the selected fields. The platform reads every field listed on the check for the current row.
3. Test each one for presence. The row passes when all of them hold a value, and fails as soon as one is NULL.
4. Apply coverage. At 100% coverage, any failing row causes the check to fail. Below 100% coverage, the check fails only when the passing fraction drops below the threshold (see Coverage and Tolerance).

One Check, Several Mandatory Fields

Selecting several fields makes the check assert them together: the row must have all of them populated. That is the opposite of Any Not Null, which passes as soon as one of the selected fields is populated. Use Not Null for a set of individually mandatory columns, and Any Not Null for a group of interchangeable alternatives.

One check covering five mandatory columns is easier to maintain than five checks, but it also reports a single anomaly stream. When each column has a different owner or a different remediation path, separate checks make triage clearer.

Empty Strings Are Not NULL

The rule tests for NULL, not for emptiness. A row where the column holds '', a space, or a placeholder such as n/a passes, because the field is populated. When blank values are a real possibility in your source, normalize them to NULL upstream (or with a Computed Field) so absence is represented consistently, or pair the check with Matches Pattern to also reject blanks.

Array Fields

By default an array field is tested as a whole: the row passes when the array itself is not NULL, even if it is empty or contains NULL elements. Turning on Array Element Context changes that: every element inside the array must then be non-NULL for the row to pass.

The Filter Clause

The filter clause is a SQL WHERE expression applied before the evaluation. Filtered-out rows are ignored entirely (they cannot trigger a violation and are not counted in the totals).

When a filter is set, both the Record Anomaly and the Shape Anomaly messages end with [filter: <expression>] so the evaluated scope is visible in the alert.

Coverage and Tolerance

Coverage is a fractional value between 0 and 1 that defines the minimum fraction of evaluated rows that must pass:

• 1.0 (100%, default): every row in the filtered set must pass. Any failing row causes the check to fail. This is the strictest setting.
• < 1.0: the check tolerates a fraction of rows failing. The check fails only when the fraction of passing rows drops below the threshold.

Lower coverage values are useful when a small, known fraction of failures is expected. Use coverage carefully: a 0.5% tolerance can mask a real regression that happens to fall just under the threshold.

See Also