# How Not Null Checks Work

Source: https://userguide.qualytics.io/data-quality-checks/not-null/deep-dive/how-it-works/

## Definition

*Asserts that every selected field holds a value on every row.*

## Overview

The Not Null rule is a completeness assertion. Each selected field must be populated on every row; a row where any one of them is `NULL` is flagged. When several fields are selected, they are evaluated together as one rule: the row must have all of them populated, so the check states "these columns are mandatory" in a single place rather than one check per column.

Typical use cases:

- Enforce mandatory columns on a table, such as an identifier, a status, or a created-at timestamp.
- Catch an upstream job that started writing rows with a column omitted.
- Pair with a format or range rule so a field is both present and valid.

## Field Scope

**Multiple:** The rule accepts one or more fields, evaluated together. A row passes only when every selected field is populated.

## General Properties

**Tab: Details**

| Name | Supported |
| --- | --- |
| Filter  Allows the targeting of specific data based on conditions | Supported |
| Coverage Customization  Allows adjusting the percentage of records that must meet the rule's conditions | Supported |

**Tab: Filter Guide**

The filter allows you to define a subset of data upon which the rule will operate.

It requires a valid **Spark SQL** expression that determines the criteria rows in the DataFrame should meet. This means the expression specifies which rows the DataFrame should include based on those criteria. Since it's applied directly to the Spark DataFrame, traditional SQL constructs like WHERE clauses **are not** supported.

> **Example: Examples**
>
> **Direct Conditions**
>
> Simply specify the condition you want to be met.
>
> > **Success: Correct usage" collapsible="true**
> >
> > ```sql
> > O_TOTALPRICE > 1000
> > ```
> >
> > ```sql
> > C_MKTSEGMENT = 'BUILDING'
> > ```
>
> > **Failure: Incorrect usage" collapsible="true**
> >
> > ```sql
> > WHERE O_TOTALPRICE > 1000
> > ```
> >
> > ```sql
> > WHERE C_MKTSEGMENT = 'BUILDING'
> > ```
>
> **Combining Conditions**
>
> Combine multiple conditions using logical operators like `AND` and `OR`.
>
> > **Success: Correct usage" collapsible="true**
> >
> > ```sql
> > O_ORDERPRIORITY = '1-URGENT' AND O_ORDERSTATUS = 'O'
> > ```
> >
> > ```sql
> > (L_SHIPDATE = '1998-09-02' OR L_RECEIPTDATE = '1998-09-01') AND L_RETURNFLAG = 'R'
> > ```
>
> > **Failure: Incorrect usage" collapsible="true**
> >
> > ```sql
> > WHERE O_ORDERPRIORITY = '1-URGENT' AND O_ORDERSTATUS = 'O'
> > ```
> >
> > ```sql
> > O_TOTALPRICE > 1000, O_ORDERSTATUS = 'O'
> > ```
>
> **Utilizing Functions**
>
> Leverage Spark SQL functions to refine and enhance your conditions.
>
> > **Success: Correct usage" collapsible="true**
> >
> > ```sql
> > RIGHT(
> >     O_ORDERPRIORITY,
> >     LENGTH(O_ORDERPRIORITY) - INSTR('-', O_ORDERPRIORITY)
> > ) = 'URGENT'
> > ```
> >
> > ```sql
> > LEVENSHTEIN(C_NAME, 'Supplier#000000001') < 7
> > ```
>
> > **Failure: Incorrect usage" collapsible="true**
> >
> > ```sql
> > RIGHT(
> >     O_ORDERPRIORITY,
> >     LENGTH(O_ORDERPRIORITY) - CHARINDEX('-', O_ORDERPRIORITY)
> > ) = 'URGENT'
> > ```
> >
> > ```sql
> > EDITDISTANCE(C_NAME, 'Supplier#000000001') < 7
> > ```
>
> **Using scan-time variables**
>
> To refer to the current dataframe being analyzed, use the reserved dynamic variable `{{_qualytics_self}}`.
>
> > **Success: Correct usage" collapsible="true**
> >
> > ```sql
> > O_ORDERSTATUS IN (
> >     SELECT DISTINCT O_ORDERSTATUS
> >     FROM {{_qualytics_self}}
> >     WHERE O_TOTALPRICE > 1000
> > )
> > ```
>
> > **Failure: Incorrect usage" collapsible="true**
> >
> > ```sql
> > O_ORDERSTATUS IN (
> >     SELECT DISTINCT O_ORDERSTATUS
> >     FROM ORDERS
> >     WHERE O_TOTALPRICE > 1000
> > )
> > ```
>
> While subqueries can be useful, their application within filters in our context has limitations. For example, directly referencing other containers or the broader target container in such subqueries **is not supported**. Attempting to do so will result in an error.
>
> > **Warning: Important Note on {{_qualytics_self}}**
> >
> > The `{{_qualytics_self}}` keyword refers to the dataframe that's currently under examination. In the context of a full scan, this variable represents the entire target container. However, during incremental scans, it only reflects a subset of the target container, capturing just the incremental data. It's crucial to recognize that in such scenarios, using `{{_qualytics_self}}` may not encompass all entries from the target container.

## Specific Properties

Not Null has one rule-specific property, which only applies to array fields:

| Name | Description |
| --- | --- |
| Array Element Context | Evaluates the elements inside an array field rather than the array as a whole. Only relevant when the selected field is an array. |

## Anomaly Types

| Type | Supported |
| --- | --- |
| Record Flag inconsistencies at the row level | Supported |
| Shape Flag inconsistencies in the overall patterns and distributions of a field | Supported |

## Evaluation Flow

Every Not Null check follows the same four-step evaluation flow:

1. **Apply the filter clause.** If the check has a `filter` set, only rows matching the filter expression continue to the next step. Rows outside the filter are ignored and cannot contribute to the violation count.
2. **Read the selected fields.** The platform reads every field listed on the check for the current row.
3. **Test each one for presence.** The row passes when all of them hold a value, and fails as soon as one is `NULL`.
4. **Apply coverage.** At 100% coverage, any failing row causes the check to fail. Below 100% coverage, the check fails only when the passing fraction drops below the threshold (see [Coverage and Tolerance](https://userguide.qualytics.io/data-quality-checks/not-null/deep-dive/how-it-works/#coverage-and-tolerance)).

## One Check, Several Mandatory Fields

Selecting several fields makes the check assert them together: the row must have **all** of them populated. That is the opposite of [Any Not Null](https://userguide.qualytics.io/data-quality-checks/any-not-null/introduction/), which passes as soon as *one* of the selected fields is populated. Use Not Null for a set of individually mandatory columns, and Any Not Null for a group of interchangeable alternatives.

One check covering five mandatory columns is easier to maintain than five checks, but it also reports a single anomaly stream. When each column has a different owner or a different remediation path, separate checks make triage clearer.

## Empty Strings Are Not NULL

The rule tests for `NULL`, not for emptiness. A row where the column holds `''`, a space, or a placeholder such as `n/a` passes, because the field *is* populated. When blank values are a real possibility in your source, normalize them to `NULL` upstream (or with a Computed Field) so absence is represented consistently, or pair the check with [Matches Pattern](https://userguide.qualytics.io/data-quality-checks/matches-pattern/introduction/) to also reject blanks.

## Array Fields

By default an array field is tested as a whole: the row passes when the array itself is not `NULL`, even if it is empty or contains `NULL` elements. Turning on **Array Element Context** changes that: every element inside the array must then be non-`NULL` for the row to pass.

## The Filter Clause

The filter clause is a SQL `WHERE` expression applied **before** the evaluation. Filtered-out rows are ignored entirely (they cannot trigger a violation and are not counted in the totals).

When a filter is set, the message names it at the end of its first sentence, as `(applied to records where <expression>)`. Record and Shape Anomaly messages both carry it.

## Coverage and Tolerance

Coverage is a fractional value between `0` and `1` that defines the minimum fraction of evaluated rows that must pass:

- **`1.0` (100%, default):** every row in the filtered set must pass. Any failing row causes the check to fail. This is the strictest setting.
- **`< 1.0`:** the check tolerates a fraction of rows failing. The check fails only when the fraction of passing rows drops below the threshold.

Lower coverage values are useful when a small, known fraction of failures is expected. Use coverage carefully: a 0.5% tolerance can mask a real regression that happens to fall just under the threshold.

## See Also

- **Anomaly Reporting**

  The anomaly messages the check produces, what the numbers mean, Source Records highlighting, and Custom Anomaly Description.

  [Anomaly Reporting](https://userguide.qualytics.io/data-quality-checks/not-null/deep-dive/anomaly-reporting/)
- **Examples**

  Three production scenarios with sample data, anomaly messages, and the SQL equivalent of what the check evaluates.

  [Examples](https://userguide.qualytics.io/data-quality-checks/not-null/deep-dive/examples/)
- **Best Practices**

  Guidelines for choosing mandatory fields, handling empty strings, pairing rules, and keeping the signal clean.

  [Best Practices](https://userguide.qualytics.io/data-quality-checks/not-null/deep-dive/best-practices/)
- **Permissions**

  The team permission each action needs: view, create, edit, archive, restore, and delete.

  [Permissions](https://userguide.qualytics.io/data-quality-checks/not-null/deep-dive/permissions/)

*Last updated: September 22, 2026.*
