Not Null Check Examples

The situation: Every row in the customers table must carry an identifier and a creation timestamp. Rows missing either one come from a partial import that ran without the header mapping, and they break joins and cohort reporting downstream.

Check configuration

Payload

{
    "description": "Customer identifier and creation timestamp must always be populated.",
    "rule": "notNull",
    "fields": ["c_custkey", "created_at"],
    "container_id": 145,
    "coverage": 1,
    "filter": null,
    "properties": {},
    "tags": ["completeness", "customers"],
    "additional_metadata": {"jira": "DATA-9001"},
    "anomaly_message_field": null,
    "template_id": null,
    "status": "Active",
    "owner_id": 7,
    "default_anomaly_assignee_id": 12
}

Sample Data

What gets flagged

Row 1002 is missing its creation timestamp and the third row is missing its identifier. Both fail, because the two fields are asserted together. Coverage is 100%, so each failing row is reported as a Record Anomaly, with the empty cell highlighted.

Flowchart

graph TD
    A["No filter, evaluate all rows"] --> B["Read c_custkey and created_at"]
    B --> C{"Are both fields populated?"}
    C -->|Yes| D["Row passes"]
    C -->|No| E["Flag row.<br/>Record Anomaly per failing row."]

Equivalent SQL

-- Rows the Not Null check would flag.
SELECT c.*
FROM customers c
WHERE c.c_custkey IS NULL
   OR c.created_at IS NULL;

The situation: Payments in the completed state must carry a settlement reference. Pending and failed payments legitimately have none, so they are excluded with a filter.

Check configuration

Payload

{
    "description": "Completed payments must carry a settlement reference.",
    "rule": "notNull",
    "fields": ["settlement_ref"],
    "container_id": 512,
    "coverage": 1,
    "filter": "status = 'completed'",
    "properties": {},
    "tags": ["payments", "completeness"],
    "additional_metadata": {"jira": "DATA-9044"},
    "anomaly_message_field": null,
    "template_id": null,
    "status": "Active",
    "owner_id": 7,
    "default_anomaly_assignee_id": 15
}

Sample Data (filtered to status = 'completed')

Why the filter matters

The filter runs before the evaluation, so pending and failed payments, which have no reference yet, are never tested. Only completed payments are evaluated.

What gets flagged

P-002 is marked completed but carries no settlement reference, which means the settlement callback never landed. Coverage is 100%, so the failure is reported as a Record Anomaly, and the message ends with the filter that scoped the evaluation.

Flowchart

graph TD
    A["Apply filter: status = 'completed'"] --> B["Read settlement_ref"]
    B --> C{"Is settlement_ref populated?"}
    C -->|Yes| D["Row passes"]
    C -->|No| E["Flag row.<br/>Anomaly message ends with<br/>[filter: status = 'completed']"]

Equivalent SQL

-- Rows the check would flag among completed payments.
SELECT p.*
FROM payments p
WHERE p.status = 'completed'
  AND p.settlement_ref IS NULL;

The situation: A newly added region_code column is being backfilled across historical rows. Until the job finishes, a small fraction of rows is expected to be empty, so the check tolerates up to 0.5% failures while still catching a backfill that stalls.

Check configuration

Payload

{
    "description": "Region code must be populated on every account.",
    "rule": "notNull",
    "fields": ["region_code"],
    "container_id": 733,
    "coverage": 0.995,
    "filter": null,
    "properties": {},
    "tags": ["completeness", "backfill"],
    "additional_metadata": {"jira": "DATA-9088"},
    "anomaly_message_field": null,
    "template_id": null,
    "status": "Active",
    "owner_id": 7,
    "default_anomaly_assignee_id": 41
}

Sample Data

What gets flagged

Accounts A-02 and A-03 have not been reached by the backfill yet. Only 50% of the rows pass, which is below the 99.5% coverage threshold, so the check reports a single Shape Anomaly for the dataset with a sample of the offending rows. Coverage below 100% does not produce Record Anomalies.

Flowchart

graph TD
    A["No filter, evaluate all rows"] --> B["Read region_code"]
    B --> C{"Is region_code populated?"}
    C -->|Yes| D["Row passes"]
    C -->|No| E["Flag row.<br/>Passing rate falls below the 99.5%<br/>coverage, so one Shape Anomaly is reported."]

Equivalent SQL

-- Rows the check would flag.
SELECT a.*
FROM accounts a
WHERE a.region_code IS NULL;