Less Than Field Check Examples

The situation: A contract cannot end before it starts. Rows where start_date is later than end_date come from a migration that swapped the two columns. A single-day contract is valid, so equal dates must pass and Inclusive is on.

Check configuration

Payload

{
    "description": "Start date must be earlier than the end date.",
    "rule": "lessThanField",
    "fields": ["start_date"],
    "container_id": 145,
    "coverage": 1,
    "filter": null,
    "properties": {"field_name": "end_date", "inclusive": true},
    "tags": ["contracts", "integrity"],
    "additional_metadata": {"jira": "DATA-8901"},
    "anomaly_message_field": null,
    "template_id": null,
    "status": "Active",
    "owner_id": 7,
    "default_anomaly_assignee_id": 12
}

Sample Data

What gets flagged

Contract C-002 starts after it ends, the signature of swapped columns. C-003 is a single-day contract and passes, because Inclusive is on. C-004 has no end date yet, so there is nothing to compare and the row passes. Coverage is 100%, so the failing row is reported as a Record Anomaly.

Flowchart

graph TD
    A["No filter, evaluate all rows"] --> B["Read start_date and end_date"]
    B --> C{"Is either value NULL?"}
    C -->|Yes| D["Row passes"]
    C -->|No| E{"Is start_date <= end_date?"}
    E -->|Yes| D
    E -->|No| G["Flag row.<br/>Record Anomaly per failing row."]

Equivalent SQL

-- Rows the Less Than Field check would flag.
SELECT c.*
FROM contracts c
WHERE c.start_date IS NOT NULL
  AND c.end_date IS NOT NULL
  AND c.start_date > c.end_date;

The situation: Metered accounts must not consume more than their provisioned quota. Unmetered accounts have no quota and are excluded with a filter. Consuming exactly the quota is allowed, so Inclusive is on.

Check configuration

Payload

{
    "description": "Metered usage must stay within the provisioned quota.",
    "rule": "lessThanField",
    "fields": ["used_units"],
    "container_id": 512,
    "coverage": 1,
    "filter": "metered = true",
    "properties": {"field_name": "quota_units", "inclusive": true},
    "tags": ["billing", "integrity"],
    "additional_metadata": {"jira": "DATA-8944"},
    "anomaly_message_field": null,
    "template_id": null,
    "status": "Active",
    "owner_id": 7,
    "default_anomaly_assignee_id": 29
}

Sample Data (filtered to metered = true)

Why the filter matters

The filter runs before the comparison, so unmetered accounts, which have no quota to respect, are never evaluated.

What gets flagged

A-02 consumed well beyond its quota, which the metering job should have blocked. A-03 used exactly its quota and passes, because Inclusive is on. Coverage is 100%, so the failure is reported as a Record Anomaly, and the message ends with the filter that scoped the evaluation.

Flowchart

graph TD
    A["Apply filter: metered = true"] --> B["Read used_units and quota_units"]
    B --> C{"Is either value NULL?"}
    C -->|Yes| D["Row passes"]
    C -->|No| E{"Is used_units <= quota_units?"}
    E -->|Yes| D
    E -->|No| F["Flag row.<br/>Anomaly message ends with<br/>[filter: metered = true]"]

Equivalent SQL

-- Rows the check would flag among metered accounts.
SELECT a.*
FROM accounts a
WHERE a.metered = true
  AND a.used_units IS NOT NULL
  AND a.quota_units IS NOT NULL
  AND a.used_units > a.quota_units;

The situation: An event must be received no later than it was created, allowing for the clock skew between the producer and the collector. A duration tolerance of 5 seconds absorbs that skew. A collector fix is rolling out, so the check tolerates up to 0.5% failures.

Check configuration

Payload

{
    "description": "Events must be created no later than they are received, allowing for clock skew.",
    "rule": "lessThanField",
    "fields": ["created_at"],
    "container_id": 733,
    "coverage": 0.995,
    "filter": null,
    "properties": {"field_name": "received_at", "inclusive": true, "duration_comparator": {"millis": 5000}},
    "tags": ["events", "integrity"],
    "additional_metadata": {"jira": "DATA-8988"},
    "anomaly_message_field": null,
    "template_id": null,
    "status": "Active",
    "owner_id": 7,
    "default_anomaly_assignee_id": 47
}

Sample Data

What gets flagged

E-02 was created three seconds after it was received, which falls inside the 5-second skew tolerance and passes. E-03 and E-04 are ahead by more than an hour and a day, far beyond any clock skew, and fail. Only 50% of the rows pass, which is below the 99.5% coverage threshold, so the check reports a single Shape Anomaly for the dataset with a sample of the offending rows. Coverage below 100% does not produce Record Anomalies.

Flowchart

graph TD
    A["No filter, evaluate all rows"] --> B["Read created_at and received_at"]
    B --> C{"Are both values NULL?"}
    C -->|Yes| D["Row passes"]
    C -->|No| H{"Is exactly one value NULL?"}
    H -->|Yes| G
    H -->|No| E{"Is created_at, within the<br/>5-second tolerance, <= received_at?"}
    E -->|Yes| D
    E -->|No| G["Flag row.<br/>Passing rate falls below the 99.5%<br/>coverage, so one Shape Anomaly is reported."]

Equivalent SQL

-- Rows the check would flag, allowing 5 seconds of skew.
-- With a comparator configured, a row where exactly one side is NULL also fails.
SELECT e.*
FROM events e
WHERE (e.created_at IS NULL) <> (e.received_at IS NULL)
   OR (e.created_at IS NOT NULL
       AND e.received_at IS NOT NULL
       AND e.created_at - INTERVAL 5 SECONDS > e.received_at);