How Before Date Time Checks Work

Definition

Asserts that every value in a Date or Timestamp field is strictly earlier than a chosen cutoff date and time.

Overview

The Before Date Time rule defines an upper time boundary on a single date or timestamp field. Each row in the target container must hold a value that is less than the cutoff; any value equal to or later than the cutoff fires an anomaly. The comparison is strict (<), not inclusive (<=), and the cutoff is stored as a UTC instant.

Typical use cases:

• Validate historical data cutoffs, such as an archive that must not contain rows past a migration date.
• Ensure a date precedes a business boundary, such as a creation date before an effective date or a period close.
• Catch implausible future-dated values, such as a birth date entered as today.

Field Scope

Single: The rule evaluates exactly one field per check.

Accepted Types

General Properties

The filter allows you to define a subset of data upon which the rule will operate.

It requires a valid Spark SQL expression that determines the criteria rows in the DataFrame should meet. This means the expression specifies which rows the DataFrame should include based on those criteria. Since it's applied directly to the Spark DataFrame, traditional SQL constructs like WHERE clauses are not supported.

O_TOTALPRICE > 1000

C_MKTSEGMENT = 'BUILDING'

WHERE O_TOTALPRICE > 1000

WHERE C_MKTSEGMENT = 'BUILDING'

O_ORDERPRIORITY = '1-URGENT' AND O_ORDERSTATUS = 'O'

(L_SHIPDATE = '1998-09-02' OR L_RECEIPTDATE = '1998-09-01') AND L_RETURNFLAG = 'R'

WHERE O_ORDERPRIORITY = '1-URGENT' AND O_ORDERSTATUS = 'O'

O_TOTALPRICE > 1000, O_ORDERSTATUS = 'O'

RIGHT(
    O_ORDERPRIORITY,
    LENGTH(O_ORDERPRIORITY) - INSTR('-', O_ORDERPRIORITY)
) = 'URGENT'

LEVENSHTEIN(C_NAME, 'Supplier#000000001') < 7

RIGHT(
    O_ORDERPRIORITY,
    LENGTH(O_ORDERPRIORITY) - CHARINDEX('-', O_ORDERPRIORITY)
) = 'URGENT'

EDITDISTANCE(C_NAME, 'Supplier#000000001') < 7

O_ORDERSTATUS IN (
    SELECT DISTINCT O_ORDERSTATUS
    FROM {{_qualytics_self}}
    WHERE O_TOTALPRICE > 1000
)

O_ORDERSTATUS IN (
    SELECT DISTINCT O_ORDERSTATUS
    FROM ORDERS
    WHERE O_TOTALPRICE > 1000
)

Specific Properties

Before Date Time has one rule-specific property:

Anomaly Types

Evaluation Flow

Every Before Date Time check follows the same four-step evaluation flow:

1. Apply the filter clause. If the check has a filter set, only rows matching the filter expression continue to the next step. Rows outside the filter are ignored and cannot contribute to the violation count.
2. Read the field value as a timestamp. The platform interprets each row's value as a timestamp for comparison; values that cannot be interpreted are treated as failing the comparison (see below).
3. Compare against the cutoff. The cutoff (stored as a UTC instant) is compared to the field value using strict less-than (<). A row passes only when the field value is strictly earlier than the cutoff.
4. Apply coverage. At 100% coverage, any failing row causes the check to fail. Below 100% coverage, the check fails only when the passing fraction drops below the threshold (see Coverage and Tolerance).

The comparison is strict, so a row whose field value equals the cutoff is flagged. To get inclusive semantics, set the cutoff one second (for timestamps) or one day (for dates) later than the boundary you want to allow.

NULL Handling

The check passes NULL values: a row with NULL in the evaluated field is not counted as a violation. This is intentional. Before Date Time only asserts that present values fall before the cutoff and does not enforce mandatory presence on the field.

If the field must also be populated, pair Before Date Time with a Not Null check on the same field. Together they enforce "the field must be present AND must be earlier than the cutoff".

Values that cannot be interpreted as a timestamp are flagged as violations. Because the field picker accepts only Date and Timestamp columns, this rarely happens in practice.

Cutoff and Time Zone Behavior

The cutoff is stored as a UTC instant. When you set the cutoff to 2025-12-01 06:15:00 in the UI, the platform records it as 2025-12-01T06:15:00Z and uses that exact instant for every evaluation.

The platform evaluates the comparison in UTC:

• Timestamp fields stored with explicit time zone offsets are compared directly against the UTC cutoff.
• Timestamp fields without an explicit time zone are interpreted as UTC instants before the comparison.
• Date fields are interpreted as midnight UTC of that calendar day, then compared to the cutoff instant.

If your source data carries values in mixed or local time zones, normalize the field upstream with a Computed Field before running Before Date Time so the comparison is unambiguous.

The Filter Clause

The filter clause is a SQL WHERE expression applied before the evaluation. Filtered-out rows are ignored entirely (they cannot trigger a violation and are not counted in the totals).

Common uses:

• Restricting the boundary to a subset of the dataset (tenant_id = 42, status = 'active').
• Excluding known-bad legacy rows that are tracked by a separate clean-up task.
• Scoping the check to a partition (event_date >= '2025-12-01').

When a filter is set, both the Record Anomaly and the Shape Anomaly messages end with [filter: <expression>] so the evaluated scope is visible in the alert.

Coverage and Tolerance

Coverage is a fractional value between 0 and 1 that defines the minimum fraction of evaluated rows that must pass:

• 1.0 (100%, default): every row in the filtered set must pass. Any failing row causes the check to fail. This is the strictest setting.
• < 1.0: the check tolerates a fraction of rows failing the comparison. The check fails only when the fraction of passing rows drops below the threshold.

Lower coverage values are useful when a small, known fraction of pre-cutover rows is expected (for example, during a slow migration). Use coverage carefully: a 0.5% tolerance can mask a real regression that happens to fall just under the threshold.

See Also