# How Predicted By Checks Work

Source: https://userguide.qualytics.io/data-quality-checks/predicted-by/deep-dive/how-it-works/

## Definition

*Asserts that the actual value of a field falls within an expected predicted range.*

## Overview

The Predicted By rule evaluates a prediction expression for each row and compares the result against the field's actual value. The row passes when the two are within the configured **Tolerance** of each other, in either direction.

In other words, the check tests `|predicted - actual| <= tolerance`. The expression supplies the prediction, the tolerance supplies the band around it, and the rule reports the rows that fall outside.

What makes it different from a fixed comparison is that the expected value is computed per row. A column that should track another column, follow a formula, or stay close to a derived figure has no single correct value, only a correct relationship, and this rule states that relationship directly.

Typical use cases:

- Confirm a derived column still matches the formula that produced it.
- Validate a forecast against what actually happened, within an accepted margin.
- Monitor a metric that should track another one closely.

## Field Scope

**Single:** The rule evaluates one field per check. It accepts `Integral`, `Fractional`, `Date`, and `Timestamp` fields.

**Accepted Types**

| Type | Supported |
| --- | --- |
| `Integral` | Yes |
| `Fractional` | Yes |
| `Date` | Yes |
| `Timestamp` | Yes |

## General Properties

**Tab: Details**

| Name | Supported |
| --- | --- |
| Filter  Allows the targeting of specific data based on conditions | Supported |
| Coverage Customization  Allows adjusting the percentage of records that must meet the rule's conditions | Supported |

**Tab: Filter Guide**

The filter allows you to define a subset of data upon which the rule will operate.

It requires a valid **Spark SQL** expression that determines the criteria rows in the DataFrame should meet. This means the expression specifies which rows the DataFrame should include based on those criteria. Since it's applied directly to the Spark DataFrame, traditional SQL constructs like WHERE clauses **are not** supported.

> **Example: Examples**
>
> **Direct Conditions**
>
> Simply specify the condition you want to be met.
>
> > **Success: Correct usage" collapsible="true**
> >
> > ```sql
> > O_TOTALPRICE > 1000
> > ```
> >
> > ```sql
> > C_MKTSEGMENT = 'BUILDING'
> > ```
>
> > **Failure: Incorrect usage" collapsible="true**
> >
> > ```sql
> > WHERE O_TOTALPRICE > 1000
> > ```
> >
> > ```sql
> > WHERE C_MKTSEGMENT = 'BUILDING'
> > ```
>
> **Combining Conditions**
>
> Combine multiple conditions using logical operators like `AND` and `OR`.
>
> > **Success: Correct usage" collapsible="true**
> >
> > ```sql
> > O_ORDERPRIORITY = '1-URGENT' AND O_ORDERSTATUS = 'O'
> > ```
> >
> > ```sql
> > (L_SHIPDATE = '1998-09-02' OR L_RECEIPTDATE = '1998-09-01') AND L_RETURNFLAG = 'R'
> > ```
>
> > **Failure: Incorrect usage" collapsible="true**
> >
> > ```sql
> > WHERE O_ORDERPRIORITY = '1-URGENT' AND O_ORDERSTATUS = 'O'
> > ```
> >
> > ```sql
> > O_TOTALPRICE > 1000, O_ORDERSTATUS = 'O'
> > ```
>
> **Utilizing Functions**
>
> Leverage Spark SQL functions to refine and enhance your conditions.
>
> > **Success: Correct usage" collapsible="true**
> >
> > ```sql
> > RIGHT(
> >     O_ORDERPRIORITY,
> >     LENGTH(O_ORDERPRIORITY) - INSTR('-', O_ORDERPRIORITY)
> > ) = 'URGENT'
> > ```
> >
> > ```sql
> > LEVENSHTEIN(C_NAME, 'Supplier#000000001') < 7
> > ```
>
> > **Failure: Incorrect usage" collapsible="true**
> >
> > ```sql
> > RIGHT(
> >     O_ORDERPRIORITY,
> >     LENGTH(O_ORDERPRIORITY) - CHARINDEX('-', O_ORDERPRIORITY)
> > ) = 'URGENT'
> > ```
> >
> > ```sql
> > EDITDISTANCE(C_NAME, 'Supplier#000000001') < 7
> > ```
>
> **Using scan-time variables**
>
> To refer to the current dataframe being analyzed, use the reserved dynamic variable `{{_qualytics_self}}`.
>
> > **Success: Correct usage" collapsible="true**
> >
> > ```sql
> > O_ORDERSTATUS IN (
> >     SELECT DISTINCT O_ORDERSTATUS
> >     FROM {{_qualytics_self}}
> >     WHERE O_TOTALPRICE > 1000
> > )
> > ```
>
> > **Failure: Incorrect usage" collapsible="true**
> >
> > ```sql
> > O_ORDERSTATUS IN (
> >     SELECT DISTINCT O_ORDERSTATUS
> >     FROM ORDERS
> >     WHERE O_TOTALPRICE > 1000
> > )
> > ```
>
> While subqueries can be useful, their application within filters in our context has limitations. For example, directly referencing other containers or the broader target container in such subqueries **is not supported**. Attempting to do so will result in an error.
>
> > **Warning: Important Note on {{_qualytics_self}}**
> >
> > The `{{_qualytics_self}}` keyword refers to the dataframe that's currently under examination. In the context of a full scan, this variable represents the entire target container. However, during incremental scans, it only reflects a subset of the target container, capturing just the incremental data. It's crucial to recognize that in such scenarios, using `{{_qualytics_self}}` may not encompass all entries from the target container.

## Specific Properties

Predicted By has the following rule-specific properties:

| Name | Description |
| --- | --- |
| Expression | The SQL expression that predicts the field's value. It is evaluated on every row that passes the filter, and its result is compared against the selected field. |
| Tolerance | The absolute margin allowed between the prediction and the field's value. A row passes while the difference stays within it. |

## Anomaly Types

| Type | Supported |
| --- | --- |
| Record Flag inconsistencies at the row level | Supported |
| Shape Flag inconsistencies in the overall patterns and distributions of a field | Supported |

## Evaluation Flow

Every Predicted By check follows the same five-step evaluation flow:

1. **Apply the filter clause.** If the check has a `filter` set, only rows matching the filter expression continue to the next step. Rows outside the filter are ignored and cannot contribute to the violation count.
2. **Resolve any check variables.** Placeholders in the expression are replaced with their values before it runs.
3. **Evaluate the prediction.** The expression runs against the current row, producing the value the field is expected to hold.
4. **Compare it against the actual value.** The row passes when the difference between the prediction and the field's value is at most **Tolerance**, in either direction. The band is inclusive: a difference exactly equal to the tolerance passes.
5. **Apply coverage.** At 100% coverage, any failing row causes the check to fail. Below 100% coverage, the check fails only when the passing fraction drops below the threshold (see [Coverage and Tolerance](https://userguide.qualytics.io/data-quality-checks/predicted-by/deep-dive/how-it-works/#coverage-and-tolerance)).

## The Tolerance Is a Band, Not a Direction

The comparison uses the absolute difference, so the tolerance applies equally above and below the prediction. A tolerance of `2` accepts anything from `predicted - 2` to `predicted + 2`.

The rule cannot express a one-sided allowance such as "the actual value may exceed the prediction but never fall short". For that, compare the two sides with [Satisfies Expression](https://userguide.qualytics.io/data-quality-checks/satisfies-expression/introduction/), which lets you write the inequality you actually want.

## The Tolerance Is Absolute, Not a Percentage

**Tolerance** is a fixed number in the field's own units. It does not scale with the size of the prediction, so a tolerance sized for values around 100 will be far too tight on values around 100,000 and far too loose the other way.

When a column spans several orders of magnitude, either scope the check to a comparable slice with a filter, or express the relative comparison yourself in a [Satisfies Expression](https://userguide.qualytics.io/data-quality-checks/satisfies-expression/introduction/) check.

## NULL Values Pass

When the expression returns `NULL` (for example, because a column it references is empty) or the selected field itself is `NULL`, the comparison has nothing to measure and the row **passes**. It counts toward the evaluated total but is never reported.

If a missing value should be reported, pair this check with [Not Null](https://userguide.qualytics.io/data-quality-checks/not-null/introduction/) on the selected field and on each column the expression depends on.

## Date and Timestamp Targets Have a Limitation

Although `Date` and `Timestamp` fields are available in the form, the **Expression** can transform only the predicted value. The selected field remains a date or timestamp, so converting the prediction to Unix time does not create a valid numeric comparison.

When both sides must be normalized to seconds, milliseconds, or another numeric unit, use [Satisfies Expression](https://userguide.qualytics.io/data-quality-checks/satisfies-expression/introduction/) to write the complete comparison explicitly.

## The Expression Runs on Every Row

The prediction is computed once per row that passes the filter, so a heavy expression, especially one containing a subquery, makes the check considerably slower than a plain field comparison.

Narrow the rows with a **Filter Clause** before reaching for a more complicated expression: the filter runs first, so scoping the rows scopes the work.

## The Filter Clause

The filter clause is a SQL `WHERE` expression applied **before** the evaluation. Filtered-out rows are ignored entirely (they cannot trigger a violation and are not counted in the totals).

When a filter is set, the message names it at the end of its first sentence, as `(applied to records where <expression>)`. Record and Shape Anomaly messages both carry it.

## Coverage and Tolerance

Coverage is a fractional value between `0` and `1` that defines the minimum fraction of evaluated rows that must pass:

- **`1.0` (100%, default):** every row in the filtered set must pass. Any failing row causes the check to fail. This is the strictest setting.
- **`< 1.0`:** the check tolerates a fraction of rows failing. The check fails only when the fraction of passing rows drops below the threshold.

Lower coverage values are useful when a small, known fraction of failures is expected. Use coverage carefully: a 0.5% tolerance can mask a real regression that happens to fall just under the threshold.

## See Also

- **Anomaly Reporting**

  The anomaly messages the check produces, what the numbers mean, Source Records behavior, and Custom Anomaly Description.

  [Anomaly Reporting](https://userguide.qualytics.io/data-quality-checks/predicted-by/deep-dive/anomaly-reporting/)
- **Examples**

  Three production scenarios with sample data, anomaly messages, and the SQL equivalent of what the check evaluates.

  [Examples](https://userguide.qualytics.io/data-quality-checks/predicted-by/deep-dive/examples/)
- **Best Practices**

  Guidelines for writing the prediction, sizing the tolerance, and choosing between this rule and a simpler one.

  [Best Practices](https://userguide.qualytics.io/data-quality-checks/predicted-by/deep-dive/best-practices/)
- **Permissions**

  The team permission each action needs: view, create, edit, archive, restore, and delete.

  [Permissions](https://userguide.qualytics.io/data-quality-checks/predicted-by/deep-dive/permissions/)

*Last updated: October 19, 2026.*
