Skip to content

How Is Replica Of Checks Work

Deprecated: use Data Diff instead

The Is Replica Of rule is no longer maintained. Data Diff replaces it with the same row-by-row comparison plus change-type filtering. This page is kept as a reference for checks that already exist; new checks should use Data Diff.

Definition

Asserts that the dataset formed by the targeted fields is replicated by the referred fields on a reference container.

Overview

The Is Replica Of rule compares two containers row by row and reports when the target is not a faithful replica of the reference. It shares its comparison engine and most of its configuration with Data Diff: the same Row Identifiers, the same Passthrough Fields, the same per-type Comparators.

The rule is deprecated. Data Diff is the maintained successor and adds diff_change_types, which restricts anomalies to a chosen subset of added, removed, and changed. Existing Is Replica Of checks keep running, but new work should use Data Diff.

Typical use cases:

  • Read an existing check that was created before Data Diff was available.
  • Understand what an Is Replica Of anomaly means while you migrate it.
  • Compare the configuration side by side with Data Diff before switching.

Field Scope

Multiple: The rule compares one or more fields between the target container and the reference container.

General Properties

Name Supported
Filter
Allows the targeting of specific data based on conditions
Coverage Customization
Allows adjusting the percentage of records that must meet the rule's conditions

The filter allows you to define a subset of data upon which the rule will operate.

It requires a valid Spark SQL expression that determines the criteria rows in the DataFrame should meet. This means the expression specifies which rows the DataFrame should include based on those criteria. Since it's applied directly to the Spark DataFrame, traditional SQL constructs like WHERE clauses are not supported.

Examples

Direct Conditions

Simply specify the condition you want to be met.

Correct usage" collapsible="true 
O_TOTALPRICE > 1000

C_MKTSEGMENT = 'BUILDING'
Incorrect usage" collapsible="true 
WHERE O_TOTALPRICE > 1000

WHERE C_MKTSEGMENT = 'BUILDING'

Combining Conditions

Combine multiple conditions using logical operators like AND and OR.

Correct usage" collapsible="true 
O_ORDERPRIORITY = '1-URGENT' AND O_ORDERSTATUS = 'O'

(L_SHIPDATE = '1998-09-02' OR L_RECEIPTDATE = '1998-09-01') AND L_RETURNFLAG = 'R'
Incorrect usage" collapsible="true 
WHERE O_ORDERPRIORITY = '1-URGENT' AND O_ORDERSTATUS = 'O'

O_TOTALPRICE > 1000, O_ORDERSTATUS = 'O'

Utilizing Functions

Leverage Spark SQL functions to refine and enhance your conditions.

Correct usage" collapsible="true 
RIGHT(
    O_ORDERPRIORITY,
    LENGTH(O_ORDERPRIORITY) - INSTR('-', O_ORDERPRIORITY)
) = 'URGENT'

LEVENSHTEIN(C_NAME, 'Supplier#000000001') < 7
Incorrect usage" collapsible="true 
RIGHT(
    O_ORDERPRIORITY,
    LENGTH(O_ORDERPRIORITY) - CHARINDEX('-', O_ORDERPRIORITY)
) = 'URGENT'

EDITDISTANCE(C_NAME, 'Supplier#000000001') < 7

Using scan-time variables

To refer to the current dataframe being analyzed, use the reserved dynamic variable {{_qualytics_self}}.

Correct usage" collapsible="true 
O_ORDERSTATUS IN (
    SELECT DISTINCT O_ORDERSTATUS
    FROM {{_qualytics_self}}
    WHERE O_TOTALPRICE > 1000
)
Incorrect usage" collapsible="true 
O_ORDERSTATUS IN (
    SELECT DISTINCT O_ORDERSTATUS
    FROM ORDERS
    WHERE O_TOTALPRICE > 1000
)

While subqueries can be useful, their application within filters in our context has limitations. For example, directly referencing other containers or the broader target container in such subqueries is not supported. Attempting to do so will result in an error.

Important Note on {{_qualytics_self}}

The {{_qualytics_self}} keyword refers to the dataframe that's currently under examination. In the context of a full scan, this variable represents the entire target container. However, during incremental scans, it only reflects a subset of the target container, capturing just the incremental data. It's crucial to recognize that in such scenarios, using {{_qualytics_self}} may not encompass all entries from the target container.

Specific Properties

Is Replica Of carries the same properties as Data Diff, minus the change-type filter:

Name Description
Row Identifiers
 The field, or combination of fields, used to pair each target row with its reference row. Without identifiers the check reports a set-level difference and cannot classify rows.
Passthrough Fields
 Extra fields shown alongside the compared ones for context. They are displayed but never compared.
Datastore
 The datastore holding the reference container. Leave it empty to use the target's own datastore.
Container
 The reference container compared against the target.
Filter Clause
 A SQL WHERE expression that scopes the reference container before the comparison.
Numeric
 Tolerance applied when comparing numeric fields.
Duration
 Tolerance applied when comparing date and timestamp fields.
String
 Normalization applied when comparing string fields, such as ignoring case or surrounding whitespace.

Anomaly Types

Type Supported
Record
Flag inconsistencies at the row level
Shape
Flag inconsistencies in the overall patterns and distributions of a field

Evaluation Flow

Every Is Replica Of check follows the same four-step evaluation flow:

  1. Apply the filter clauses. The target filter scopes the target container and the reference filter scopes the reference container, before anything is compared.
  2. Pair the rows. With Row Identifiers set, each target row is matched to its reference row. Without them, the two sides are compared as sets.
  3. Compare the listed fields. Values that differ beyond the configured Comparators count as a difference.
  4. Report. When any difference remains, the check produces one Shape Anomaly for the container.

What Data Diff Adds

The two rules share the comparison engine and the configuration. Data Diff adds Diff Change Types, which restricts anomalies to a chosen subset of added, removed, and changed. That single addition is what makes a one-way flow quiet: a target that legitimately keeps receiving new rows no longer reports every one of them as a difference.

Because the configuration is otherwise the same, migrating is mostly a matter of recreating the check as Data Diff with the same fields, identifiers, and comparators.

No Coverage

The rule does not use a coverage threshold: any difference that a Comparator does not tolerate is reported. The coverage field on the API payload accepts only null or 1; any other value is rejected with 422.

NULL Handling

A row where a compared value is NULL on both sides matches. A value present on one side and NULL on the other counts as a difference, which is what makes the rule useful for catching a replica that dropped data.

The Filter Clause

The filter clause is a SQL WHERE expression applied before the evaluation. Filtered-out rows are ignored entirely.

When a filter is set, the Shape Anomaly message ends with [filter: <expression>] so the evaluated scope is visible in the alert.

See Also

  • Anomaly Reporting

    What the Shape Anomaly reports and how the differing rows are presented.

    Anomaly Reporting

  • Examples

    A production scenario showing a replica that drifted from its source.

    Examples

  • Best Practices

    Why to migrate to Data Diff, and how to keep an existing check useful until you do.

    Best Practices

  • Permissions

    The team permission each action needs: view, create, edit, archive, restore, and delete.

    Permissions