Skip to content

Read Strategies

A Scan operation reads data using one of two strategies: Incremental or Full. The choice affects how much data is processed, which records become candidates for anomaly detection, and whether previously open anomalies can be automatically resolved at the end of the scan.

Incremental

Incremental scans process only the new or updated records since the previous scan operation. They rely on an incremental identifier declared on the container (a timestamp column or a monotonically increasing batch column). Records whose identifier value is less than or equal to the highest value seen in the prior scan are skipped.

First incremental scan

The first incremental scan against a container behaves like a Full scan, since there is no prior baseline. After it completes, Qualytics stores the highest identifier value and uses it as the starting threshold for the next run.

Incremental scans are designed for routine pipelines where only new or changed records need to be re-checked. They save compute and finish faster than a Full scan on the same dataset, but they cannot re-evaluate records they did not read, including records that previously caused open anomalies and that may have since been corrected upstream.

Warning

If a selected container does not have an incremental identifier configured, Qualytics falls back to a Full read for that container even when the Read Strategy is set to Incremental.

Full

Full scans process every record in each selected container, regardless of any prior scan. They are the only way to verify the entire dataset against every check in a single operation, and they are the only strategy under which Auto-Resolve is evaluated.

Full scans are well-suited for periodic deep checks, for re-baselining a container after upstream changes, or for any scenario where Incremental cannot be used (no incremental identifier, file pattern without ordering guarantees, etc.).

Auto-Resolve on Full Scans

When a Full scan completes successfully with Auto Resolve Anomalies enabled, Qualytics reconciles previously open anomalies against the work this scan actually did and resolves the ones that no longer apply.

When it runs

Auto-Resolve runs exactly once per scan, at the end, and only when all of the following are true:

  • the operation is a Scan, not a Profile or Sync;
  • the read strategy is Full (Incremental scans are excluded by design);
  • the operation finished with success (Failure and Aborted operations do not trigger Auto-Resolve);
  • the Auto Resolve Anomalies toggle was enabled when the scan started.

What gets resolved

The candidates are anomalies that are currently in an open status: Active or Acknowledged. Anomalies that are already in Resolved, Invalid, Duplicate, or Discarded are not re-evaluated and are not touched.

For each candidate anomaly, Qualytics resolves it only when both conditions hold:

  1. Every check that originally flagged the anomaly ran successfully in this scan. If even one of those checks was not asserted (for example, the container was excluded from this run, or the check was archived), the anomaly is left as-is.
  2. None of those same checks raised the same issue against the same fingerprint in this scan. If the new run produced a new anomaly for the same field combination, the previous anomaly stays open and the new one is recorded independently.

When both conditions hold, the anomaly's status is set to Resolved and the scan that resolved it is recorded as part of the anomaly's history.

What the user sees

The resolution is reflected on three surfaces:

  • Operation summary: the Anomalies Auto-Resolved count appears alongside the existing Anomalies Identified count once the scan finishes. See Scan — Success.
  • Scan Results modal: a dedicated Auto-Resolved tab lists the previously open anomalies this scan resolved. See the Scan — Success page (Results tab section).
  • Anomaly history: each auto-resolved anomaly records an entry attributed to Qualytics, referencing the resolving scan.

Why Incremental scans never auto-resolve

Auto-Resolve depends on the scan having read the records that originally flagged the anomaly. An Incremental scan, by definition, only reads records that arrived since the last scan, so it cannot confirm whether older records still violate a check. Allowing Auto-Resolve under Incremental would risk resolving anomalies on records that were never re-read in the current run. For this reason the toggle is hidden in the Scan Settings step when Incremental is selected, and any value sent through the API is forced off before the scan starts.

Permissions

Auto-Resolve does not introduce a new permission. Any user with the Editor team permission on the target datastore (the same permission required to run or schedule a scan) can enable or disable the toggle. See Permissions.