Anomaly Detection Process

Qualytics records anomalies when a Scan evaluates active quality checks and finds failures. These results provide evidence of potential data quality issues within the configured Scan scope. Data stewards use the failed checks, affected records or structures, and business context to decide what the anomaly means and how to resolve it.

1. Create a Datastore and Connection

Create a source datastore and configure its connection so Qualytics can access the selected database or file storage for Sync, Profile, and Scan operations. The connection and datastore settings determine which data Qualytics can evaluate.

2. Sync Operation

The Sync operation detects new, changed, or removed containers and fields by comparing the current datastore state with the existing inventory in Qualytics. Sync updates that inventory but does not evaluate data against quality checks.

3. Profile Operation

The Profile operation calculates statistics and observed patterns for the selected containers and fields. Qualytics uses those results to create or refresh most AI Managed checks. Volumetric and Freshness checks are maintained from Observability measurements. Data stewards review the inferred coverage and add business expectations that cannot be determined from observed data alone.

4. Create Authored Checks

Authored Checks are data quality checks that users define in the Qualytics interface or through the API. Use them to express business expectations, regulatory requirements, and other conditions that observed patterns alone cannot supply. Authored checks include guided templates and SQL-based options for specialized business logic, along with settings such as scope, filters, coverage, tags, and ownership.

5. Scan Operation

The Scan operation evaluates active AI Managed and Authored checks within the selected scope. Failed checks can produce Record or Shape Anomalies and store the related evidence in the configured enrichment datastore for investigation. A Scan result reflects only the checks and data included in that operation.

6. Anomaly Analysis

An anomaly records one or more check failures for a record, data structure, or summarized set of violations. It is evidence of a potential issue, not proof that the data is incorrect or that every possible condition was checked. Review the failed-check messages, affected source records or structure, description, history, and business impact before changing the anomaly's status or documenting a resolution.

When every failed check linked to an anomaly is AI Managed, the anomaly is marked with an AI badge. This badge identifies the source of the checks; it does not replace human review.