Skip to content

Anomaly Analysis

The Anomaly Details modal provides specific details about the Anomaly, including source record and checks that failed assertions, along with ability to take Notes and provide feedback to ML methods through updating Status of the anomaly.

Screenshot Screenshot

Users have the ability to provide feedback to the ML methods through Supervised Learning. Specifically, the user can Acknowledge, Resolve or Invalidate an anomaly. Each of these actions will change the tolerances of the data quality checks behind the anomaly.

Anomaly Header

Screenshot Screenshot

  1. The # ID of the anomaly: 32102
  2. The Datastore name: SAP Financial Exports
  3. The Location where this data is stored: s3a://qualytics-demo-data/orders/demo/customer-20220223.csv
  4. The Record Type of the anomaly: Shape
  5. The Tags are labels that serve the purpose of grouping anomalies and driving downstream workflows: Migration Low
  6. The Status of the selected Anomaly: Users can edit the status to Acknowledged, Resolved or Invalid.
  7. The Detected at of the anomaly creation time: 1 year ago

Failed Checks:

Screenshot Screenshot

Field Screenshot Screenshot

All the fields of that specific anomaly found: company_name

Rule

Rule type that failed the assertion(s)

You can check all the Rule types here.

Violation

Details of how the anomaly failed the assertion(s) of rule(s)

Coverage Screenshot Screenshot

The expected tolerance of the rule

Type Screenshot Screenshot

Infered or Authored

Coverage Screenshot Screenshot

The expected tolerance of the rule

Tag

The rule type Tag Screenshot Screenshot

Source Records

Show in a tabular view all the records and fields and highlight records with anomaly data based on the rule type.

Screenshot Screenshot

Comparison Source Records

Anomalies generated by data quality checks of rule type Is Replica Of with Row Identifiers configured, show a tailored source record visualization that provides more comparison information about each row retrieved accounting for row key matching.

Screenshot Screenshot

Each field is divided between left or right, which represents respectively the target and reference table/file. When the right value differs the left value the cell is highlighted to indicate an anomalous value.

The _qualytics_diff column informs the status of each row, which can be one of the following:

Row Status Description
added Row missing in the left side but found in the right side.
removed Row found in the left side but missing in the right side.
changed Row found on both sides but there is at least one field value differing.

Infered Check Details

  • Clicking into a Rule will highlight its details and enables users to edit the rule as well:

Screenshot Screenshot

  • An anomaly can be archived pressing the Check Actions menu and Archive.

Screenshot Screenshot

  • If you expand the section Advanced Options you can add a Filter clause and also change the Coverage percentage for that anomaly.

Note

The Filter clause is a WHERE statement against your table. For example:
price != 33 or price > discount + 20

Info

You can create a computed table and use multiple fields from different tables in a filter clause

Suggested Remediation

Qualytics can also provide you a suggested value

ScreenshotScreenshotScreenshotScreenshot

Download Source Records

Download the source records for further analysis or external use.

Screenshot Screenshot


Last update: May 30, 2024