Anomaly Analysis
The Anomaly Details modal provides specific details about the Anomaly, including source record and checks that failed assertions, along with ability to take Notes and provide feedback to ML methods through updating Status of the anomaly.
Users have the ability to provide feedback to the ML methods through Supervised Learning. Specifically, the user can Acknowledge, Resolve or Invalidate an anomaly. Each of these actions will change the tolerances of the data quality checks behind the anomaly.
Anomaly Header
- The
# IDof the anomaly: 32102 - The
Datastorename: SAP Financial Exports - The
Locationwhere this data is stored: s3a://qualytics-demo-data/orders/demo/customer-20220223.csv - The
Record Typeof the anomaly: Shape - The
Tagsare labels that serve the purpose of grouping anomalies and driving downstream workflows: Migration Low - The
Statusof the selected Anomaly: Users can edit the status toAcknowledged,ResolvedorInvalid. - The
Detected atof the anomaly creation time: 1 year ago
Failed Checks:
Field 
All the fields of that specific anomaly found: company_name
Rule
Rule type that failed the assertion(s)
You can check all the Rule types here.
Violation
Details of how the anomaly failed the assertion(s) of rule(s)
Coverage 
The expected tolerance of the rule
Type 
Infered or Authored
Coverage
The expected tolerance of the rule
Tag
The rule type Tag 
Source Records
Show in a tabular view all the records and fields and highlight records with anomaly data based on the rule type.
Comparison Source Records
Anomalies generated by data quality checks of rule type Is Replica Of with Row Identifiers configured, show a tailored source record visualization that provides more comparison information about each row retrieved accounting for row key matching.
Each field is divided between left or right, which represents respectively the target and reference table/file. When the right value differs the left value the cell is highlighted to indicate an anomalous value.
The _qualytics_diff column informs the status of each row, which can be one of the following:
| Row Status | Description |
|---|---|
| added | Row missing in the left side but found in the right side. |
| removed | Row found in the left side but missing in the right side. |
| changed | Row found on both sides but there is at least one field value differing. |
Infered Check Details
- Clicking into a
Rulewill highlight its details and enables users to edit the rule as well:
- An anomaly can be archived pressing the
Check Actionsmenu andArchive.
- If you expand the section
Advanced Optionsyou can add aFilterclause and also change theCoveragepercentage for that anomaly.
Note
The Filter clause is a WHERE statement against your table. For example:
price != 33 or price > discount + 20
Info
You can create a computed table and use multiple fields from different tables in a filter clause
Suggested Remediation
Qualytics can also provide you a suggested value
Download Source Records
Download the source records for further analysis or external use.
API Payload Examples
Retrieving Anomaly by UUID or Id
Endpoint (Get)
/api/anomalies/{id} (get)
{
"id": 0,
"created": "2024-06-10T21:29:42.695Z",
"uuid": "3fa85f64-5717-4562-b3fc-2c963f66afa6",
"type": "shape",
"is_new": true,
"archived": true,
"status": "Active",
"source_enriched": true,
"datastore": {
"id": 0,
"name": "string",
"store_type": "jdbc",
"type": "athena",
"enrich_only": true,
"enrich_container_prefix": "string",
"favorite": true
},
"container": {
"id": 0,
"name": "string",
"container_type": "table",
"table_type": "table"
},
"partition": {
"name": "string",
"location": "string"
},
"weight": 0,
"global_tags": [
{
"type": "external",
"name": "string",
"color": "string",
"description": "string",
"weight_modifier": 0,
"integration": {
"id": 0,
"created": "2024-06-10T21:29:42.696Z",
"name": "string",
"type": "atlan",
"api_url": "string",
"overwrite": true,
"last_synced": "2024-06-10T21:29:42.696Z",
"status": "syncing"
}
},
{
"type": "external",
"name": "string",
"color": "string",
"description": "string",
"weight_modifier": 0,
"integration": {
"id": 0,
"created": "2024-06-10T21:29:42.696Z",
"name": "string",
"type": "atlan",
"api_url": "string",
"overwrite": true,
"last_synced": "2024-06-10T21:29:42.696Z",
"status": "syncing"
}
}
],
"anomalous_records_count": 0,
"comments": [
{
"id": 0,
"created": "2024-06-10T21:29:42.696Z",
"message": "string",
"user": {
"id": 0,
"created": "2024-06-10T21:29:42.696Z",
"user_id": "string",
"email": "string",
"name": "string",
"picture": "string",
"role": "Member",
"deleted_at": "2024-06-10T21:29:42.696Z",
"teams": [
{
"id": 0,
"name": "string",
"permission": "Read"
}
]
}
}
],
"failed_checks": [
{
"quality_check": {
"id": 0,
"created": "2024-06-10T21:29:42.696Z",
"fields": [
{
"id": 0,
"created": "2024-06-10T21:29:42.696Z",
"name": "string",
"type": "Unknown",
"completeness": 0,
"weight": 0,
"global_tags": [
{
"type": "global",
"name": "string",
"color": "string",
"description": "string",
"weight_modifier": 0
},
{
"type": "external",
"name": "string",
"color": "string",
"description": "string",
"weight_modifier": 0,
"integration": {
"id": 0,
"created": "2024-06-10T21:29:42.697Z",
"name": "string",
"type": "atlan",
"api_url": "string",
"overwrite": true,
"last_synced": "2024-06-10T21:29:42.697Z",
"status": "syncing"
}
}
],
"latest_profile_id": 0,
"quality_score": {
"total": 0,
"completeness": 0,
"coverage": 0,
"conformity": 0,
"consistency": 0,
"precision": 0,
"timeliness": 0,
"volumetrics": 0,
"accuracy": 0
}
}
],
"description": "string",
"rule_type": "afterDateTime",
"coverage": 0,
"inferred": true,
"template_locked": true,
"is_new": true,
"num_container_scans": 0,
"filter": "string",
"properties": {
"allow_other_fields": true,
"assertion": "string",
"comparison": "string",
"datetime": "2024-06-10T21:29:42.697Z",
"expression": "string",
"field_name": "string",
"field_type": "Unknown",
"id_field_names": [
"string"
],
"inclusive": true,
"inclusive_max": true,
"inclusive_min": true,
"interval_name": "Yearly",
"last_value": 0,
"list": [
"string"
],
"max": 0,
"max_size": 0,
"max_time": "2024-06-10T21:29:42.697Z",
"min": 0,
"min_size": 0,
"min_time": "2024-06-10T21:29:42.697Z",
"pattern": "string",
"ref_container_id": 0,
"ref_datastore_id": 0,
"tolerance": 0,
"value": 0,
"ref_expression": "string",
"ref_filter": "string",
"required_labels": [
"road"
],
"numeric_comparator": {
"epsilon": 0,
"as_absolute": true
},
"duration_comparator": {
"millis": 0
},
"string_comparator": {
"ignore_whitespace": false
},
"distinct_field_name": "string",
"pair_substrings": true,
"pair_homophones": true,
"spelling_similarity_threshold": 0
},
"template_checks_count": 0,
"anomaly_count": 0,
"active_anomaly_count": 0,
"deleted_at": "2024-06-10T21:29:42.697Z",
"global_tags": [
{
"type": "global",
"name": "string",
"color": "string",
"description": "string",
"weight_modifier": 0
},
{
"type": "external",
"name": "string",
"color": "string",
"description": "string",
"weight_modifier": 0,
"integration": {
"id": 0,
"created": "2024-06-10T21:29:42.697Z",
"name": "string",
"type": "atlan",
"api_url": "string",
"overwrite": true,
"last_synced": "2024-06-10T21:29:42.697Z",
"status": "syncing"
}
}
],
"last_editor": {
"id": 0,
"created": "2024-06-10T21:29:42.697Z",
"user_id": "string",
"email": "string",
"name": "string",
"picture": "string",
"role": "Member",
"deleted_at": "2024-06-10T21:29:42.697Z",
"teams": [
{
"id": 0,
"name": "string",
"permission": "Read"
}
]
},
"last_updated": "2024-06-10T21:29:42.697Z",
"last_asserted": "2024-06-10T21:29:42.697Z",
"has_passed": true,
"weight": 0,
"additional_metadata": {
"additionalProp1": "string",
"additionalProp2": "string",
"additionalProp3": "string"
}
},
"message": "string",
"suggested_field": "string",
"suggested_value": "string",
"suggested_score": 0
}
]
}
Retrieving Anomaly Source Records
Endpoint (Get)
/api/anomalies/{id}/source-record (get)