Skip to content

Check Overview

A Check is an encapsulation of a data quality rule along with additional context such as associated notifications, tags, filters and tolerances.

There are two types of Checks in Qualytics: inferred and authored:

  • Inferred checks are automatically generated by the Qualytics platform as part of a Profile Operation. Typically, inferred checks will cover 80 - 90% of rules that a user would need.

  • Authored checks are manually authored by a user via UI or API. There are many types of checks that can be authored through templates for common checks as well as more complex rules through SQL and UDF (in Scala).

Checks Tab

  1. Select a Datastore.
  2. Click on Checks tab

    Screenshot Screenshot
  3. You are going to see all checks related to your Datastore

    Screenshot Screenshot

Check Rule Types

Rule Type Description
After Date Time Asserts that the field is a timestamp later than a specific date and time.
Any Not Null Asserts that one of the fields must not be null.
Before DateTime Asserts that the field is a timestamp earlier than a specific date and time.
Between Asserts that values are equal to or between two numbers.
Between Times Asserts that values are equal to or between two dates or times.
Contains Credit Card Asserts that the values contain a credit card number.
Contains Email Asserts that the values contain email addresses.
Contains Social Security Number Asserts that the values contain social security numbers.
Contains Url Asserts that the values contain valid URLs.
Distinct Count Asserts on the approximate count distinct of the given column.
Entity Resolution Asserts that every distinct entity is appropriately represented once and only once
Equal To Field Asserts that this field is equal to another field.
Exists in Asserts if the rows of a compared table/field of a specific Datastore exists in the selected table/field.
Expected Schema Asserts that all selected fields are present and that all declared data types match expectations.
Expected Values Asserts that values are contained within a list of expected values.
Field Count Asserts that there must be exactly a specified number of fields.
Greater Than Asserts that the field is a number greater than (or equal to) a value.
Greater Than Field Asserts that this field is greater than another field.
Is Address Asserts that the values contain the specified required elements of an address.
Is Credit Card Asserts that the values are credit card numbers.
Is Replica Of Asserts that the dataset created by the targeted field(s) is replicated by the referred field(s).
Is Type Asserts that the data is of a specific type.
Less Than Asserts that the field is a number less than (or equal to) a value.
Less Than Field Asserts that this field is less than another field.
Matches Pattern Asserts that a field must match a pattern.
Max Length Asserts that a string has a maximum length.
Max Value Asserts that a field has a maximum value.
Metric Records the value of the selected field during each scan operation and asserts that the value is within a specified range (inclusive).
Min Length Asserts that a string has a minimum length.
Min Partition Size Asserts the minimum number of records that should be loaded from each file or table partition.
Min Value Asserts that a field has a minimum value.
Not Exists In Asserts that values assigned to this field do not exist as values in another field.
Not Future Asserts that the field's value is not in the future.
Not Negative Asserts that this is a non-negative number.
Not Null Asserts that the field's value is not explicitly set to nothing.
Positive Asserts that this is a positive number.
Predicted By Asserts that the actual value of a field falls within an expected predicted range.
Required Values Asserts that all of the defined values must be present at least once within a field.
Satisfies Expression Evaluates the given expression (any valid Spark SQL) for each record.
Sum Asserts that the sum of a field is a specific amount.
Time Distribution Size Asserts that the count of records for each interval of a timestamp is between two numbers.
Unique Asserts that the field's value is unique.
User Defined Function Asserts that the given user-defined function (as Scala script) evaluates to true over the field's value.

Last update: June 14, 2024