Rule Types Overview

In Qualytics, a variety of rule types are provided to maintain data quality and integrity.These rules define specific criteria that data must meet, and checks apply these rules during the validation process.

Here’s an overview of the rule types and their purposes:

Check Rule Types

Rule Type	Description
After Date Time	Asserts that the field is a timestamp later than a specific date and time.
Any Not Null	Asserts that one of the fields must not be null.
Before DateTime	Asserts that the field is a timestamp earlier than a specific date and time.
Between	Asserts that values are equal to or between two numbers.
Between Times	Asserts that values are equal to or between two dates or times.
Contains Credit Card	Asserts that the values contain a credit card number.
Contains Email	Asserts that the values contain email addresses.
Contains Social Security Number	Asserts that the values contain social security numbers.
Contains Url	Asserts that the values contain valid URLs.
Distinct Count	Asserts on the approximate count distinct of the given column.
Entity Resolution	Asserts that every distinct entity is appropriately represented once and only once
Equal To Field	Asserts that this field is equal to another field.
Exists in	Asserts if the rows of a compared table/field of a specific Datastore exists in the selected table/field.
Expected Schema	Asserts that all selected fields are present and that all declared data types match expectations.
Expected Values	Asserts that values are contained within a list of expected values.
Field Count	Asserts that there must be exactly a specified number of fields.
Greater Than	Asserts that the field is a number greater than (or equal to) a value.
Greater Than Field	Asserts that this field is greater than another field.
Is Address	Asserts that the values contain the specified required elements of an address.
Is Credit Card	Asserts that the values are credit card numbers.
Is Replica Of	Asserts that the dataset created by the targeted field(s) is replicated by the referred field(s).
Is Type	Asserts that the data is of a specific type.
Less Than	Asserts that the field is a number less than (or equal to) a value.
Less Than Field	Asserts that this field is less than another field.
Matches Pattern	Asserts that a field must match a pattern.
Max Length	Asserts that a string has a maximum length.
Max Value	Asserts that a field has a maximum value.
Metric	Records the value of the selected field during each scan operation and asserts that the value is within a specified range (inclusive).
Min Length	Asserts that a string has a minimum length.
Min Partition Size	Asserts the minimum number of records that should be loaded from each file or table partition.
Min Value	Asserts that a field has a minimum value.
Not Exists In	Asserts that values assigned to this field do not exist as values in another field.
Not Future	Asserts that the field's value is not in the future.
Not Negative	Asserts that this is a non-negative number.
Not Null	Asserts that the field's value is not explicitly set to nothing.
Positive	Asserts that this is a positive number.
Predicted By	Asserts that the actual value of a field falls within an expected predicted range.
Required Values	Asserts that all of the defined values must be present at least once within a field.
Satisfies Expression	Evaluates the given expression (any valid Spark SQL) for each record.
Sum	Asserts that the sum of a field is a specific amount.
Time Distribution Size	Asserts that the count of records for each interval of a timestamp is between two numbers.
Unique	Asserts that the field's value is unique.
User Defined Function	Asserts that the given user-defined function (as Scala script) evaluates to true over the field's value.
Volumetric Check	Asserts that the volume of the data asset has not changed by more than an inclusive percentage amount for the prescribed moving daily average.