Getting Started with Checks
A check applies a rule type to a table or a file. The rule type decides what is asserted: a boundary on a number, the shape of a text value, a relationship between two datasets, the structure of the container itself. Everything else on a check, the filter, the coverage, the ownership, and the metadata, works the same way whichever rule you pick.
This page groups every rule type by the question it answers, so you can find the right one without reading all of them. Each rule has its own section with the full reference, examples, how-tos, and API.
Not sure which rule to pick?
Start from the question you want answered, not from the field. "Is this value inside a range" leads to a different rule than "do these two tables agree", even when both look at the same column.
Numbers and ranges
Rules that constrain a numeric value against a boundary, a range, or another column.
-
Between
Asserts that values are equal to or between two numbers.
-
Greater Than
Asserts that the field is a number greater than (or equal to) a value.
-
Greater Than Field
Asserts that this field is greater than another field.
-
Less Than
Asserts that the field is a number less than (or equal to) a value.
-
Less Than Field
Asserts that this field is less than another field.
-
Max Value
Asserts that a field has a maximum value.
-
Min Value
Asserts that a field has a minimum value.
-
Positive
Asserts that this is a positive number.
-
Not Negative
Asserts that this is a non-negative number.
-
Equal To
Asserts that all of the selected fields equal a value.
-
Equal To Field
Asserts that this field is equal to another field.
-
Sum
Asserts that the sum of a field is a specific amount.
Text and format
Rules that validate the shape or the content of a text value.
-
Matches Pattern
Asserts that a field must match a pattern.
-
Max Length
Asserts that a string has a maximum length. On array fields, checks the array has at most N elements. With element context, checks each string element's length.
-
Min Length
Asserts that a string has a minimum length. On array fields, checks the array has at least N elements. With element context, checks each string element's length.
-
Is Type
Asserts that the data is of a specific type.
-
Is Address
Asserts that the values are addresses of the required format.
-
Is Credit Card
Asserts that the values are credit card numbers.
-
Contains Credit Card
Asserts that the values contain a credit card number.
-
Contains Email
Asserts that the values contain email addresses.
-
Contains Social Security Number
Asserts that the values contain social security numbers.
-
Contains Url
Asserts that the values contain valid URLs.
Dates and time
Rules that place a value on the timeline, or watch how recent the data is.
-
After Date Time
Asserts that the field is a timestamp later than a specific date and time.
-
Before Date Time
Asserts that the field is a timestamp earlier than a specific date and time.
-
Between Times
Asserts that values are equal to or between two dates or times.
-
Not Future
Asserts that the field's value is not in the future.
-
Freshness
Asserts that the data was added or updated within a maximum allowed age.
-
Time Distribution Size
Asserts that the count of records for each interval of a timestamp is between two numbers.
Completeness and uniqueness
Rules about values being present, distinct, or drawn from an agreed set.
-
Not Null
Asserts that the field's values are not null. On array fields with element context, asserts that no elements within the array are null.
-
Not Empty
Asserts that a text or array field is not empty. A value made only of spaces counts as empty.
-
Any Not Null
Asserts that at least one of the selected fields must hold a value.
-
Unique
Asserts that each field's value is unique across rows. When multiple fields are selected, asserts that the combination of values across those fields is unique (composite key).
-
Distinct Count
Asserts on the approximate count distinct of the given column.
-
Expected Values
Asserts that values are contained within a list of expected values.
-
Required Values
Asserts that all of the defined values must be present at least once within a field.
Comparing two datasets
Rules that read a second container and assert a relationship between the two.
-
Data Diff
Asserts that the dataset created by the targeted field(s) has differences compared to the referred field(s).
-
Aggregation Comparison
Verifies that the specified comparison operator evaluates true when applied to the two aggregation expressions.
-
Exists In
Asserts that values assigned to this field exist as values in another field.
-
Not Exists In
Asserts that values assigned to this field do not exist as values in another field.
-
Entity Resolution
Asserts that records describing the same entity are clustered together, even when their values vary, and that every record in a cluster shares one value of a chosen distinction field.
Structure and volume
Rules that look at the shape of the container rather than at the values inside it.
-
Expected Schema
Asserts that all selected fields are present and that all declared data types match expectations.
-
Field Count
Asserts that there must be exactly a specified number of fields.
-
Max Partition Size
Asserts the maximum number of records that should be loaded from each file or table partition.
-
Min Partition Size
Asserts the minimum number of records that should be loaded from each file or table partition.
-
Volumetric
Asserts that the volume of the data asset has not changed by more than an inclusive percentage amount for the prescribed moving daily average.
-
Metric
Records the value of the selected field during each scan operation and asserts limits based upon an expected change or absolute range (inclusive).
Custom logic
Rules where you supply the expression or let the platform learn the expectation.
-
Satisfies Expression
Evaluates a SQL expression you provide against every record.
-
Predicted By
Asserts that the actual value of a field falls within an expected predicted range.
Deprecated rule types
These rule types still evaluate on existing checks, but they are no longer maintained and should not be used for new ones.
-
Is Replica Of
Replaced by Data Diff, which performs the same row-by-row comparison and adds change-type filtering.