Create a Check¶

DO Creating metadata or metric checks or templates

DQ Creating integrity checks or templates

Create a check of any rule type with the check form. Every rule type uses the same form and the same steps. Only the fields under Target and Properties change with the rule. For the fields of one rule type, see Create a Check by Rule Type.

Permission Required

Creating a check as Draft requires the Drafter team permission on its datastore. Creating it as Active requires the Author team permission. Either way, your organization needs the module for the rule type. See Team Permissions.

Show me how

The app can walk you through this. Click Show me how in the check creation form's header, or press H while it is open, and the Add a Check walkthrough highlights each step while you fill in the real form.

Let AgentQ fill in the form

When AgentQ is configured and its data-sharing level is Metadata shared or higher, the Check Assistant panel next to the form can write the check for you: describe the rule in plain language, or pick one of its suggested prompts, and apply the configuration it proposes. Review the fields before you save.

Field Reference¶

The form has the same sections for every rule type. A field shows only when the rule type you picked uses it, so a check on a whole table, for example, has no Field and no Coverage. The values that belong to one rule type live under Properties, and each rule type's own page explains them. See Create a Check by Rule Type.

Target¶

What the check evaluates: the rule, where it runs, and which records it looks at.

Field	Required	Description
Associate with a Check template		A toggle shown when your organization has check templates. Turn it on to pick a Template instead of a Rule Type: the check takes the template's settings and stays linked to it. A Locked template keeps its settings on the check and passes later changes to it. An Unlocked template lets you change them, and later changes to the template do not reach the check. See How Templates Work.
Rule Type		The rule the check applies, picked from groups such as Single Field, Multiple Fields, and Calculated. It decides which of the other fields appear. A deprecated rule type shows a warning when you pick it.
Table / File		The table or file the check runs on. The label follows the datastore, so it reads Table on a database and File on a file storage. Already filled in when you start from a table, file, or field. Add Computed Table / File creates a computed table or file from the form when none is selected yet.
Field or Fields		The field, or fields, the rule applies to. Rules on one field show Field, and rules that take several fields show Fields. Already filled in when you start from a field. Rules that evaluate the table as a whole have none.
Fields (expression rules)		On rules that evaluate an expression you write, the fields are worked out from the expression when you leave this empty. Selecting fields replaces that list.
Filter Clause		A Spark SQL `WHERE` condition that limits the records the check evaluates, such as `country = 'US'`. Leave it empty to evaluate every record.
Custom Anomaly Description		A toggle under Advanced. Turn it on and pick a field of the table or file: the anomaly then shows that field's value from the failing record as its description, instead of the one Qualytics writes.

Properties¶

The values the rule type needs, such as a minimum, a pattern, a list of expected values, or another table to compare with. They are different for every rule type, and some rule types have none. See Create a Check by Rule Type.

Pass Criteria¶

Field	Required	Description
Coverage		A slider that sets the percentage of evaluated records that must satisfy the rule for the check to pass, shown as "Must be true for 100% of records". Starts at 100%. Lower it to tolerate a share of failing records. Rules that evaluate the table as a whole have no coverage.

Impact¶

Field	Required	Description
Metrics		Impact metrics that turn the failed records into one number each, such as the total amount of the failing orders, to show how big a failure is. Click Add your first impact metric to add the first one, then Add Impact Metric for each one after it, up to 10. Only rule types that flag individual records offer them. See Impact Metrics.

Ownership¶

Field	Required	Description
Owner		The user responsible for the check. Already filled in with you. The list shows users with at least the Drafter team permission on the datastore. See Check Ownership.
Anomaly assignee		The user who is automatically assigned to every new anomaly the check finds. The list shows users with at least the Viewer team permission on the datastore.

Metadata¶

Field	Required	Description
Description		What the check verifies and why, written in a Markdown editor. It is the text your team reads on the check and on its anomalies. With an AI integration configured, Suggest a description writes one from the check's definition.
Tags		Labels that group checks for filtering, notifications, and importance. A tag can raise or lower the check's importance. See Check Importance.
Additional Metadata		Your own key-value pairs, such as a ticket number or a business owner, added in a dialog with Add New Meta.

Create a Check by Rule Type¶

Each rule type has its own Create a Check page with the fields under Properties and a filled-in example. The rule types are listed in the same order as the rule type picker. Each link opens in a new tab.

Rule Type	Group	What It Checks
After Date Time	Single Field	Asserts that the field is a timestamp later than a specific date and time.
Before Date Time	Single Field	Asserts that the field is a timestamp earlier than a specific date and time.
Between	Single Field	Asserts that values are equal to or between two numbers.
Between Times	Single Field	Asserts that values are equal to or between two dates or times.
Contains Credit Card	Single Field	Asserts that the values contain a credit card number.
Contains Email	Single Field	Asserts that the values contain email addresses.
Contains Social Security Number	Single Field	Asserts that the values contain social security numbers.
Contains Url	Single Field	Asserts that the values contain valid URLs.
Distinct Count	Single Field	Asserts on the approximate count distinct of the given column.
Equal To Field	Single Field	Asserts that this field is equal to another field.
Exists In	Single Field	Asserts that values assigned to this field exist as values in another field.
Expected Values	Single Field	Asserts that values are contained within a list of expected values.
Greater Than	Single Field	Asserts that the field is a number greater than (or equal to) a value.
Greater Than Field	Single Field	Asserts that this field is greater than another field.
Is Address	Single Field	Asserts that the values are addresses of the required format.
Is Credit Card	Single Field	Asserts that the values are credit card numbers.
Is Type	Single Field	Asserts that the data is of a specific type.
Less Than	Single Field	Asserts that the field is a number less than (or equal to) a value.
Less Than Field	Single Field	Asserts that this field is less than another field.
Matches Pattern	Single Field	Asserts that a field must match a pattern.
Max Length	Single Field	Asserts that a string has a maximum length. On array fields, checks the array has at most N elements. With element context, checks each string element's length.
Max Value	Single Field	Asserts that a field has a maximum value.
Min Length	Single Field	Asserts that a string has a minimum length. On array fields, checks the array has at least N elements. With element context, checks each string element's length.
Min Value	Single Field	Asserts that a field has a minimum value.
Not Empty	Single Field	Asserts that a text or array field is not empty. A value made only of spaces counts as empty.
Not Exists In	Single Field	Asserts that values assigned to this field do not exist as values in another field.
Not Future	Single Field	Asserts that the field's value is not in the future.
Not Negative	Single Field	Asserts that this is a non-negative number.
Positive	Single Field	Asserts that this is a positive number.
Predicted By	Single Field	Asserts that the actual value of a field falls within an expected predicted range.
Required Values	Single Field	Asserts that all of the defined values must be present at least once within a field.
Sum	Single Field	Asserts that the sum of a field is a specific amount.
Time Distribution Size	Single Field	Asserts that the count of records for each interval of a timestamp is between two numbers.
Any Not Null	Multiple Fields	Asserts that at least one of the selected fields must hold a value.
Data Diff	Multiple Fields	Asserts that the dataset created by the targeted field(s) has differences compared to the referred field(s).
Equal To	Multiple Fields	Asserts that all of the selected fields equal a value.
Is Replica Of	Multiple Fields	Deprecated. Replaced by Data Diff, which performs the same row-by-row comparison and adds change-type filtering.
Not Null	Multiple Fields	Asserts that the field's values are not null. On array fields with element context, asserts that no elements within the array are null.
Unique	Multiple Fields	Asserts that each field's value is unique across rows. When multiple fields are selected, asserts that the combination of values across those fields is unique (composite key).
Aggregation Comparison	Calculated	Verifies that the specified comparison operator evaluates true when applied to the two aggregation expressions.
Entity Resolution	Calculated	Asserts that records describing the same entity are clustered together, even when their values vary, and that every record in a cluster shares one value of a chosen distinction field.
Satisfies Expression	Calculated	Evaluates a SQL expression you provide against every record.
Metric	Aggregated	Records the value of the selected field during each scan operation and asserts limits based upon an expected change or absolute range (inclusive).
Freshness	Metadata	Asserts that the data was added or updated within a maximum allowed age.
Volumetric	Metadata	Asserts that the row count of a table or file stays within an expected range, or changes by no more than expected since an earlier day.
Expected Schema	Schema	Asserts that all selected fields are present and that all declared data types match expectations.
Field Count	Schema	Asserts that there must be exactly a specified number of fields.
Max Partition Size	Schema	Asserts the maximum number of records that should be loaded from each file or table partition.
Min Partition Size	Schema	Asserts the minimum number of records that should be loaded from each file or table partition.

Steps¶

Each field is described in the Field Reference above.

Step 1: Open the datastore, table or file, or field the check is for. Starting from a table or file, or a field, fills it in on the form for you.

Step 2: Click Add at the top right. A menu opens.

Step 3: Click Check . The Authored Check Details form opens. You can also press C on the datastore's pages to open it.

No tables available

On a datastore that has no tables yet, for example one that was just added and has not been synced, the form shows a No tables available warning (No files available on a file-based datastore) that reads Run a Sync on this datastore before adding checks. instead of the check fields. When you open the form from the datastore page and you can run operations on the datastore, the warning also offers a Run Sync button, which closes the form and opens the Sync dialog. Add the check once the Sync finishes. The warning does not appear when you add a check from a check template.

Step 4: Under Target, click the Rule Type list and pick the rule. The list groups the rule types into Single Field, Multiple Fields, Calculated, Aggregated, Metadata, and Schema. The rest of the form adapts to the rule you pick.

Step 5: Select the Table / File, if it is not filled in already.

Step 6: Select the Field or Fields, if the rule asks for them and they are not filled in already.

Step 7: Optionally, enter a Filter Clause to limit the records the check evaluates.

Step 8: Under Properties, fill in the values the rule type asks for. Some rule types have none. See Create a Check by Rule Type for the values of each one.

Step 9: Under Pass Criteria, drag the Coverage slider to the percentage of records that must satisfy the rule. It starts at 100%.

Step 10: Optionally, under Impact, click Add your first impact metric and fill in the metric. See Impact Metrics.

Step 11: Under Ownership, keep yourself as the Owner, or click the list and pick another user.

Step 12: Optionally, click the Anomaly assignee list and pick the user who gets the check's new anomalies.

Step 13: Under Metadata, enter the Description. With an AI integration configured, click Suggest a description to have one written from the check's definition.

Step 14: Optionally, add Tags and Additional Metadata.

Step 15: Click Validate. Qualytics tests the check against the selected data without saving it, and a message shows whether validation succeeded.

Step 16: Click Save. A success message appears, and the check is created as Active.

Save as a Draft or a template

To save the check another way, click the arrow next to Save and pick Save as Draft or Save as Template . With only the Drafter team permission, the button reads Save as Draft, and the check is created as a Draft.

A filter that can never run

Validate refuses a filter clause that can never convert a value, such as amount <> '' on a number field, or that uses a function Qualytics does not allow, and names the reason. This applies to every rule type that accepts a filter, including checks that compute one value for the whole table, such as Sum, and checks with a coverage below 100%. If a check with such a filter reaches a scan, it is reported as Not Asserted with that reason, and the other checks on the table still run.

Locked after save

The Rule Type and the Table / File cannot change once the check is saved. To use another one, create a new check, or clone this one. See Clone a Check.

To create many checks at once from the same rule, use a check template. See Add Multiple Checks from a Template.