Manage Checks in Datastore

Managing your checks within a datastore is important to maintain data integrity and ensure quality. You can categorize, create, update, archive, restore, delete, and clone checks, making it easier to apply validation rules across the datastores. The system allows for checks to be set as active, draft, or archived based on their current state of use. You can also define reusable templates for quality checks to streamline the creation of multiple checks with similar criteria. With options for important and favorite, users have full flexibility to manage data quality efficiently.

Let's get started 🚀

Navigation

Step 1: Log in to your Qualytics account and select the datastore from the left menu on which you want to manage your checks.

Step 2: Click on the "Checks" from the Navigation Tab.

You will be navigated to the Checks section within the selected datastore. Here, you can view checks categorized as Active, Draft, Archived (including Invalid and Discarded), or All.

Status Management of Checks

Set Check as Draft

You can move an active check into a draft state, allowing you to work on the check, make adjustments, and refine the validation rules without affecting live data. This is useful when you need to temporarily deactivate a check for review and updates. There are two methods from which you can move your active check to draft: you can either draft specific checks or draft multiple checks in bulk.

Method I: Draft Specific Check

Step 1: Click on the vertical ellipsis (⋮) next to the active check you want to move to draft state, and select Edit from the dropdown menu.

For Demonstration purpose, we have selected the "After Date Time" check.

Step 2: A modal window will appear displaying the check details. Click on the vertical ellipsis (⋮) located in the upper-right corner of the modal window, and select "Draft" from the drop-down menu.

Step 3: After clicking on "Draft", the check will be successfully moved to the draft state, and a success flash message will appear stating, "The checks have been successfully updated."

Method II. Draft Checks in Bulk

You can move multiple checks into the draft state in one action, allowing you to pause or make adjustments to several checks without affecting your active validation process.

Step 1: Hover over the active checks and click on the checkbox to select multiple checks.

Step 2: Click on the vertical ellipses (⋮) and select "Draft" from the dropdown menu to move active checks to the draft state.

A confirmation modal window titled Bulk Update Checks to Draft will appear, indicating the number of checks being moved to draft.

Step 3: Click the "Update" button to move the selected active checks to draft.

After clicking the "Update" button, your selected checks will be moved to draft, and a success message will appear stating, "The checks have been successfully updated."

Activate Draft Check

You can activate the draft checks after when you have worked on the check, make adjustments, and refine the validation rules. By activating the draft check and making it live, ensures that the defined criteria are enforced on the data. There are two ways to activate draft checks: you can activate specific checks or activate multiple checks in bulk.

Method I. Activate Specific Check

Step 1: Navigate to the Draft check section, and click on the vertical ellipsis (⋮) next to the draft check you want to activate, and select Edit from the dropdown menu.

For Demonstration purpose, we have selected the "Metric" check.

A modal window will appear with the check details. If you want to make any changes to the check details, you can edit them.

Step 2: Click on the down arrow icon with the Update button. A dropdown menu will appear, click on the Activate button.

Step 3: After clicking on the activate button, your check is now successfully moved to the active checks and a success flash message will appear stating "Check successfully updated"

Method II. Activate Draft Checks in Bulk

Step 1. Hover over the draft checks and click on the checkbox to select multiple checks in bulk.

When multiple checks are selected, an action toolbar appears, displaying the total number of checks chosen along with a vertical ellipsis for additional bulk action options.

Step 2. Click on the vertical ellipsis (⋮) and choose "Activate" from the dropdown menu to activate the selected checks.

Step 3. A confirmation modal window “Bulk Activate Check” will appear, click on the “Activate” button to activate the draft checks.

After clicking on the activate button, your drafts checks will be activated and a success message flash will appear stating “The checks have been successfully updated”

Set Check as Archived

You can move an active or draft check into the archive when it is no longer relevant but may still be needed for historical purposes or future use. Archiving helps keep your checks organized without permanently deleting them. There are two ways to archive checks: you can archive individual checks or archive multiple checks in bulk.

Method I: Archive Specific Check

You can archive a specific check using two ways: either by directly clicking the archive button on the check or by opening the check and selecting the archive option from the action menu.

1. Archive Directly

Step 1: Locate the check (whether Active or Draft) which you want to archive and click the vertical ellipsis (⋮) next to it, and select Archive from the dropdown menu.

For Demonstration purpose, we have selected the "After Date Time" check.

Step 2: A modal window titled "Archive Check" will appear, providing you with the following archive options:

• Discarded: Select this option if the check is no longer relevant or suitable for the current business rules or data requirements. This helps in archiving checks that are obsolete but still exist for historical reference.

• Invalid: Choose this option if the check is not valid and should be retired from future inference. This helps the system learn from invalid checks and improves its ability to infer valid checks in the future.

Step 3: Once you've made your selection, click the Archive button to proceed.

Step 4: After clicking on the Archive button your check is moved to the archive and a flash message will appear saying " The check has been successfully archived"

2. Archive from Action Menu

Step 1: Locate the check (whether Active or Draft) which you want to archive and click the vertical ellipsis (⋮) next to it, and select Edit from the dropdown menu.

For Demonstration purpose, we have selected the "Metric" check.

Step 2: A modal window will appear displaying the check details. Click on the vertical ellipsis (⋮) located in the upper-right corner of the modal window, and click on the "Archive" from the drop-down menu.

Step 3: A modal window titled “Archive Check” will appear, providing you with the following archive options:

• Discarded: Select this option if the check is no longer relevant or suitable for the current business rules or data requirements. This helps in archiving checks that are obsolete but still exist for historical reference.

• Invalid: Choose this option if the check is not valid and should be retired from future inference. This helps the system learn from invalid checks and improves its ability to infer valid checks in the future.

Step 4: Once you've made your selection, click the Archive button to proceed.

Step 5: After clicking on the Archive button your check is moved to the archive and a flash message will appear saying " The check has been successfully archived"

Method II: Archive Checks in Bulk

You can archive multiple checks in a single step, deactivating and storing them for future reference or restoration while keeping your active checks uncluttered.

Step 1: Hover over the checks (whether Active or Draft) and click on the checkbox to select multiple checks.

When multiple checks are selected, an action toolbar appears, displaying the total number of selected checks along with a vertical ellipsis for additional bulk action options.

Step 2: Click on the vertical ellipsis (⋮) and choose "Archive" from the dropdown menu to archive the selected checks.

A modal window will appear, providing you with the following archive options:

1. Delete all anomalies associated with the checks: Toggle this option "On" if you want to delete any anomalies related to the selected checks when archiving them.

2. Archive Options: You are presented with two options to categorize why the checks are being archived:

• Discarded: Select this option if the check is no longer relevant or suitable for the current business rules or data requirements. This helps in archiving checks that are obsolete but still exist for historical reference.

• Invalid: Choose this option if the check is not valid and should be retired from future inference. This helps the system learn from invalid checks and improves its ability to infer valid checks in the future.

Step 3: Once you've made your selections, click the "Archive" button to confirm and archive the checks.

Step 4: After clicking the "Archive" button, your selected checks (whether Active or Draft) will be archived successfully and a success flash message will appear stating, "The checks have been successfully archived."

Restore Archived Checks

If a check has been archived, then you can restore it back to an active state or in a draft state. This allows you to reuse your checks that were previously archived without having to recreate them from scratch.

Step 1: Click on Archived from the navigation bar in the Checks section to view all archived checks.

Step 2: Click on the vertical ellipsis (⋮) next to the active check you want to restore as an active or draft check and select edit form the drop down menu.

For Demonstration purpose, we have selected the "Metric" check.

A modal window will appear with the check details.

Step 3: If you want to make any changes to the check, you can edit it. Otherwise, click on the Restore button to restore it as an active check.

To restore the check as a draft, click on the arrow icon next to the Restore button. A dropdown menu will appear—select Restore as Draft from the options.

After clicking the Restore button, the check will be successfully restored as either an active or draft check, depending on your selection. A success message will appear confirming, "Check successfully updated."

Edit Check

You can edit an existing check to modify its properties, such as the rule type, coverage, filter clause, or description. Updating a check ensures that it stays aligned with evolving data requirements and maintains data quality as conditions change. There are two methods for editing checks: you can either edit specific checks or edit multiple checks in bulk.

Note

When editing multiple checks in bulk, only the filter clause and tags can be modified.

Method I. Edit Specific Check

Step 1: Click on the vertical ellipsis (⋮) next to the check you want to edit, whether it is an active or draft check and select Edit from the drop down menu

For Demonstration purpose, we have selected the "Metric" check.

A modal window will appear with the check details.

Step 2: Modify the check details as needed based on your preferences.

Step 3: Once you have edited the check details, then click on the Validate button. This will perform a validation operation on the check without saving it. The validation allows you to verify that the logic and parameters defined for the check are correct.

If the validation is successful, a green message saying "Validation Successful" will appear.

If the validation fails, a red message saying "Failed Validation" will appear. This typically occurs when the check logic or parameters do not match the data properly.

Step 3: Once you have a successful validation, click the "Update" button. The system will update the changes you've made to the check, including changes to the fields, filter clause, coverage, description, tags, or metadata.

After clicking on the Update button, your check is successfully updated and a success flash message will appear stating "The check has been successfully updated".

Method II. Edit Checks in Bulk

You can easily apply changes to multiple checks at once, saving time by editing several checks simultaneously without having to modify each one individually.

Step 1: Hover over the checks (whether Active or Draft) and click on the checkbox to select multiple checks.

When multiple checks are selected, an action toolbar appears, displaying the total number of selected checks along with a vertical ellipsis for additional bulk action options.

Step 2: Click on the vertical ellipses (⋮) and select "Edit" from the dropdown menu to make changes to the selected checks.

Step 3: A modal window titled "Bulk Edit Checks" will appear. Here you can only modify the "filter clause" and "tags" of the selected checks.

Step 4: Toggle on the options (Filter Clause or Tags) that you want to modify for the selected checks, and make the necessary changes.

Note

This action will overwrite the existing data for the selected checks.

Step 5: Once you have made the changes, click on the "Save" button.

After clicking the "Save" button, your selected checks will be updated with the new changes. A success message will appear stating, "The checks have been successfully updated."

Delete Checks

You can delete a check permanently, removing it from the system, and this is an irreversible action. Once you delete it, the check cannot be restored. By deleting the check, you ensure it will no longer appear in active or archived lists, making the system more streamlined and organized. There are two methods for deleting checks: you can either delete individual checks or delete multiple checks in bulk.

Note

You can only delete archived checks. If you want to delete an active or draft check, you must first move it to the archive, and then you can delete it.

Warning

Deleting a check is a one-time action. It cannot be restored after deletion.

Method I. Delete Specific Check

Step 1: Click on Archived from the navigation bar in the Checks section to view all archived checks.

Step 2: Locate the check, that you want to delete and click on the vertical ellipsis (⋮) and select Delete from the drop-down menu.

For Demonstration purpose, we have selected the "Time Distribution Size" check.

Step 3: A confirmation modal window will appear, click on the Delete button to permanently remove the check from the system.

Step 4: After clicking on the delete button, your check is successfully deleted and a success flash message will appear saying "The check has been successfully deleted"

Method II. Delete Check in Bulk

You can permanently delete multiple checks from the system in one action. This process is irreversible, so it should be used when you are certain that the checks are no longer needed.

Note

For bulk archiving checks, the only available option is Bulk Delete. There is no option to Bulk Restore to draft or activate archived checks.

Step 1: Hover over the archived checks and click on the checkbox to select checks in bulk.

When multiple checks are selected, an action toolbar appears, displaying the total number of selected checks along with a vertical ellipsis for additional bulk action options.

Step 2: Click on the vertical ellipsis (⋮) and choose "Delete" from the dropdown menu to delete the selected checks.

Step 3: A confirmation modal window will appear, click on the "Delete" button to permanently delete the selected checks.

After clicking on the "Delete" button, your selected checks will be permanently deleted, and a success flash message will appear stating, "The checks have been successfully deleted."

Mark Check as Favorite

Marking a check as a favorite allows you to quickly access and prioritize the checks that are most important to your data validation process. This helps streamline workflows by keeping frequently used or critical checks easily accessible, ensuring you can monitor and manage them efficiently. By marking a check as a favorite, it will appear in the "Favorite" category for faster retrieval and management.

Step 1: Locate the check which you want to mark as a favorite and click on the bookmark icon located on the right side of the check.

After Clicking on the bookmark icon your check is successfully marked as a favorite and a success flash message will appear stating "The check has been favorited"

To unmark a check, simply click on the bookmark icon of the marked check. This will remove it from your favorites.

Clone Check

You can clone both active and draft checks to create a duplicate copy of an existing check. This is useful when you want to create a new check based on the structure of an existing one, allowing you to make adjustments without affecting the original check.

Step 1: Click the vertical ellipsis(⋮) next to the check (whether Active or Draft) that you want to clone and select Edit from the drop-down menu.

For Demonstration purpose, we have selected the "Metric" check.

Step 2: A modal window will appear displaying the check details. Click on the vertical ellipsis (⋮) located in the upper-right corner of the modal window, and select "Clone" from the drop-down menu.

Step 3: After clicking the Clone button, a modal window will appear. This window allows you to adjust the cloned check's details.

1. If you toggle on the "Associate with a Check Template" option, the cloned check will be linked to a specific template.

Choose a Template from the dropdown menu that you want to associate with the cloned check. The check will inherit properties from the selected template.

• Locked: The check will automatically sync with any future updates made to the template, but you won't be able to modify the check's properties directly.

• Unlocked: You can modify the check, but future updates to the template will no longer affect this check.

2. If you toggle off the "Associate with a Check Template" option, the cloned check will not be linked to any template, which allows you full control to modify the properties independently.

Select the appropriate Rule type for the check from the dropdown menu.

Step 4: Once you have selected the template or rule type, fill up the remaining check details as required.

Step 5: After completing all the check details, click on the "Validate" button. This will perform a validation operation on the check without saving it. The validation allows you to verify that the logic and parameters defined for the check are correct. It ensures that the check will work as expected by running it against the data without committing any changes.

If the validation is successful, a green message saying "Validation Successful" will appear.

If the validation fails, a red message saying "Failed Validation" will appear. This typically occurs when the check logic or parameters do not match the data properly.

Step 6: Once you have a successful validation, click the "Save" button. The system will save any modifications you've made to the check, and create a clone of that check on basis of your changes.

After clicking on the "Save" button your check is successfully created and a success flash message will appear stating "Check successfully created".

Create a Quality Check template

You can add checks as a Template, which allows you to create a reusable framework for quality checks. By using templates, you standardize the validation process, enabling the creation of multiple checks with similar rules and criteria across different datastores. This ensures consistency and efficiency in managing data quality checks.

Step 1: Locate the check (whether Active or Draft) which you want to archive and click on that check.

For Demonstration purpose, we have selected the "Not Exists In" check.

Step 2: A modal window will appear displaying the check details. Click on the vertical ellipsis (⋮) located in the upper-right corner of the modal window, and select "Template" from the drop-down menu.

After clicking the "Template" button, the check will be saved and created as a template in the library, and a success flash message will appear stating, "The quality check template has been created successfully." This allows you to reuse the template for future checks, streamlining the validation process.

Filter and Sort

Filter and Sort options allow you to organize your checks by various criteria, such as Weight, Active Anomalies, Coverage, Created Date, and Rules. You can also apply filters to refine your list of checks based on Check Type, Asserted State (Passed, Failed, Not Asserted), Tags, Tables, and Fields.

Sort

You can sort your checks by Active Anomalies, Coverage, Created Date, Last Asserted, Rules, and Weight to easily organize and prioritize them according to your needs.

No	Sort By Option	Description
1	Active Anomalies	Sort checks based on the number of active anomalies.
2	Coverage	Sort checks by data coverage percentage.
3	Created Date	Sort checks according to the date they were created.
4	Last Asserted	Sorts by the last time the check was executed.
5	Rules	Sort checks based on specific rules applied to the checks.
6	Weight	Sort checks by their assigned weight or importance level.

Whatever sorting option is selected, you can arrange the data either in ascending or descending order by clicking the caret button next to the selected sorting criteria.

Filter

You can filter your checks based on values like Check Type, Asserted State, Rule, Tags, Table, Field, and Template.

No	Filter	Filter Value	Description
1	Check Type	All	Displays all types of checks, both inferred and authored.
		Inferred	Shows system-generated checks that automatically validate data based on detected patterns or logic.
		Authored	Displays user-created checks, allowing the user to focus on custom validations tailored to specific requirements.
2	Asserted State	All	Displays all checks, regardless of their asserted status. This provides a full overview of both passed, failed, and not asserted checks.
		Passed	Shows checks that have been asserted successfully, meaning no active anomalies were found during the validation process.
		Failed	Displays checks that have failed assertion, indicating active anomalies or issues that need attention.
		Not Asserted	Filters out checks that have not yet been asserted, either because they haven’t been processed or validated yet.
3	Rule	N/A	Select this to filter the checks based on specific rule type for data validation, such as checking non-null values, matching patterns, comparing numerical ranges, or verifying date-time constraints. By clicking on the caret down button next to the Rule field, the available rule types will be dynamically populated based on the rule types present in the results. The rules displayed are based on the current dataset and provide more granular control over filtering. Each rule type will show a counter next to it, displaying the total number of occurrences for that rule in the dataset. For example, the rule type After Date Time is displayed with a total of 2 occurrences.

No	Filter	Filter Value	Description
4	Tag	N/A	Tag Filter displays only the tags associated with the currently visible items, along with their color icon, name, type, and the number of matching records. Selecting one or more tags refines the list based on your selection. If no matching items are found, a No option found message is displayed.
5	Table	N/A	Filters checks by the table to which they are applied.
6	Field	N/A	Filters checks by the specific field/column name within a table.
7	Template	N/A	This filter allows users to view and apply predefined check templates.