Check Templates

Check Templates empower users to efficiently create, manage, and apply standardized checks across various datastores, acting as blueprints that ensure consistency and data integrity across different datasets and processes.

Check templates streamline the validation process by enabling check management independently of specific data assets such as datastores, containers, or fields. These templates reduce manual intervention, minimize errors, and provide a reusable framework that can be applied across multiple datasets, ensuring all relevant data adheres to defined criteria. This not only saves time but also enhances the reliability of data quality checks within an organization.

Let's get started 🚀

Step 1: Log in to your Qualytics account and click the “Library” button on the left side panel of the interface.

Step 2: Click on the “Add Check Template” button located in the top right corner.

A modal window titled “Check Template Details” will appear, providing you the options to add the check template details.

Step 3: Enter the following details to add the check template:

• Rule Type (Required)
• Filter Clause
• Description (Required)
• Tag
• Additional Metadata
• Template Locked

1. Rule Type (Required): Select a Rule type from the dropdown menu for data validation, such as checking for non-null values, matching patterns, comparing numerical values, or verifying date-time constraints. Each rule type defines the specific validation logic to be applied.

For more details about the available rule types, refer to the "Check Rule Types" section.

Note

Different rule types have different sets of fields and options appearing when selected.

2. Filter Clause: Specify a valid Spark SQL WHERE expression to filter the data on which the check will be applied.

The filter clause defines the conditions under which the check will be applied. It typically includes a WHERE statement that specifies which rows or data points should be included in the check.

Example: A filter clause might be used to apply the check only to rows where a certain column meets a specific condition, such as WHERE status \= 'active'.

Adjust the Coverage setting to specify the percentage of records that must comply with the check.

Note

The Coverage setting applies to most rule types and allows you to specify the percentage of records that must meet the validation criteria.

3. Description (Required): Enter a detailed description of the check template, including its purpose, applicable data, and relevant information to ensure clarity for users. If you're unsure of what to include, click on the "💡" lightbulb icon to apply a suggested description based on the rule type.

Example: "The < field > must exist in bank_transactions_*.csv.Total_Transaction_Amount (Bank Dataset - Staging)".

This description clarifies that the specified field must be present in a particular file (bank_transactions_*.csv) and column (Total_Transaction_Amount) within the Bank Dataset.

4. Tag: Assign relevant tags to your check template to facilitate easier searching and filtering based on categories like "data quality," "financial reports," or "critical checks."

5. Additional Metadata: Add key-value pairs as additional metadata to enrich your check. Click the plus icon (+) next to this section to open the metadata input form, where you can add key-value pairs.

Enter the desired key-value pairs (e.g., DataSourceType: SQL Database and PriorityLevel: High). After entering the necessary metadata, click "Confirm" to save the custom metadata.

6. Template Locked: Check or uncheck the "Template Locked" option to determine whether all checks created from this template will have their properties automatically synced to any changes made to the template.

For more information about the template state, jump to the "Template Statesection below.

Step 4: Once you have entered all the required fields, click the “Save” button to finalize the template.

Warning

Once a template is saved, the selected rule type becomes locked and cannot be changed.

After clicking the "Save" button, your check template is created, and a success flash message will appear stating, "Check Template successfully created."

After saving the check template, you can now Apply a Check Template to create Quality Checks, which will enforce the validation rules defined in the template across your datastores. This ensures consistent data quality and compliance with the criteria you’ve established.

Template State

Any changes to a template may or may not impact its related checks, depending on whether the template state is locked or unlocked. Managing the template state allows you to control if updates automatically apply to all related checks or let them function independently.

Unlocked

• Quality Checks can evolve independently of the template. Subsequent updates to an unlocked Check Template do not affect its related quality checks

Locked

• Quality Checks from a locked Check Template will inherit changes made to the template. Subsequent updates to a locked Check Template do affect its related quality checks

Info

Tags will be synced independently of unlocked and locked Check Templates, while Description and Additional Metadata will not be synced. This behavior is general for Check Templates.

Template State

graph TD
A[Start] -->|Is `Template Locked` enabled?| B{Yes/No}
B -->|No| E[The quality check can evolve independently]
B -->|Yes| C[They remain synchronized with the template]
C --> D[End]
E --> D[End]