Skip to content

Flows

Flows enable users to create pipelines by chaining actions and configuring how they are triggered. Triggers can be set based on predefined events and filters, offering a flexible and efficient way to automate processes. These actions can be notifications or operations, allowing users to inform various notification channels or execute tasks based on specific operations.

Step 1: Log in to your Qualytics account and click on the Flows on the left side panel of the interface.

flows flows

You will navigate to the Flows interface, where you can add and manage flows. At the top, you will see two tabs:

  • Definitions: Displays a list of all flows along with details like triggers, actions, tags, and the last triggered time.

definition definition

  • Executions: Provides the execution history of flows, including their status and timestamps.

execution execution

Add Flow

Step 1: Click on the Add Flow button from the top right corner.

addflow addflow

A modal window, Add Flow, will appear, providing options to create a flow. Each flow starts by default with two nodes: Flow and Trigger.

flowchart flowchart

Flow

Step 1: Click on the Flow node.

flow flow

A panel will appear on the right-hand side, allowing you to:

No. Field Name Description
1. Name Enter the name for the flow.
2. Description Provide a brief description of the flow (optional) to clarify its purpose or functionality.
3. Deactivated Check the box to deactivate the flow. If selected, the flow won't start even if the trigger conditions are met.

flow flow

Step 2: Once the details are filled in, click the Save button to save the flow settings.

save save

Trigger

Step 1: After completing the "Flow" node setup, users can click on the "Trigger" node.

trigger trigger

A panel will appear on the right-hand side, enabling users to define when the flow should start. The panel provides four options for initiating the flow. Users can choose one of the following options:

  • Operation Completes.

  • Anomalous Table and File Detection.

  • Anomaly Detected.

  • Manual

triggersetting triggersetting

Operation Completes

This type of flow is triggered whenever an operation, such as a catalog, profile, or scan, is completed on a source datastore. Upon completion, teams are promptly notified through in-app messages and, if configured, via external notification channels such as email, Slack, Microsoft Teams, and others. For example, the team is notified whenever the catalog operation is completed, helping them proceed with the profile operation on the datastore.

operation operation

Filter Conditions

Filters can be set to narrow down which operations should trigger the flow execution:

  1. Source Datastore Tags: The flow is triggered only for source datastores that have all the selected tags assigned.

  2. Source Datastores: The flow is triggered only for the selected source datastores.

  3. Operation Types: The flow is triggered only for operations that match one or more of the selected types.

  4. Operation Status: The flow is triggered for operations with a status of either Success or Failure.

operation operation

After defining the conditions, users must click the Save button to finalize the trigger configuration.

save save

Anomalous Table and File Detected

This flow is triggered when anomalies are detected within a specific table, file and check rule types. It includes information about the number of anomalies found and the specific scan target within the datastore. This is useful for assessing the overall health of a particular datastore.

table table

Filter Conditions

Users can optionally set filters to specify which tables or files should trigger the flow execution.

  1. Tables / Files Tags: Only tables or files with all the selected tags assigned will trigger the flow.

  2. Source Datastores: The flow is triggered only for the selected source datastores.

  3. Check Rule Types: Only anomalies identified by one or more of the selected check rule types will initiate the flow.

table table

After defining the conditions, users must click the Save button to finalize the trigger configuration.

save save

Anomaly Detected

This type of flow is triggered when any single anomaly is identified in the data. The flow message typically includes the type of anomaly detected and the datastore where it was found. It provides specific information about the anomaly type, which helps quickly understand the issue's nature.

anomaly anomaly

Filter Condition

Users can define specific conditions to determine when the flow should be initiated.

  1. Anomaly’s Tags: Only anomalies with all selected tags assigned will trigger the flow.

  2. Source Datastores: Only triggered when anomalies are detected in the selected datastores.

  3. Check Rule Types: Only anomalies identified by one or more of the selected check rule types will initiate the flow.

  4. Anomaly Weight (Min): Only anomalies with a weight equal to or greater than the specified value will trigger the flow.

anomaly anomaly

Step 2: Once the filter conditions are set, users must click the Save button to finalize the configuration.

save save

Manual

The flow starts only when the user manually triggers it. It doesn’t depend on any automatic conditions or detections, giving the user full control.

manual manual

Once selected, users must click the Save button to confirm the manual trigger configuration.

save save

Actions

Actions define the specific steps the system will execute after a flow is triggered. They allow users to automate tasks, send notifications, or interact with external systems.

Step 1: After completing the "Trigger" node setup, users can click on the "Actions" node.

action action

A panel will appear on the right-hand side displaying the list of available actions. These actions define what the system will execute after the flow is triggered. The actions are categorized into three groups:

  • Operations.

  • Notifications.

  • HTTP.

actionlist actionlist

Operation

Users can execute specific operations when the trigger activates. They can choose from the following options:

  • Catalog.

  • Profile.

  • Scan.

operations operations

Catalog

Step 1: Click on Catalog.

catalog catalog

A panel Catalog Settings will appear on the right-hand side, This window allows you to configure the catalog operation.

No. Field Description
1. Source Datastore Select the source datastore to catalog.
2. Prune Checkbox to enable or disable the removal of named collections (tables, views, files, etc.) that no longer exist in the datastore.
3. Recreate Checkbox to enable or disable the recreation of previously deleted named collections in Qualytics for the catalog.
4. Include Checkboxes to select Tables, Views, or both, specifying the resources to include in the catalog.

catalog catalog

Step 2: After configuring the settings, click Save to apply and proceed with the catalog operation.

save save

Profile

Step 1: Click on Profile.

profile profile

A panel Profile Settings will appear on the right-hand side, This window allows you to configure the Profile operation.

profile profile

No. Field Description
1. Source Datastore Select the source datastore to catalog.
2. Select Tables Allows users to select all tables, specific tables, or tables associated with selected tags to profile.
3. Read Settings Configure the starting point for profiling and set a maximum record limit per table for profiling.
4. Inference Settings Set the level of automated checks and decide whether inferred checks should be saved in draft mode.

profile profile

Step 2: Click Save to finalize the profile configuration.

save save

Scan

Step 1: Click on Scan.

scan scan

A panel Scan Settings will appear on the right-hand side, This window allows you to configure the Scan operation.

scan scan

Source Datastore: Select the datastore to be scanned.

scan scan

Selected Tables: Choose all tables, specific tables, or tables associated with selected tags to include in the scan .

scan scan

Read Settings: Define the scan strategy: incremental scans updated records; full scans all records.

scan scan

Starting Threshold: Set a starting point for scanning based on an incremental identifier.

scan scan

Record Limit: Specify the maximum number of records to scan per table.

scan scan

Scan Settings: Select categories of checks to include, such as table properties (Metadata) or value checks (Data Integrity).

scan scan

Anomaly Options: Automatically archive duplicate anomalies from previous scans.

scan scan

Enrichment Settings: Select a remediation strategy for anomalies and source tables: no replication, append, or overwrite.

scan scan

Source Record Limit: Define the number of source records to include in the enrichment operation

scan scan

Step 2: Click Save to finalize the scan configuration.

save save

Notification

Users can configure the application to send notifications through various channels. The available notification options include:

  • In App.

  • Email.

  • Slack.

  • Microsoft Teams.

  • PagerDuty.

notification notification

In App

This will send an app notification to all users that use Qualytics. Users can set a custom message using variables and modify the standard text.

Step 1: Click on In App.

notification notification

A panel In App Settings will appear on the right-hand side, allowing you to configure the notification message.

notification notification

Message: Enter your custom message using variables in the Message field, where you can specify the content of the notification that will be sent out.

notification notification

Tip

You can write your custom notification message by utilizing the autocomplete feature. This feature allows you to easily insert internal variables such as {{ flow_name }}, {{ container_name }}, and {{ datastore_name }}. As you start typing, the autocomplete will suggest and recommend relevant variables in the dropdown.

Step 2: After configuring the message, click Save to finalize the settings.

save save

Email

Adding email notifications allows users to receive timely updates or alerts directly in their inbox. By setting up notifications with specific triggers and channels, you can ensure that you are promptly informed about critical events, such as operation completions or detected anomalies. This proactive approach allows you to take immediate action when necessary, helping to address issues quickly and maintain the smooth and efficient operation of your processes.

Step 1: Click on Email.

notification notification

A panel Email Settings will appear on the right-hand side, allowing you to add email address and configure the notification message.

notification notification

No. Field Description
1. Email Address Enter the email address where the notification should be sent.
2. Message Text area to customize the notification message content with dynamic placeholders like {{ flow_name }}, {{ operation_type }}, and {{ operation_result }}.

notification notification

Step 2: Click the Test Notification button to send a test email to the provided address. If the email is successfully sent, you will receive a confirmation message indicating Notification successfully sent.

test test

Step 3: Once all fields are configured, click the Save button to finalize the email notification setup.

save save

Slack

Slack notifications keep your team informed by sending updates directly to your Slack workspace. With a webhook, you can deliver customized messages to specific channels, including key details about events and operations.

Step 1: Click on Slack.

notification notification

A panel Slack Settings will appear on the right-hand side, allowing you to add a webhook url and configure the notification message.

notification notification

No. Field Description
1. Webhook URL Enter the webhook url where the notification should be sent.
2. Message Text area to customize the notification message content with dynamic placeholders like {{ flow_name }}, {{ operation_type }}, and {{ operation_result }}.

notification notification

Step 2: Click the "Test Notification" button to send a test message to the provided Webhook URL. If the message is successfully sent, you will receive a confirmation notification indicating "Notification successfully sent".

test test

Step 3: Once all fields are configured, click the Save button to finalize the slack notification setup.

save save

Microsoft Teams

Step 1: Click on Microsoft Teams.

notification notification

A panel Microsoft Teams Settings will appear on the right-hand side, allowing you to add a webhook url and configure the notification message.

notification notification

No. Field Description
1. Teams Webhook URL Enter the teams webhook url where the notification should be sent.
2. Message Text area to customize the notification message content with dynamic placeholders like {{ flow_name }}, {{ operation_type }}, and {{ operation_result }}.

notification notification

Step 2: Click the "Test Notification" button to send a test message to the provided “Webhook URL”. If the message is successfully sent, you will receive a confirmation notification indicating "Notification successfully sent".

test test

Step 3: Once all fields are configured, click the Save button to finalize the microsoft teams notification setup.

save save

PagerDuty

Integrating PagerDuty with Qualytics ensures that your team gets instant alerts for critical data events and system issues. With this connection, you can automatically receive real-time notifications about anomalies, operation completions and other important events directly in your PagerDuty account. By categorizing alerts based on severity, it ensures the right people are notified at the right time, speeding up decision-making and resolving incidents efficiently. This helps your team respond quickly to issues, reducing downtime and keeping data operations on track.

Step 1: Click on PagerDuty.

notification notification

A PagerDuty Settings panel will appear on the right-hand side, enabling users to configure and send PagerDuty notifications.

notification notification

Integration Key: Enter the Integration Key where you want the notification to be sent.

notification notification

Severity: Select the appropriate PagerDuty severity level to categorize incidents based on their urgency and impact. The available severity levels are:

  • Info: For informational messages that don't require immediate action but provide helpful context.

  • Warning: For potential issues that may need attention but aren't immediately critical.

  • Error: For significant problems that require prompt resolution to prevent disruption.

  • Critical: For urgent issues that demand immediate attention due to their severe impact on system operations.

notification notification Message: Enter your custom message using variables in the Message field, where you can specify the content of the notification that will be sent out.

notification notification

Tip

You can write your custom notification message by utilizing the autocomplete feature. This feature allows you to easily insert internal variables such as {{ flow_name }}, {{ operation_type }}, and {{ datastore_name }}. As you start typing, the autocomplete will suggest and recommend relevant variables in the dropdown. |

Step 2: Click on the Test notification button to check if the integration key is functioning correctly. Once the test notification is sent, you will see a success message, "Notification successfully sent."

test test

Step 3: Once you have entered all the values, then click on the Save button.

save save

HTTP

User can connect to external apps for notifications using one of these services:

  • Webhook.

  • HTTP Action.

notification notification

Webhook

Qualytics allows you to connect external apps for notifications using webhooks, making it easy to stay updated in real time. When you set up a webhook, it sends an instant alert to the connected app whenever a specific event or condition occurs. This means you can quickly notify about important events as they happen and respond right away. By using webhook notifications, you can keep your system running smoothly, keep everyone informed, and manage your operations more efficiently.

Step 1: Click on Webhook.

notification notification

A Webhook Settings panel will appear on the right-hand side, enabling users to configure and send PagerDuty notifications.

notification notification

No. Field Description
1. Webhook URL Enter the desired "Webhook URL" of the target system where you want to receive notifications.
2. Message Text area to customize the notification message content with dynamic placeholders like {{ flow_name }}, {{ operation_type }}, and {{ operation_result }}.

notification notification

Step 2: Click on the "Test HTTP" button to send a test notification to the webhook URL you provided. If the webhook URL is correct, you will receive a confirmation message saying "Notification successfully sent." This indicates that the webhook is functioning correctly.

test test

Step 3: Once you have entered all the values, then click on the Save button.

save save

HTTP Action

Integrating HTTP Action notifications allows users to receive timely updates or alerts directly to a specified server endpoint. By setting up HTTP Action notifications with specific trigger conditions, you can ensure that you are instantly informed about critical events, such as operation completions or anomalies detected. This approach enables you to take immediate action when necessary, helping to address issues quickly and maintain the smooth and efficient operation of your processes.

Step 1: Click on HTTP Action.

notification notification

A HTTP Action Settings panel will appear on the right-hand side, enabling users to configure and send HTTP Action notifications.

notification notification

Step 2: Enter the following detail where you want the notification to be sent.

1. Action URL: Enter the “Action URL” in this field, which specifies the server endpoint for the HTTP request, defining where data will be sent or retrieved. It must be correctly formatted and accessible, including the protocol (http or https), domain, and path.

2. HTTP Verbs: HTTP verbs specify the actions performed on server resources. Common verbs include:

  • POST: Use POST to send data to the server to create something new. For example, it's used for submitting forms or uploading files. The server processes this data and creates a new resource.
  • PUT: Updates or creates a resource, replacing it entirely if it already exists. For example, updating a user’s profile information or creating a new record with specific details.
  • GET: Retrieves data from the server without making any modifications. For example, requesting a webpage or fetching user details from a database.

3. Username: Enter the username needed for authentication.

4. Auth Type: This field specifies how to authenticate requests. Choose the method that fits your needs:

  • Basic: Uses a username and password sent with each request. Example: “Authorization: Basic ”.
  • Bearer: Uses a token included in the request header to access resources. Example: “Authorization: Bearer ”.
  • Digest: Provides a more secure authentication method by using a hashed combination of the username, password, and request details. Example: Authorization: Digest username=" ", realm=" ", nonce=" ", uri=" ", response=" ".

5. Secret: Enter the password or token used for authentication. This is paired with the Username and Auth Type to securely access the server. Keep the secret confidential to ensure security.

6. Message: Enter your custom message using variables in the Message field, where you can specify the content of the notification that will be sent out.

notification notification

Tip

You can write your custom notification message by utilizing the autocomplete feature. This feature allows you to easily insert internal variables such as {{ flow_name }}, {{ operation_type }}, and {{ datastore_name }}. As you start typing, the autocomplete will suggest and recommend relevant variables in the dropdown. |

Step 3: Click the "Test HTTP" button to verify the correctness of the Action URL. If the URL is correct, a confirmation message saying "Notification successfully sent" will appear, confirming that the HTTP action is set up and functioning properly.

test test

Step 4: Once you have entered all the values, then click on the Save button.

save save

Step 3: After completing all the required details in the "Add Flow" section, click on the Publish button to finalize the process.

publish publish

Step 4: Once published, a success message will appear, stating, "Your flow has been successfully added.

sucess sucess

View Created Flows

Once a flow is added, it will be visible in the Definitions tab, where you can view all the created flows.

panel panel

Clone a Flow

Users can duplicate existing flows to simplify the reuse and modification of flow configurations for similar scenarios.

Step 1: Click on the existing flow you want to clone.

panel panel

Step 2: A new window will open displaying the flow's detailed configuration. Click the settings icon and select Clone.

panel panel

Step 3: After selecting the clone button. Click the Publish button to publish it.

panel panel

Once published, a success message will appear: "Your flow has been successfully added" .

panel panel

Sort Flows

Qualytics allows you to sort your flows by Created Date and Name to easily organize and prioritize them according to your needs.

sort sort

Whatever sorting option is selected, you can arrange the data either in ascending or descending order by clicking the caret button next to the selected sorting criteria.

sort sort

Execute Manual Flows

Users can start a manual flow from the vertical ellipsis menu for greater flexibility in executing flows.

Step 1: Locate the manual flow in your list of flows.

manual-flow manual-flow

Step 2: Click the vertical ellipsis (⋮) next to the manual flow you wish to execute, then select "Execute" from the dropdown menu to trigger the flow.

manual-flow manual-flow

After clicking the execute button, a confirmation message will appear saying The Flow has been successfully executed.

sort sort

Manage Flows

Manage Flow allows users to edit, delete, or deactivate flows. Users can update configurations, remove outdated flows, or pause triggers to maintain an organized and efficient workflow system.

Edit Flow

Edit Flow feature lets users update existing flows by modifying configurations or adding actions.

Step 1: Click the flow you want to edit.

panel panel

Step 2: After clicking the flow, a new window will open displaying the flow's detailed configuration. Click on the boxes you want to edit.

For demonstration purposes we have selected the Flow node.

result result

Step 3: Click the Save button to apply the updates.

save save

Step 4: After clicking the Save button, click the Publish button located in the top right corner to finalize and publish the changes.

publish publish

Delete Flow

Delete Flow feature allows you to permanently remove unwanted or outdated flows from the system. This helps in maintaining a clean and organized list of active flows.

Step 1: Click the vertical ellipsis (⋮) next to the flows that you want to delete, then click on Delete from the dropdown menu.

delete delete

After clicking the delete button, a confirmation modal window Delete Flow will appear.

delete delete

Step 2: Click on the Delete button to delete the flow.

delete delete

After clicking on the Delete button, your flow will be deleted and a confirmation message will display saying The flow has been successfully deleted.

delete delete

Deactivate Flow

User can deactivate a flow to pause its triggers by simply disabling it. This prevents the flow from being executed until it is reactivated.

Step 1: Click the vertical ellipsis (⋮) next to the flows that you want to deactivate, then click on Deactivate from the dropdown menu.

deactivate deactivate

After clicking the deactivate button, a confirmation message will appear saying The Flow has been successfully deactivated.

deactivate deactivate

Step 2: Click on the Delete button to delete the flow.

delete delete

After clicking on the Delete button, your flow will be deleted and a confirmation message will display saying The flow has been successfully deleted.

delete delete

Flows Execution

Execution tab allows users to view the execution history and current status of a flow. It provides detailed timestamps, status updates, and a comprehensive record of flow executions for efficient tracking and analysis.

Click on the Execution tab.

executions executions

You will be navigated to the Execution tab, where you can view the complete execution history of all created flows.

executions executions

See a Flow Execution

Users can view flow execution in real-time by clicking on the desired flow operation. The page shows detailed operations but does not allow editing.

Step 1: Click on the flow operation you want to view.

manual-flow manual-flow

After clicking, the user will navigate to the selected flow operation details.The page displays all operational details in real-time. Note that this page is for viewing only, and no edits can be made here.

flow flow

Understanding Flow States

On the bottom-right corner, there is a Legend indicating the possible states of an action, such as:

  • Success (Green)

  • Failure (Red)

  • Aborted (Orange)

  • Skipped (Yellow)

  • Running (Blue with dotted lines animation)

  • Pending (Gray)

chart chart

If a step is running, you will see a dot-line animation, signaling that the step is in progress.
Once completed, the Action box will change its color to reflect the final state.

chart chart

Accessing Operation Results

To view detailed results of specific operations:

Step 1: Click the Top Right Arrow button within the action operation box.

chart chart

Step 2: You will navigate to the Activity page, where a Result Modal will open, displaying in-depth details of the operation.

result result

Delete Flow Execution

Step 1: Click the Delete icon next to the flow execution you want to remove.

delete delete

A confirmation modal window Delete Flow Execution will appear.

delete delete

Step 2: Click on the Delete button to delete the flow execution.

delete delete

After clicking on the Delete button, your flow execution will be deleted and a confirmation message will display saying The flow execution has been successfully deleted.

delete delete

Filter and Sort

Filter and Sort in the Executions tab help organize flow execution data. Users can sort by creation date or duration and filter by flow name, status, or trigger type for quick access to specific details.

Sort

Sort By feature allows users to organize executions by Created Date or Duration, simplifying the process of reviewing flow executions based on their creation or runtime.

sort sort

Filter

Filter feature allows users to refine flow execution results based on specific criteria. By clicking the filter icon, users can choose from the following options:

No. Filter Description
1. Flows Select a specific flow to view its executions.
2. Status Filter executions by their completion status (e.g., success, failure and running).
3. Trigger When Filter executions based on their trigger condition.

filter filter

Operations

In the Activity tab, users can easily identify flow executions. The Flow column shows the flow name and includes a button to redirect users to the flow's operation. This feature is available in Explore Activities, Datastore Activity, and Container Activity.

explore explore