Profile Operation

The Profile operation examines the containers and records selected in its scope, then records statistics and observed patterns for their fields. Read settings such as thresholds and record limits determine which records are included.

When AI Effort is enabled, Profile uses those observations to generate, validate, and maintain most AI Managed checks. Oversight determines whether new checks become Active automatically or remain Draft for review. These checks provide starting coverage from observed behavior. Data stewards still decide whether they fit the business use of the data and add Authored checks for requirements that Profile cannot infer. Volumetric and Freshness checks are maintained separately from recorded Observability measurements.

This guide explains how to choose the Profile scope, read settings, AI Effort, Oversight, and schedule options.

How Profiling Works

Fields Identification

Profile identifies the fields in the selected containers. The operation scope controls which tables, files, or tagged groups are included.

Statistical Data Gathering

Profile calculates statistics for each field based on its declared or detected data type. Available measures can include minimum and maximum values, mean, standard deviation, completeness, value distributions, and other type-specific statistics. Use these results to understand what was observed in the profiled records.

Metadata Generation

Qualytics records the calculated statistics as profile metadata. When AI Effort is enabled, the observed metadata provides the basis for candidate AI Managed checks.

Data Quality Checks

Qualytics validates candidate AI Managed checks against the data included in Profile before creating them. This validation shows that a candidate fits the profiled data, but it does not establish that the rule reflects every business requirement or future condition. Use Oversight to keep new checks in Draft when they require review before activation.

Navigation to Profile Operation

Step 1: Select a source datastore from the side menu on which you would like to perform the profile operation.

Step 2: Clicking on your preferred datastore will navigate you to the datastore details page. Within the overview tab (default view), click on the Run button under Profile to initiate the profile operation.

Configuration

Step 1: Click on the Run button to initiate the profile operation.

Step 2: Select tables (in your JDBC datastore) or file patterns (in your DFS datastore) and tags you would like to be profiled.

1. All Tables/File Patterns

This option includes all tables or files currently available in the datastore for profiling. Selecting this will profile every table within the source datastore without the need for further selection.

2. Specific

This option allows users to manually select individual tables or files for profiling. It provides the flexibility to focus on particular tables of interest, which can be useful if the user is only interested in a subset of the available data.

3. Tag

This option automatically profiles tables associated with selected tags. Tags are used to categorize tables, and by selecting a specific tag, all tables associated with that tag will be profiled. This option helps in managing and profiling grouped data efficiently.

Step 3: After making the relevant selections, click on the Next button to configure the Operation Settings.

Step 4: Configure the following two Read Settings:

• Starting Threshold
• Record Limit

Starting Threshold

This setting allows users to specify a minimum incremental identifier value to set a starting point for the profile operation. It helps in filtering data from a specific point in time or a particular batch value.

• Greater Than Time: Users can select a timestamp in UTC to start profiling data from a specific time onwards. This is useful for focusing on recent data or data changes since a particular time.

• Greater Than Batch: Users can enter a batch value to start profiling from a specific batch. This option is helpful for scenarios where data is processed in batches, allowing the user to profile data from a specific batch number onwards.

Record Limit

Define the number of records to be profiled per table: This feature allows users to manually enter a custom record limit value using a text field in the profile operation. This setting helps in controlling the scope of the profiling operation, particularly for large datasets, by capping the number of records to analyze.

You can also use a drop-down menu to quickly select from commonly used limits such as 1M, 10M, 100M, and All.

Step 5: After making the relevant selections, click on the Next button to configure the AI Management step.

Step 6: Configure the two sections inside the AI Management step:

• Effort
• Oversight

Effort

The AI Effort setting controls how much work Qualytics AI puts into generating data quality checks during profiling. Selecting a higher level expands the families of checks AI considers; selecting a lower level limits AI to faster, simpler checks (or disables AI generation entirely).

The control is a labeled scale with six clickable markers. Selecting a level shows a plain-language description of what AI does at that level.

Default Configuration

By default, AI Effort is set to Medium, which balances coverage and runtime for most datasets. You can adjust the level at any time based on the depth of analysis you want AI to perform.

AI Effort Levels

The AI Effort scale has six levels. Each higher level includes all the check families from the previous levels and adds new ones, so depth and runtime grow together.

Off

At this level, Profile does not generate or update Profile-generated AI Managed checks. Qualytics still records profile statistics. Use Off when your team wants Profile to collect observations without proposing checks from them. This setting does not disable Observability-managed Volumetric or Freshness checks.

Low: Basic Data Integrity and Simple Value Threshold Checks

The fastest level. Catches basics like missing values, negative numbers, future dates, and unexpected values in known-option fields. Included check families:

• Completeness Checks: Ensure data fields are complete if previously complete.

• Categorical Range Checks: Validate if values fall within a predefined set of categories.

• Non-Negative Numbers: Ensure numeric values are non-negative.

• Non-Future Date/Time: Ensure datetime values are not set in the future.

Use Case: Suitable for datasets where basic integrity checks are sufficient.

The following table shows the AI Managed check families that Profile can generate from observed data at Low.

Medium: Value Range and Pattern Checks

Balanced. Adds checks for unusual value ranges, unexpected text formats, and fields that should be unique. Included check families:

• Date Range Checks: Ensure dates fall within a specified range.

• Numeric Range Checks: Validate that numeric values are within acceptable ranges.

• String Pattern Checks: Ensure strings match specific patterns (e.g., email formats).

• Approximate Uniqueness: Validate uniqueness of values if they are approximately unique.

Use Case: Ideal for datasets where patterns and ranges are important for ensuring data quality.

The following table shows the four additional AI Managed check families Profile can generate at Medium.

High: Time Series and Comparative Relationship Checks

Deeper. Adds checks that spot mismatches between related fields, including time-series patterns and comparative relationships between datasets. Included check families:

• Date Granularity Checks: Ensure the granularity of date values is consistent (e.g., day, month, year).

• Consistent Relationships: Validate that relationships between overlapping datasets are consistent.

Use Case: Suitable for scenarios where data quality depends on time-series data or when comparing data across different datasets.

The following table shows the eight additional AI Managed check families Profile can generate at High.

Extra High: Linear Regression and Cross-Datastore Relationship Checks

Intensive. Adds predictive checks that flag values inconsistent with others in the same row (powered by linear regression models) and validate relationships across different data stores. Included check families:

• Linear Regression Checks: Validate data using regression models to identify trends and outliers.

• Cross-Datastore Relationships: Ensure that data relationships are maintained across different data sources.

Use Case: Best for complex datasets where advanced analytical checks are necessary.

The following table shows the three additional AI Managed check families Profile can generate at Extra High.

Max: Shape Checks

Maximum. Adds fine-grained checks for patterns, completeness, and date precision (uses statistical distribution analysis), validating the shape of distribution patterns that appear in your data. Included check families:

• Shape Checks: Checks that define an expectation for some percentage of your data less than 100%. The property “coverage” holds the percentage of your data for which the expressed check should be true.

Use Case: Ideal for scenarios where each incremental set of scanned data should exhibit the same distributions of values as the training set. For example, a transactions table is configured for a weekly incremental scan after each week’s data is loaded. A shape check could define that 80% of all transactions are expected to be performed using “cash” or “credit”.

The following table shows the three additional AI Managed check families Profile can generate at Max.

Oversight

Use the Activate AI managed checks toggle to control the initial status of newly generated AI Managed checks:

• Toggle ON: Checks are created in Active status.
• Toggle OFF: Checks are created in Draft status, so you can review and activate them manually.

User Overrides on AI Managed Checks

When you edit an AI Managed check's assertion (for example, adjusting its filter or coverage), the platform records the override and continues to honor it on subsequent profile operations. AI will not overwrite your adjustments. Once you take ownership, future profile runs treat the check as Authored and skip regenerating it.

Run Now

Click on the Run Now button, and perform the profile operation immediately.

Schedule

Step 1: Click on the Schedule button to configure the available schedule options in the profile operation.

Step 2: Choose the Timezone for this schedule. UTC is selected by default. To run in a different timezone, type to search by city, region, or abbreviation and pick an IANA timezone from the list. The selected timezone applies to every tab below, and the banner above the tabs shows the current time in that timezone.

Step 3: Set the scheduling preferences for the profile operation.

1. Hourly: This option allows you to schedule the profile operation to run every hour at a specified minute. You can define the frequency in hours and the exact minute within the hour the profiling should start. Example: If set to "Every 1 hour(s) on minute 0," the profile operation will run every hour at the top of the hour (e.g., 1:00, 2:00, 3:00).

2. Daily: This option schedules the profile operation to run once every day at a specific time. You specify the number of days between operations and the exact time of day in the selected timezone. Example: If set to "Every 1 day(s) at 00:00" with the timezone set to UTC, the profile operation will run every day at midnight UTC.

3. Weekly: This option schedules the profile operation to run on specific days of the week at a set time. You select the days of the week and the exact time of day in the selected timezone for the profile operation to run. Example: If configured to run on "Sunday" and "Friday" at 00:00 with the timezone set to UTC, the profile operation will execute at midnight UTC on these days.

4. Monthly: This option schedules the profile operation to run once a month on a specific day at a set time. You specify the day of the month and the time of day in the selected timezone. If set to "On the 1st day of every 1 month(s), at 00:00" with the timezone set to UTC, the profile operation will run on the first day of each month at midnight UTC.

5. Advanced: The advanced section for scheduling operations allows users to set up more complex and custom scheduling using Cron expressions. This option is particularly useful for defining specific times and intervals for profile operations with precision.

Cron expressions are a powerful and flexible way to schedule tasks. They use a syntax that specifies the exact timing of the task based on five fields:

• Minute (0 - 59)
• Hour (0 - 23)
• Day of the month (1 - 31)
• Month (1 - 12)
• Day of the week (0 - 6) (Sunday to Saturday)

Each field can be defined using specific values, ranges, or special characters to create the desired schedule.

Example: For instance, the Cron expression 0 0 * * * schedules the profile operation to run at midnight (00:00) every day. Here’s a breakdown of this expression:

• 0 (Minute) - The task will run at the 0th minute.
• 0 (Hour) - The task will run at the 0th hour (midnight).
• *(Day of the month) - The task will run every day of the month.
• *(Month) - The task will run every month.
• *(Day of the week) - The task will run every day of the week.

Users can define other specific schedules by adjusting the Cron expression. For example:

• 0 12 * * 1-5 - Runs at 12:00 PM from Monday to Friday.
• 30 14 1 * * - Runs at 2:30 PM on the first day of every month.
• 0 22 * * 6 - Runs at 10:00 PM every Saturday.

To define a custom schedule, enter the appropriate Cron expression in the Custom Cron Schedule field before specifying the schedule name. The field label shows the abbreviation for the currently selected timezone (for example, Custom Cron Schedule (UTC) or Custom Cron Schedule (EST)), and the cron fields are interpreted in that timezone.

Step 4: Define the Schedule Name to identify the scheduled operation at the running time.

Step 5: Click on the Schedule button to activate your profile operation schedule.

Field Masking and Profiling

Profiling runs normally on masked fields. Masking does not affect the collection of statistical metadata. The platform continues to compute all profile metrics (distinct values, min/max, mean, standard deviation, completeness, etc.) using the actual source data.

However, when profile results are displayed or exported, masked field values are obfuscated:

• Field Profile Histograms in the UI show obfuscated bucket values for masked fields
• Exported Field Profiles (_field_profiles_export) written to the enrichment datastore contain obfuscated histogram values for masked fields

Operation Insights

When the profile operation is completed, you will receive the notification and can navigate to the Activity tab for the datastore on which you triggered the Profile Operation and learn about the operation results.

Top Panel

1. Runs (Default View): Provides insights into the operations that have been performed

2. Schedule: Provides insights into the scheduled operations.

3. Search: Search any operation (including profile) by entering the operation ID

4. Sort by: Organize the list of operations based on the Created Date or the Duration.

5. Filter: Narrow down the list of operations based on:

6. Type: the kind of operation, such as Sync, Profile, or Scan

7. Status: the operation result, such as Success, Failure, Running, Queued, or Aborted
8. Table: the container the operation ran on. On DFS datastores this row is labeled File
9. Has logs: a toggle that narrows the list to operations that recorded log messages

Activity Heatmap

The activity heatmap shown in the snippet below represents activity levels over a period, with each square indicating a day and the color intensity representing the number of operations or activities on that day. It is useful in tracking the number of operations performed on each day within a specific timeframe.

Operation Detail

Queued

This status indicates that the profile operation was submitted and is waiting to be picked up for processing. The operation stays in Queued while earlier work finishes, then moves to Running automatically once processing begins. A profile operation having a queued status reflects the following details and actions:

• The progress bar shows how long the operation has been waiting (for example, Queued for 2 minutes).
• The Abort button lets you stop the operation before it starts.
• You can filter the operations list by the Queued status.

Running

This status indicates that the profile operation is still running at the moment and is yet to be completed. A profile operation having a running status reflects the following details and actions:

Aborted

This status indicates that the profile operation was manually stopped before it could be completed. A profile operation having an aborted status reflects the following details and actions:

Failure

This status indicates that the profile operation failed due to unrecoverable errors such as connection timeouts, refused connections, or permission issues. A profile operation having a failure status reflects the following details and actions:

Success with Warning

This status signals that the profile operation completed but encountered some issues, and displays the logs that facilitate improved tracking of the blockers and issue resolution. A profile operation having a success with warning status reflects the following details and actions:

Success

This status confirms that the profile operation was completed successfully without any issues. A profile operation having a success status reflects the following details and actions:

Full View of Metrics in Operation Summary

Users can now hover over abbreviated metrics to see the full value for better clarity. For demonstration purposes, we are hovering over the Records Profiled field to display the full value.

Post Operation Details

Step 1: Click on any of the successful Profile Operations from the list and hit the Results button. This opens the Results tab of the operation page.

Step 2: The Results tab lists the containers covered by the operation. You can search containers by name, sort the list, and use the Filter results menu to narrow it down by container type or tag. Turn on the Not analyzed filter to see only the containers that were not profiled.

The Results tab also provides two analysis options for you:

• Details for a Specific Container (Container's Profile)
• Details for a Specific Field of a Container (Field Profile)

Expand a profiled container from the Results tab by clicking on the arrow icon.

For detailed information about container quality metrics, observability charts, and profiling history, see Container Attributes. For field-level profile statistics such as distinct values, mean, median, and distribution metrics, see Field Profiles.

API Payload Examples

This section provides payload examples for initiating and checking the running status of a profile operation. Replace the placeholder values with data specific to your setup.

Running a Profile Operation

To run a profile operation, use the API payload example below and replace the placeholder values with your specific values:

Endpoint (Post): /api/operations/run (post)

The profile request accepts the following AI-related fields:

• ai_effort (string): Controls how much work AI puts into authoring checks. Accepted values: "off", "low", "medium", "high", "xhigh", "max".
• infer_as_draft (boolean): When true, AI Managed checks are created in Draft state for review. When false (or omitted), AI Managed checks are activated automatically.

Option I: Running a profile operation of all containers

• container_names: []: This setting indicates that profiling will encompass all containers.

• max_records_analyzed_per_partition: null: This setting implies that all records within all containers will be profiled.

• ai_effort: "max": AI will generate checks across every supported family (equivalent to the legacy inference_threshold: 5).

{
    "type": "profile",
    "datastore_id": "datastore-id",
    "container_names": [],
    "max_records_analyzed_per_partition": null,
    "ai_effort": "max",
    "infer_as_draft": false
}

Option II: Running a profile operation of specific containers

• container_names: ["table_name_1", "table_name_2"]: Profiling will only cover the tables named table_name_1 and table_name_2.

• max_records_analyzed_per_partition: 1000000: Up to 1 million rows per container will be profiled.

• ai_effort: "off": AI will not generate any checks for this run (equivalent to the legacy inference_threshold: 0).

{
    "type": "profile",
    "datastore_id": "datastore-id",
    "container_names": [
        "table_name_1",
        "table_name_2"
    ],
    "max_records_analyzed_per_partition": 1000000,
    "ai_effort": "off"
}

Scheduling a Profile Operation

Below is a sample payload for scheduling a profile operation. Please substitute the placeholder values with the appropriate data relevant to your setup.

Endpoint (Post): /api/operations/schedule (post)

INFO: This payload is to run a scheduled profile operation every day at 00:00 in the America/New_York timezone.

Scheduling profile operation of all containers

• timezone: "America/New_York": The schedule follows this timezone and adjusts automatically for daylight saving time. When omitted, the schedule runs in UTC.

• trigger_type: "operation": Identifies this as an operation schedule. Include this field in the payload, since the endpoint uses it to determine the request type.

{
    "type": "profile",
    "name": "My scheduled Profile operation",
    "datastore_id": "datastore-id",
    "trigger_type": "operation",
    "container_names": [],
    "max_records_analyzed_per_partition": null,
    "ai_effort": "max",
    "infer_as_draft": false,
    "crontab": "0 0 * * *",
    "timezone": "America/New_York"
}

Retrieving Profile Operation Status

To retrieve the profile operation status, use the API payload example below and replace the placeholder values with your specific values:

Endpoint (Get): /api/operations/{id} (get)

{
    "items": [
        {
            "id": 12345,
            "created": "YYYY-MM-DDTHH:MM:SS.ssssssZ",
            "type": "profile",
            "start_time": "YYYY-MM-DDTHH:MM:SS.ssssssZ",
            "end_time": "YYYY-MM-DDTHH:MM:SS.ssssssZ",
            "result": "success",
            "message": null,
            "triggered_by": "user@example.com",
            "datastore": {
                "id": 101,
                "name": "Sample-Store",
                "store_type": "jdbc",
                "type": "db_type",
                "enrichment_only": false,
                "enrichment_prefix": "data_prefix",
                "favorite": false
            },
            "schedule": null,
            "ai_effort": "max",
            "max_records_analyzed_per_partition": -1,
            "max_count_testing_sample": 100000,
            "histogram_max_distinct_values": 100,
            "greater_than_time": null,
            "greater_than_batch": null,
            "percent_testing_threshold": 0.4,
            "high_correlation_threshold": 0.5,
            "status": {
                "total_containers": 2,
                "containers_analyzed": 2,
                "containers_succeeded": 2,
                "partitions_analyzed": 2,
                "records_processed": 1126,
                "fields_profiled": 9,
                "checks_synchronized": 26,
                "checks_added": 20,
                "checks_updated": 6
            },
            "containers": [
                {
                    "id": 123,
                    "name": "Container1",
                    "container_type": "table",
                    "table_type": "table"
                },
                {
                    "id": 456,
                    "name": "Container2",
                    "container_type": "table",
                    "table_type": "table"
                }
            ],
            "container_profiles": [
                {
                    "id": 789,
                    "created": "YYYY-MM-DDTHH:MM:SS.ssssssZ",
                    "parent_profile_id": null,
                    "container": {
                        "id": 456,
                        "name": "Container2",
                        "container_type": "table",
                        "table_type": "table"
                    },
                    "records_count": 550,
                    "records_processed": 550,
                    "checks_synchronized": 11,
                    "checks_added": 8,
                    "checks_updated": 3,
                    "field_profiles_count": 4,
                    "result": "success",
                    "message": null
                },
                {
                    "id": 790,
                    "created": "YYYY-MM-DDTHH:MM:SS.ssssssZ",
                    "parent_profile_id": null,
                    "container": {
                        "id": 123,
                        "name": "Container1",
                        "container_type": "table",
                        "table_type": "table"
                    },
                    "records_count": 576,
                    "records_processed": 576,
                    "checks_synchronized": 15,
                    "checks_added": 12,
                    "checks_updated": 3,
                    "field_profiles_count": 5,
                    "result": "success",
                    "message": null
                }
            ],
            "tags": []
        }
    ],
    "total": 1,
    "page": 1,
    "size": 50,
    "pages": 1
}

Troubleshooting

Unloadable Container Error

A profile operation may surface the following error for specific containers:

Container '<CONTAINER_NAME>' is marked as Unloadable. No attempt was made to load the container due to multiple consecutive failures in prior operations.

This happens after a container has failed in 3 consecutive scan or profile operations. Qualytics then skips it in subsequent profiles, scans, and check validations until the status is cleared.

• For tables, views, and file patterns, run a Sync Operation on the datastore to reset the status.
• For computed assets (Computed Tables, Computed Files, Computed Joins), force an edit on the asset (click Edit), then click Validate and Save to re-evaluate the definition.

See Unloadable Container Error for the full resolution steps and common root causes.