Cost and Performance
A Computed Field is metadata attached to a container. Nothing is materialized: the transformation runs against the rows Qualytics reads during Profile and Scan. This page walks through where the cost of a Computed Field lives, how transformation choice affects it, and when to move logic out of the Computed Field.
Where the Cost Lives
Every operation that inspects the container evaluates the transformation for the rows it reads:
- Profile: the transformation runs for every row in the profile sample, and the derived value is included in the field-level statistics (distinct count, min/max, mean, quartiles).
- Scan: for each quality check that references the Computed Field, the transformation runs against the rows the scan pulls to evaluate the check.
- Preview: sampling rows from the container evaluates the transformation for the sampled rows.
There is no storage cost. The derived value is never persisted between runs; the next profile or scan recomputes it against the current source values.
Transformation Type Determines Cost
Different transformation types have different per-row costs:
| Transformation | Per-row cost | Notes |
|---|---|---|
| Cast | Essentially free | Type conversion is a lightweight operation. |
| Cleaned Entity Name | Low | String pattern removal on a single field. Cost scales linearly with the row count. |
| Convert Formatted Numeric | Low | Regex-style character stripping on a single field. Similar profile to Cleaned Entity Name. |
| Custom Expression | Variable | Depends on the expression. Simple arithmetic and CASE statements are cheap; window functions (LEAD, LAG, ROW_NUMBER OVER (...)) and aggregate expressions add sort or partition work on top of the row read. |
Signs a Computed Field Is Slowing a Profile Down
- Profile duration climbs after a Computed Field is added, especially one with a Custom Expression using window functions.
- A container that used to profile in seconds now takes minutes.
- Scans that reference the Computed Field stretch past their usual window.
Ways to Cut Cost
- Prefer specialized transformations. Reach for Cast, Cleaned Entity Name, or Convert Formatted Numeric before writing a Custom Expression that reproduces the same logic. The specialized transformations are optimized paths.
- Simplify Custom Expressions. If the same result is achievable with a straightforward arithmetic expression instead of a window function, prefer the simpler form. Reserve window and aggregate expressions for cases they are genuinely needed.
- Push heavy transformations upstream. If the derived value is used by many quality checks and the transformation is expensive, consider producing the value in the source system (SQL warehouse view, dbt model, ETL step) and letting Qualytics read it as a native field.
- Limit the number of Computed Fields per container. Each Computed Field runs its own transformation during Profile. Ten simple Cast fields are fine; ten window-function Custom Expressions add substantial per-profile cost.
When to Move the Logic Somewhere Else
If a Computed Field is:
- reused across many containers or reports,
- expensive to compute (window function, aggregate, or complex conditional),
- and stable enough that the source system could produce it,
it is usually cheaper to build the derived value in the source system (a warehouse view, a dbt column, an ETL step) and let Qualytics see it as a native field. Reserve Computed Fields for values that are unique to Qualytics's monitoring flow, or for values whose transformation logic changes often.