Skip to content

How Computed Fields Work

A Computed Field is metadata stored on a container. Its transformation definition and the source fields it reads live with the container's configuration; the value itself is produced when Qualytics evaluates the transformation during Profile or Scan. Nothing is written back to the source system, and no separate table or view is created.

The Execution Model

The transformation runs inside Qualytics's own execution engine on top of the rows Qualytics reads from the container. The engine uses Spark SQL semantics for casting rules, function names, and expression syntax, which is why every transformation type accepts Spark SQL patterns.

Practical consequences:

  • The value is computed on demand. During Profile the transformation runs against the sample the platform reads; during Scan it runs against the rows Scan pulls to evaluate quality checks. There is no persistent copy of the derived value.
  • Every run reflects the current source. If a source field's value changes between profiles, the next profile of the Computed Field reflects the new input.
  • Source access is required. The Computed Field can only reference fields visible in the container at profile time. If a source field is excluded, missing, or dropped, the transformation cannot resolve it.

When Values Are Computed

The Computed Field is evaluated at two points:

  • Profile. When the container is profiled, Qualytics computes the Computed Field for every sampled row and produces the same field-level statistics you get for a native field (distinct count, min/max length, mean, quartiles, and so on). See Field Profiles for the full metric list.
  • Scan. When a scan runs quality checks that reference the Computed Field, the transformation is applied to the rows the scan inspects. Anomalies raised on the Computed Field appear in the container's anomaly stream alongside anomalies from native fields.

Nothing is materialized between runs. The Computed Field is always as fresh as the most recent profile or scan.

Validation

The Add and Edit modal validates the transformation before Save. What gets validated depends on the transformation type:

  • Cast, Cleaned Entity Name, and Convert Formatted Numeric validate that the source field exists on the container and the target type (for Cast) is one of the supported values.
  • Custom Expression validates the Spark SQL syntax against the container's field schema. Unknown fields, unsupported functions, or malformed expressions surface an inline error on the modal.

If validation succeeds, Save persists the definition and the field appears in the container's field listing right away. The first values populate on the next profile.

Interaction With Sync

Sync discovers the container's schema and its native fields. Computed Fields are declared by you on top of that schema, so Sync never removes a Computed Field itself; only the source fields the transformation reads can change on Sync.

  • If a source field is dropped from the container between syncs, the Computed Field's next profile marks the derived value as Missing. The definition remains, so restoring the source field brings the Computed Field back to life on the next profile.
  • If a source field is renamed at the source, Sync treats the old name as Missing and the new name as a new field. The Computed Field continues to reference the old name until you edit the transformation.

Interaction With Field Exclusion

Excluding a source field via Qualytics also excludes the Computed Field's output, but the definition is preserved so restoring the source field brings the Computed Field back. This is different from deleting a Computed Field, which permanently removes both the definition and the output. See Computed Fields and Exclusion for the exclusion flow.

Editing a Computed Field

Saving an edit to a Computed Field re-registers the definition on the container. Existing quality checks that reference the field are preserved; the field's next profile computes values against the new transformation. Anomalies raised against the previous definition remain in the anomaly history for auditability.

Delete Behavior

Deleting a Computed Field is a hard delete. The definition and the derived output are both removed, and any quality checks that targeted the field are removed with it. Source fields are not affected. See Delete a Computed Field for the full cascade.

Permissions

Managing a Computed Field follows a hybrid gate on the parent datastore: Editor for the full set of actions, or Author when the caller is (or will be) the owner. Reassigning ownership always requires Editor. Users with the Admin role bypass every team-level check. See Permissions for the full action-by-action matrix.