Computed Field vs Computed Container
Qualytics offers two levels of derived data: the Computed Field, which adds a single derived column to an existing container, and the computed containers (Computed Table, Computed File, Computed Join), which produce an entire virtual container from a SQL definition. Both are stored as metadata and evaluated on demand; both are monitored the same way as regular fields and containers. The choice comes down to whether you want to add one derived value alongside a container's native fields, or produce a whole new container.
When a Computed Field fits
A Computed Field is the right choice when:
- You need one or a small number of derived values on top of an existing container.
- The transformation is a type conversion, string cleanup, formatted-numeric conversion, or a Spark SQL expression across the container's own fields.
- The rest of the container's fields should be monitored as-is, and the derived value should sit alongside them in the same field listing.
- Quality checks target the derived value at the field level (completeness on the derived field, comparison of the derived field against another field, and so on).
When a computed container fits instead
Switch to a computed container if:
- You need an entire new container shaped differently from the source (a subset of rows, an aggregation, a join across multiple sources).
- The derived data reshapes multiple columns at once, not one derived value on an otherwise unchanged container.
- You want to monitor a projection that does not exist in the source system as a table, view, or file.
Which computed container flavor to pick:
- Computed Table: source lives in a JDBC warehouse, SQL runs in the warehouse itself.
- Computed File: source lives in a DFS datastore (S3, GCS, Azure Data Lake Storage), Spark SQL runs on Qualytics's own engine.
- Computed Join: source spans two containers, possibly across different datastores.
Side-by-side comparison
| Feature | Computed Field | Computed Table / File / Join |
|---|---|---|
| Scope of the derivation | One derived field on an existing container | An entire virtual container |
| Where the definition lives | On the parent container | As a new container inside the datastore |
| Query language | Spark SQL (for Custom Expression) or a declarative transformation | Warehouse SQL (Computed Table), Spark SQL (Computed File), or join spec (Computed Join) |
| Source scope | Fields on the parent container | Base tables, views, files, or other containers |
| Joins across sources | No | Yes, in Computed Table (within one datastore) or Computed Join (across datastores) |
| Row shaping (filter, aggregate, subset) | No | Yes, via WHERE, GROUP BY, DISTINCT, and so on |
| Materialized on storage | No | No |
| Permissions | Editor on the parent datastore, or Author + ownership | Editor on the parent datastore, or Author + ownership |
Combining them
Nothing prevents mixing the two. A Computed Table can carry its own Computed Fields on top of the columns it projects. If a Computed Table produces a derived container and you also need a Cast or Cleaned Entity Name on top of one of its columns, add the Computed Field to the Computed Table just like you would to a base table.