Computed Files Introduction¶

A Computed File is a Spark SQL-defined virtual container that lives inside a DFS source datastore (Amazon S3, Google Cloud Storage, or Azure Data Lake Storage). You describe the shape of the data you want with a Select Expression over a single base file pattern, and Qualytics stores the definition alongside the datastore's other file containers. The file is not written back to storage; it is evaluated on demand when Qualytics profiles or scans it.

Once created, a Computed File behaves exactly like any base file container: it appears in the tree view, produces a profile, accepts quality checks, and raises anomalies when its data violates one of those checks.

Why Use One¶

Computed Files give you a place to prepare file data for quality analysis without changing anything in the underlying storage. Common scenarios:

• Data preparation and transformation. Clean, shape, or restructure raw files (Parquet, CSV, JSON, Avro, and similar) using Spark SQL.
• Column-level transformations. Normalize, cast, split, or combine columns before scanning with Spark SQL functions.
• Data subsetting. Narrow a large file pattern down to the rows you actually want to monitor with a WHERE clause.
• Aggregations. Compute derived columns or roll-ups with GROUP BY and standard Spark SQL grouping semantics.

If you need to combine two containers into one, use a Computed Join instead. A plain Computed File reads from a single source pattern.

What a Computed File Can Reference¶

A Computed File can only reference base file patterns: physical file patterns already registered under the parent DFS datastore. It cannot reference:

• Other Computed Files inside the same datastore.
• Computed Tables (those live in JDBC and native datastores).
• Computed Joins.
• File patterns outside its parent datastore.
• More than one source pattern at a time.

See the How It Works page for the full referencing matrix and the source-file requirements.

Computed Files vs Computed Tables¶

Computed Files and Computed Tables solve the same problem in different environments: Files target DFS datastores using Spark SQL. Tables target JDBC warehouses using the warehouse's own SQL dialect, and native datastores using Spark SQL. See the Computed File vs Computed Table comparison for a side-by-side.

Next Steps¶

Deep Dive pages covering the mechanics of Computed Files:

See also: