Computed Files Introduction

A Computed File is a Spark SQL-defined virtual container that lives inside a DFS source datastore (Amazon S3, Google Cloud Storage, or Azure Data Lake Storage). You describe the shape of the data you want with a Select Expression over a single base file pattern, and Qualytics stores the definition alongside the datastore's other file containers. The file is not written back to storage; it is evaluated on demand when Qualytics profiles or scans it.

Once created, a Computed File behaves exactly like any base file container: it appears in the tree view, produces a profile, accepts quality checks, and raises anomalies when its data violates one of those checks.

Why Use One

Computed Files give you a place to prepare file data for quality analysis without changing anything in the underlying storage. Common scenarios:

• Data preparation and transformation. Clean, shape, or restructure raw files (Parquet, CSV, JSON, Avro, and similar) using Spark SQL.
• Column-level transformations. Normalize, cast, split, or combine columns before scanning with Spark SQL functions.
• Data subsetting. Narrow a large file pattern down to the rows you actually want to monitor with a WHERE clause.
• Aggregations. Compute derived columns or roll-ups with GROUP BY and standard Spark SQL grouping semantics.

If you need to combine two containers into one, use a Computed Join instead. A plain Computed File reads from a single source pattern.

What a Computed File Can Reference

A Computed File can only reference base file patterns: physical file patterns already registered under the parent DFS datastore. It cannot reference:

• Other Computed Files inside the same datastore.
• Computed Tables (those live in JDBC datastores).
• Computed Joins.
• File patterns outside its parent datastore.
• More than one source pattern at a time.

See the How It Works page for the full referencing matrix and the source-file requirements.

Computed Files vs Computed Tables

Computed Files and Computed Tables solve the same problem in different environments: Files target DFS datastores using Spark SQL; Tables target JDBC warehouses using the warehouse's own SQL dialect. See the Computed File vs Computed Table comparison for a side-by-side.

Next Steps

Deep Dive pages covering the mechanics of Computed Files:

See also: