Referencing

A Computed File is built on top of a single base file container in the parent DFS datastore. This page covers what that source is allowed to be, how to work around the "one source only" restriction, and how a Computed File can compose with a Computed Join.

What a Computed File Can Reference

A Computed File can only reference base file patterns in the parent datastore:

The rule exists because Qualytics stores the Computed File's clauses as metadata; it does not create a new file on storage. The analytical engine only ever reads from the one source_container_id you selected. See Workarounds below for two ways to structure your expression when you would otherwise need to chain Computed Files.

Qualytics also requires the source file container to be profiled at least once before you can create a Computed File on top of it. The Select Expression, filter, and group-by must reference field names, and those field names are only known after the source has been profiled. If the source is not yet profiled, the save is rejected with an explicit error; run Scan on the source once (or wait for its next scheduled profile) and then come back to the Add Computed File modal.

Workarounds for Referencing Other Computed Files

Because a Computed File cannot reference another Computed File, and the form accepts only clause fragments (not a full SQL statement with CTEs or subqueries in the FROM position), chained transformation logic has to be expressed either inside a single set of clauses or by landing intermediate results upstream.

Fold Steps into a Single Select Expression

Push the projections, conditional logic, and window functions the intermediate would provide directly into select_clause, where_clause, and group_by_clause. Spark SQL functions like CASE WHEN, COALESCE, MAX_BY, COLLECT_LIST, and column references to fields produced by lateral views cover most cases where a reader would reach for a CTE.

Materialize the Intermediate as a Base File Pattern

If you need to reuse the intermediate result across multiple Computed Files, land it as a real file (or file pattern) on DFS storage in your own upstream pipeline, catalog it in Qualytics as a base file container, and reference the cataloged pattern from your Computed File clauses.

Using Computed Files in a Computed Join

A Computed File can be used as an input to a Computed Join as long as it has been profiled at least once. The Computed Join picks up the Computed File's field schema at the time of the join and treats it like any other container on either side. The only container type a Computed Join refuses is another Computed Join.