Computed Files FAQ
Referencing and Joins
Can a Computed File reference another Computed File?
No. Computed Files are stored as metadata inside Qualytics; they are not created as real files on storage. When the query runs, it can only read the base file container you selected as the source. If you need to chain logic, merge the intermediate step into a single Select Expression, or produce the intermediate result as real files on storage (via your own data pipeline) and catalog it in Qualytics as a base file container.
Can a Computed File reference files from a different datastore?
No. A Computed File only sees the base file containers of its own parent DFS source datastore. For cross-datastore composition, use a Computed Join.
Can I use a Computed File as an input to a Computed Join?
Yes. A Computed Join can be built on top of tables, views, Computed Tables, or Computed Files as long as each input has been profiled. The only input a Computed Join cannot accept is another Computed Join.
Materialization and Cost
Are Computed Files materialized on storage?
No. The Spark SQL definition is stored as metadata inside Qualytics and executed against the source files every time the container is scanned, profiled, or previewed. Nothing is written back to storage.
Does saving a Computed File trigger a scan?
No. Saving persists the definition and runs a basic profile that samples up to 1000 records per partition so field statistics appear immediately. A full profile and any scans run only when you or a schedule kick them off.
How can I reduce the cost of a slow Computed File?
The Spark SQL expression runs every time the container is scanned or profiled, so slow queries scale their cost with scan frequency. To speed things up: narrow the input with a WHERE clause, prefer columnar formats (Parquet, ORC) over row-based formats (CSV, JSON) for large sources, and split very heavy preparation into an upstream pipeline that writes cleaned files that Qualytics catalogs as a base file container.
File Format Behavior
Which file formats can back a Computed File?
Any format the parent DFS datastore already supports for base file containers: Parquet, ORC, Avro, CSV, TSV, PSV, JSON, plus Excel (.xls, .xlsx, .xlsm) and table formats such as Delta and Iceberg. Compression codecs (Snappy, gzip, and so on) are decoded automatically. If a source file container already scans and profiles successfully, it is eligible as a Computed File source.
Where does the Computed File SQL execute?
Through Qualytics's own analytical engine. Unlike Computed Tables, which push their query down to the parent warehouse's engine, Computed Files evaluate the stored Spark SQL clauses directly on the files Qualytics reads from the DFS datastore. Write the Select Expression in Spark SQL syntax regardless of whether Amazon S3, Google Cloud Storage, or Azure Data Lake Storage backs the datastore.
Why does the DISTINCT keyword fail validation?
The DISTINCT keyword is disallowed anywhere in the Select Expression, including COUNT(DISTINCT ...), APPROX_COUNT_DISTINCT(...), and DISTINCT(col). Use GROUP BY on the same columns for de-duplication and explode(collect_set(field)) AS distinct_field_values for distinct-value semantics.
Editing and History
Does editing the definition re-run the profile?
Yes. Saving an edit immediately triggers a synchronous slim profile so field statistics refresh right away, followed by a full asynchronous profile. Existing quality checks are preserved, and anomalies raised against the older version remain in the anomaly history.
What happens if my edit drops a field that a quality check depends on?
Qualytics blocks the save and shows a Dropped Fields Warning listing every affected check and anomaly. You can either go back to the modal and adjust the Select Expression so the field is still emitted, or confirm Proceed Anyway to save the new definition. Affected checks are marked inactive and anomalies stay in the history but become detached.
Can I see previous versions of the definition?
Yes. Open the container's History panel to see the timeline of changes. For each edited clause in a revision, a View Select Clause Changes, View Where Clause Changes, or View Group By Clause Changes action appears next to the change entry and opens a From/To dialog with two side-by-side labeled blocks showing the previous and new value, along with the user and timestamp for that change.
Permissions
Who can create a Computed File?
Any user with the Editor or Author team permission on the parent datastore. Authors can only create Computed Files they own themselves; Editors can create Computed Files owned by any user.
Who can reassign ownership?
Only users with the Editor team permission. Authors cannot change the owner of a Computed File, even one they own.