Skip to content

How Computed Files Work

A Computed File is a piece of metadata inside Qualytics. Its Spark SQL definition lives with the datastore configuration; nothing is written back to DFS storage. This page walks through how the platform resolves that metadata into actual data every time it needs to look at the container.

For what a Computed File can reference and how it composes with a Computed Join, see Referencing. For what happens on create/edit/delete and how tracking works, see Lifecycle. For who can do what, see Permissions.

The Execution Model

A Computed File is never materialized as a physical file on storage. When Qualytics needs to look at the container (during a scan, a profile, a quality check run, or a preview), it takes the stored Spark SQL clauses, expands the source file pattern into the matching files, and evaluates the expression on top of them. The analytical engine returns the rows, and Qualytics treats those rows as the container's live data.

This has three practical consequences you should keep in mind while designing a Computed File:

  • Every scan is fresh. There is no cached copy of the previous run. If the underlying files changed on storage, the next scan sees the new data.
  • Query cost is real. Complex expressions run every time the container is scanned, profiled, or previewed. Aggregations over very large file patterns can slow the run down; consider narrowing the where_clause, using partition-aware source patterns, or moving heavy preparation into a separate base file pattern you catalog independently.
  • Validation is not execution. The Validate button (see below) confirms the clauses are syntactically and semantically valid against the source file schema. It does not run the expression end-to-end over the entire file pattern.

Validation

Every Add and Edit modal exposes a Validate button. Clicking it sends the current form values to the platform for parse and semantic checks:

  • The select_clause, where_clause, group_by_clause, and any lateral_views must all parse as valid Spark SQL.
  • Every field referenced by any clause must exist in the source file's profiled schema.
  • The source file container identified by source_container_id must be reachable on storage.
  • Aggregation and grouping must be internally consistent (every non-aggregated field in the Select Expression must appear in Group By, and so on).

If any of those checks fail, the modal surfaces an inline error that quotes the failed clause and the reason so you can fix it in place. Running Validate is optional: you can click Save directly, in which case Qualytics validates the clauses as part of the save request and returns any error inline.

When you click the Validate button, the request runs synchronously with a default timeout of 150 seconds (configurable per request between 30 and 300 seconds). If the analytical engine does not respond in time, the request returns 408 Request Timeout. If you skip Validate and click Save, the same validation runs inline as part of the Save request under a longer platform-level synchronous timeout (500 seconds by default), and any error still surfaces inline. Very complex expressions against very large file patterns can push either path toward its limit.

Query Editor

The Query editor in the Add and Edit modals is a code-aware surface with a few features that speed up writing Spark SQL:

  • Syntax highlighting. SQL keywords, identifiers, string literals, and comments are colored.
  • Autocomplete. Press Ctrl+Space inside the editor to see a hint list based on the source file's profiled schema (fields Qualytics has already discovered).
  • Inline hint label. The editor shows a subtle "Any valid SparkSQL" caption in the expanded editor identifying which dialect applies, and a "Press Ctrl+Space for hints" hint under the inline input.
  • Fullscreen editor. Click the expand icon in the editor's toolbar to open a larger dialog when the expression grows beyond a few lines.