Skip to content

How Computed Files Work

A Computed File is a piece of metadata inside Qualytics. Its Spark SQL definition lives with the datastore configuration; nothing is written back to DFS storage. This page walks through how the platform resolves that metadata into actual data every time it needs to look at the container.

For what a Computed File can reference and how it composes with a Computed Join, see Referencing. For what happens on create/edit/delete and how tracking works, see Lifecycle. For who can do what, see Permissions.

The Execution Model

A Computed File is never materialized as a physical file on storage. When Qualytics needs to look at the container (during a scan, a profile, a quality check run, or a preview), it takes the stored Spark SQL clauses, expands the source file pattern into the matching files, and evaluates the expression on top of them. The analytical engine returns the rows, and Qualytics treats those rows as the container's live data.

This has three practical consequences you should keep in mind while designing a Computed File:

  • Every scan is fresh. There is no cached copy of the previous run. If the underlying files changed on storage, the next scan sees the new data.
  • Query cost is real. Complex expressions run every time the container is scanned, profiled, or previewed. Aggregations over very large file patterns can slow the run down; consider narrowing the where_clause, using partition-aware source patterns, or moving heavy preparation into a separate base file pattern you catalog independently.
  • Validation is not execution. The Validate button (see below) confirms the clauses are syntactically and semantically valid against the source file schema. It does not run the expression end-to-end over the entire file pattern.

Validation

Every Add and Edit modal exposes a Validate button. Clicking it sends the current form values to the platform for parse and semantic checks:

  • The select_clause, where_clause, group_by_clause, and any lateral_views must all parse as valid Spark SQL.
  • Every lateral view must begin with a supported generator function (see Lateral Views below).
  • Every field referenced by any clause must exist in the source file's profiled schema.
  • The source file container identified by source_container_id must be reachable on storage.
  • Aggregation and grouping must be internally consistent (every non-aggregated field in the Select Expression must appear in Group By, and so on).

If any of those checks fail, the modal surfaces an inline error that quotes the failed clause and the reason so you can fix it in place. Running Validate is optional: you can click Save directly, in which case Qualytics validates the clauses as part of the save request and returns any error inline.

When you click the Validate button, the request runs synchronously with a default timeout of 150 seconds (configurable per request between 30 and 300 seconds). If the analytical engine does not respond in time, the request returns 408 Request Timeout. If you skip Validate and click Save, the same validation runs inline as part of the Save request under a longer platform-level synchronous timeout (500 seconds by default), and any error still surfaces inline. Very complex expressions against very large file patterns can push either path toward its limit.

Lateral Views

The Advanced Options section of the Add and Edit forms exposes a multi-row Lateral Views input. Each row defines a single LATERAL VIEW clause, and you supply only the body: Qualytics prepends the LATERAL VIEW keyword for you. Rows are applied in order, so a later row can reference an alias produced by an earlier one.

Supported Generators

A lateral view must begin with one of these generator functions:

Generator What it does
EXPLODE Turns each element of an array, or each key-value pair of a map, into its own row.
POSEXPLODE Like EXPLODE, and also returns the position of each element.
INLINE Expands an array of structs, giving each struct field its own column.
STACK Reshapes a set of values into a fixed number of rows.
JSON_TUPLE Pulls several keys out of a JSON string in one pass.
PARSE_URL_TUPLE Pulls several parts (host, path, query, and so on) out of a URL in one pass.

Every generator also has an outer form, which keeps the source row even when there is nothing to expand (an empty or null array) instead of dropping it. Write it either way:

  • With the keyword in front: OUTER EXPLODE(tags) t AS tag
  • As a single word: EXPLODE_OUTER(tags) t AS tag, POSEXPLODE_OUTER(...), INLINE_OUTER(...)

What Gets Rejected

Validation runs when you click Validate, and again on Save if you skip it. Two mistakes account for most failures:

  • The clause does not start with a supported generator. The error names the generators that are accepted and quotes the start of the offending clause. A common cause is putting the LATERAL VIEW keyword in the row yourself, since Qualytics already adds it.
  • The Select Expression references an alias the lateral view never declared. EXPLODE(tags) t AS tag declares tag, so selecting tags_value fails as an unresolved column. Check that the alias after AS matches what the Select Expression uses.

A lateral view is a generator expression and nothing else, so the clause is also refused if it contains a semicolon, a set operator (UNION, INTERSECT, EXCEPT), a subquery, or a data-modifying keyword.

Query Editor

The Query editor in the Add and Edit modals is a code-aware surface with a few features that speed up writing Spark SQL:

  • Syntax highlighting. SQL keywords, identifiers, string literals, and comments are colored.
  • Autocomplete. Press Ctrl+Space inside the editor to see a hint list based on the source file's profiled schema (fields Qualytics has already discovered).
  • Inline hint label. The editor shows a subtle "Any valid SparkSQL" caption in the expanded editor identifying which dialect applies, and a "Press Ctrl+Space for hints" hint under the inline input.
  • Fullscreen editor. Click the expand icon in the editor's toolbar to open a larger dialog when the expression grows beyond a few lines.