Add a Computed File
Add a Computed File to a DFS source datastore. The Computed File becomes a container inside that datastore, alongside the base file patterns, and can be profiled and monitored like any other container.
Related
- Concept: Introduction for what a Computed File is.
- Mechanics: How It Works for validation, referencing rules, and execution.
- Comparison: Computed File vs Computed Table if you are not sure which one you need.
Field Reference
Every field on the Add Computed File modal, in three sections that mirror the layout of the form.
| REF. | Field | Description |
|---|---|---|
| 1 | Name (Required) | A descriptive, meaningful name for the Computed File (for example, customer_order_statistics). Spaces are replaced with underscores automatically. Must be unique inside the datastore. |
| 2 | Source File (Required) | The base file pattern this Computed File is built on. The dropdown lists file patterns in this datastore that have been profiled at least once; unprofiled files and other computed containers are filtered out. |
| 3 | Select Expression (Required) | The Spark SQL projection defining the shape of the Computed File. Field references must exist in the source's profiled schema. The DISTINCT keyword is not allowed in any form (including COUNT(DISTINCT ...), APPROX_COUNT_DISTINCT(...), and DISTINCT(col)); use GROUP BY on the same columns for de-duplication, or explode(collect_set(field)) AS distinct_field_values for distinct-value semantics. Press Ctrl+Space inside the editor for hints. See How It Works for the full referencing rules. |
| REF. | Field | Description |
|---|---|---|
| 4 | Filter Clause (Optional) | A Spark SQL WHERE predicate to narrow the rows. Standard Spark SQL boolean expressions are supported. |
| 5 | Group By Clause (Optional) | A Spark SQL GROUP BY clause. Required when the Select Expression uses aggregation functions (COUNT, SUM, AVG, MIN, MAX). Every non-aggregated column in the Select Expression must appear here. |
| 6 | Suggest | Generates an AI description for the Computed File based on the current form values. Only visible when an LLM provider is configured under AgentQ. See Description Suggestions for the full workflow. |
| 7 | Description (Optional) | A short business summary of what the Computed File represents. |
| REF. | Field | Description |
|---|---|---|
| 8 | Owner (Required) | The Qualytics user who owns this Computed File. Defaults to you. Only users with the Editor team permission can assign a different owner; users with the Author team permission can create Computed Files they own but cannot reassign later. |
| 9 | Additional Metadata (Optional) | Custom key-value pairs. Click the plus icon to open the editor and add rows. |
| REF. | Field | Description |
|---|---|---|
| 10 | Lateral View input (Optional, Advanced) | Located under Advanced Options. Each entry is a Spark SQL expression such as EXPLODE(tags) t AS tag; Qualytics prepends LATERAL VIEW automatically. Use this when the source has array-valued columns you want to expand into rows. |
| 11 | Add Lateral View button | Adds another lateral view entry to the list so you can chain multiple explodes or posexplode operations in the same Computed File. |
Profile the source first
The base file pattern you want to build on must already have been profiled at least once. Qualytics uses the profiled schema to validate every field reference in the Select Expression. If the source is not yet profiled, open it in the tree view, run Scan on it once, then come back to the Add Computed File modal.
Steps
Step 1: Click the Add button in the top-right of the datastore overview.
Step 2: A dropdown appears with the containers you can add to a DFS datastore.
Step 3: Select Computed File. This option appears in the dropdown only when the parent datastore is a DFS type (S3, GCS, or Azure Data Lake Storage).
Step 4: The Add Computed File modal opens. See the Field Reference above for what each input controls.
Step 5: Fill in the fields. Name, Source File, and Select Expression are required; Filter Clause, Group By Clause, Description, Owner, Additional Metadata, and Lateral Views are set as needed.
Step 6: Click Validate. Qualytics parses every clause as Spark SQL and confirms that every referenced field exists in the source's profiled schema and that the source is still reachable on storage.
If validation fails, the modal surfaces an inline error that quotes the failed clause so you can fix it in place. Running Validate is optional; if you skip it, Qualytics validates the definition as part of the Save request and surfaces any error inline. Clicking Validate returns 408 Request Timeout after 150 seconds if the analytical engine cannot respond; Save uses a longer 500-second window for the same inline validation. Simplifying the filter or narrowing the source pattern usually resolves timeouts on either path.
Step 7: Once the validation success message appears, click Save.
Step 8: A success message confirms the creation and the new Computed File appears in the datastore's containers list. Qualytics automatically runs a slim profile of up to 1000 records per partition so you can see field statistics right away, followed by a full profile in the background.
