Skip to content

Getting Started with Computed Files

Computed Files are Spark SQL-defined virtual containers built on top of the base file patterns of a DFS source datastore (Amazon S3, Google Cloud Storage, or Azure Data Lake Storage). You write a single Spark SQL query; Qualytics stores the definition and runs it whenever the container is scanned or profiled. The result behaves like any other container: you can profile it, add quality checks, and detect anomalies.

Use Computed Files to prepare data (clean, shape, restructure), run complex calculations, subset data with a WHERE clause, or join file patterns inside the same DFS datastore.

Add Computed File modal with fields for Name, Source File, Select Expression, and additional clauses

Permissions

You need the Member user role and one of the following on the parent datastore to add, edit, or delete a Computed File: the Editor team permission, or the Author team permission when you are (or will be) the owner of the Computed File. Viewing one requires the Reporter team permission (or above) on the datastore. See the Permissions page for the full matrix.

Deep Dive

  • Introduction

    What a Computed File is, when to reach for one, and how it differs from a Computed Table.

    Introduction

  • How It Works

    Execution model, validation semantics, and the query editor for Spark SQL over DFS file patterns.

    How It Works

  • Computed File vs Computed Table

    Side-by-side comparison: when to reach for a Computed File over a Computed Table, and where each one shines.

    Computed File vs Computed Table

  • Referencing

    What a Computed File can reference, workarounds for chained transformations, and composing Computed Files into a Computed Join.

    Referencing

  • Lifecycle

    Profile-on-create, volumetric and freshness tracking, edit re-profiling, history diff, delete cascades, and notifications.

    Lifecycle

  • Cost and Performance

    Where the cost of a Computed File lives and how to keep scans and profiles fast on large data.

    Cost and Performance

  • Permissions

    Who can view, create, edit, delete, reassign, and run operations on a Computed File. Includes the Author-and-owner hybrid gate.

    Permissions

  • Best Practices

    A checklist for naming, query style, metadata, profiling, ownership, and sunsetting Computed Files.

    Best Practices

  • Examples

    Real-world Computed Files across JSON log parsing, IoT time-series rollup, array explosion, and event deduplication.

    Examples

How-tos

  • Add a Computed File

    Step-by-step walkthrough of the Add Computed File modal, from datastore selection to Save.

    Add

  • Edit a Computed File

    Change the Spark SQL clauses, description, owner, or metadata of an existing Computed File.

    Edit

  • Delete a Computed File

    Remove a Computed File and understand its impact on quality checks, anomalies, and downstream operations.

    Delete

Reference

  • API

    REST endpoints to add, edit, delete, and read Computed Files programmatically.

    API

  • FAQ

    Answers to common questions about Computed Files: references, joins, materialization, and file-format quirks.

    FAQ