Skip to content

Cost and Performance

A Computed File is metadata; nothing is materialized in your DFS storage. Every operation that needs to look at the container (a profile, a scan, a quality check run, a preview) executes the stored query through Qualytics's analytical engine, which reads the underlying files from the parent DFS datastore. This page walks through where the cost lives, how to spot slow queries early, and when to move logic out of the Computed File entirely.

Where the Cost Lives

Each of these operations runs the full query end-to-end against the source files:

  • Scans: every quality check on the Computed File triggers a full read.
  • Full profile: every field's statistics are recomputed from the current query result.
  • Preview: the Container detail view samples rows from a live query.

Validation is different. Validate is a schema-only check: it parses the SQL and runs it against a zero-row view through the analytical engine to confirm every referenced file container and field resolves, but it does not read production data. Validate is cheap and safe to run frequently while iterating on a query.

Signs a Computed File Is Getting Slow

  • The slim profile that runs on Create or Edit takes longer than a few seconds.
  • Validate hits its timeout window (default 150 seconds) and returns 408 Request Timeout.
  • Scheduled datastore profiles run past their usual window.
  • Individual scans stretch or fail with reader timeouts.

Ways to Cut Cost

  • Narrow with a WHERE clause. Filtering out rows that never contribute to anomalies is the highest-impact change you can make.
  • File format matters. Columnar formats like Parquet and ORC are much cheaper to scan than row-based formats like CSV or JSON, because the engine only reads the columns your query references.
  • Lean on partitioned sources. If the base file container uses Hive-style partition folders (for example, date=2026-07-01/), adding WHERE predicates on partition columns lets the analytical engine prune whole partitions instead of listing every file.
  • Use compression. Snappy and gzip are decoded automatically by the reader; enabling them on your upstream files reduces storage-bandwidth cost with no query-side changes.
  • Split large source patterns. Rather than pointing one Computed File at a single massive glob, use narrower patterns per Computed File so each query only scans the files it needs.

When to Land Files Upstream

If a Computed File is:

  • reused across multiple quality checks or downstream reports,
  • expensive to compute (heavy joins, window functions, or aggregations over billions of rows), and
  • based on data that changes on a predictable schedule,

it is usually cheaper to produce the rolled-up files upstream in your data pipeline, land them back into the same DFS storage, and catalog them as a base file container in Qualytics. The trade-off is that the pre-aggregated files have to be refreshed by whatever pipeline produces them; for logic that changes often or is unique to Qualytics's monitoring flow, a Computed File is usually the right home.