Skip to content

Best Practices

Recommendations for authoring Computed Files that stay fast, understandable, and easy to govern over time. Nothing here is enforced by the platform; treat it as a checklist you can walk before saving a new query.

Naming

  • Use snake_case with a purpose-oriented suffix that hints at the shape: _agg, _summary, _flat, _score, _metrics, _daily, _monthly.
  • Prefer the noun the container represents to the business (customer_lifetime_value, daily_revenue_summary) over the SQL operation (customers_group_by).
  • Keep the name unique inside the datastore; Qualytics rejects duplicates with 409 Conflict.

Query Style

  • Filter early. Push WHERE clauses toward the base files. A WHERE status = 'active' before a GROUP BY cuts scan cost linearly.
  • Avoid SELECT *. Explicit column lists make missing-field lifecycle detection meaningful and keep the schema stable when base files add columns.
  • Fold multi-step logic into a single set of clauses. A Computed File cannot reference another Computed File, and the form does not accept a full SQL statement with CTEs. Push projections, conditional logic, and window functions directly into select_clause, where_clause, and group_by_clause; land larger intermediates as base file patterns upstream and catalog them.
  • Prefer columnar file formats (Parquet, ORC) for large sources. Column pruning and predicate pushdown make scans much faster than row-based formats like CSV or JSON.
  • Use partition predicates in WHERE when the source uses Hive-style partition folders (year=/month=/day=). The analytical engine can prune partitions and skip entire directories.
  • Land pre-aggregated files upstream. If the same aggregation is used across many Computed Files, produce it once in your upstream pipeline and catalog the resulting files as a base file container.

Metadata and Discoverability

  • Description: write one sentence that answers "what does this container represent to the business?" Not the SQL, the semantics. A downstream reader searching for revenue metrics should recognize the container from the description alone.
  • Tags: use them for cross-cutting facets like domain:revenue-cycle, sla:gold, owner:analytics-team. Tags surface in the container list filters.
  • Additional Metadata: freeform key/value pairs are the right home for things that do not fit a tag but need to travel with the container. Common patterns: refresh_cadence, sla_tier, data_contract_url.

Profiling and Tracking

  • Turn on volumetric and freshness tracking on containers that back business SLAs. The automation toggles (automate_volumetric_checks, automate_freshness_checks) create the anomaly detectors for you.
  • Validate before saving on large queries. Validate is schema-only and safe; catching a missing column or bad dialect during authoring is cheaper than debugging a failed scan.

Ownership and Governance

  • Assign ownership to a domain lead rather than the person who happened to write the query. The owner receives an in-app notification when someone else edits the container or reassigns ownership, and typically drives the review of related quality checks.
  • Reassign after handoffs. When someone leaves a team, use the API or Edit modal to move ownership. Reassignment requires the Editor team permission on the parent datastore, even when the caller is the current owner.
  • Split by team boundary. If two teams need overlapping data with different rules, prefer two Computed Files with distinct ownership over one shared container with mixed quality checks.

Deletion and Sunsetting

  • Copy the query out before deleting. There is no undo. Paste the SQL into your team's DFS storage documentation or into an issue-tracker comment first.
  • Repoint cross-container quality checks first. If any existsIn, notExistsIn, dataDiff, aggregationComparison, or isReplicaOf (sunsetting; use dataDiff for new checks) check in another container references this Computed File, the delete returns 409 Conflict. Repoint or delete those checks before retrying.
  • Sunset before deleting. For containers that other teams may still consume, add a tag like status:deprecated, update the Description with the removal date, and delete only after the deprecation window closes.