# Examples

Source: https://userguide.qualytics.io/container/computed-files/deep-dive/examples/

Four production patterns for Computed Files across JSON log parsing, IoT time-series rollup, array explosion, and event deduplication. Each scenario shows the clause fragments to paste into the Add Computed File modal. Every fragment uses Spark SQL syntax.

**Tab: JSON Log Parsing**

**Context.** A platform team ingests raw application logs as JSON files into their data lake. Each line carries a nested payload (`level`, `service`, `trace_id`, `latency_ms`, `error.message`). They want Qualytics to monitor error rates and latency percentiles without first running an ETL job to flatten the logs into a warehouse table.

**What happens.** The team creates a Computed File on top of the base JSON file container that already scans the log path, then projects the nested payload into typed top-level columns and filters to the events they care about. Every scan re-reads the underlying files, so newly landed logs are covered automatically. Quality checks land on `latency_ms` and `error_message`; a spike in error volume or p99 latency flags the incident directly on the container.

Fill the Add Computed File modal with:

- **Source File**: the base JSON file container that scans the log path (for example, `s3://logs-raw/app/`).
- **Select Expression**:

  ```text
  event_time,
  payload.level AS level,
  payload.service AS service,
  payload.trace_id AS trace_id,
  payload.latency_ms AS latency_ms,
  payload.error.message AS error_message
  ```
- **Filter Clause**:

  ```text
  payload.level IN ('ERROR', 'WARN')
  ```

**Why it works.** Parsing structured fields inside the Computed File keeps the raw JSON immutable while still giving business quality checks a typed surface to target. If a new deploy renames `payload.error.message` to `payload.err.msg`, the next profile surfaces the loss as a Missing field on the Computed File and the team sees the schema drift before a downstream dashboard does.

**Tab: IoT Daily Rollup**

**Context.** A manufacturing team lands sensor readings as Parquet files partitioned by date (`dt=YYYY-MM-DD/`). They monitor daily min, max, and average readings per device and want an anomaly the moment a day is missing or a device goes silent.

**What happens.** The Computed File aggregates by day and device across the partitioned Parquet source. A partition-column predicate in the Filter Clause lets the Dataplane prune whole partitions instead of listing every file. Freshness tracking and an automated freshness check turn a delayed ingest into an anomaly the same day.

Fill the Add Computed File modal with:

- **Source File**: the base Parquet file container that scans `s3://iot-sensors/dt=*/`.
- **Select Expression**:

  ```text
  date_trunc('day', reading_time) AS reading_date,
  device_id,
  site_code,
  MIN(reading_value) AS min_reading,
  MAX(reading_value) AS max_reading,
  AVG(reading_value) AS avg_reading,
  COUNT(*) AS reading_count
  ```
- **Filter Clause**:

  ```text
  dt >= '2026-01-01' AND reading_value IS NOT NULL
  ```
- **Group By Clause**:

  ```text
  date_trunc('day', reading_time), device_id, site_code
  ```

**Why it works.** A rolling metric that gets stale defeats its own purpose. Because a Computed File is not materialized, every scan re-reads the object store; there is no refresh job to forget. The partition predicate on `dt` lets the Dataplane skip whole directories instead of listing every file, so the aggregation cost scales with the window you monitor rather than the full history.

**Tab: Array Explosion**

**Context.** A product-analytics team lands events as JSON files. Each event carries a `tags` array (`["premium", "beta", "us-west"]`) and they want one row per (user, tag) pair for downstream tag-level quality checks.

**What happens.** A **Lateral View** with `EXPLODE(tags)` unfolds the array into rows. The Computed File exposes a flat `user_id, tag` shape that quality checks can target without dealing with array types.

Fill the Add Computed File modal with:

- **Source File**: the base JSON file container that scans the events path.
- **Select Expression**:

  ```text
  user_id,
  tag
  ```
- **Filter Clause**:

  ```text
  tag IS NOT NULL
  ```
- **Lateral Views** (Advanced Options):

  ```text
  EXPLODE(tags) t AS tag
  ```

**Why it works.** Lateral views are the CF-native pattern for flattening nested arrays. Qualytics prepends `LATERAL VIEW` automatically, so the user only supplies the body. Quality checks like "at most one row per user per tag" or "no null tags" can then target the flat `tag` column directly.

**Tab: Deduplicated Events**

**Context.** A data-engineering team lands click-stream events as Parquet files, but the upstream ingest occasionally replays batches, producing duplicate rows for the same `event_id`. They want to run uniqueness and reconciliation checks against a clean projection without changing the raw file layout.

**What happens.** Because the CF form does not accept a full subquery with a `ROW_NUMBER()` filter, deduplication is expressed with `MAX_BY` on the columns that carry the "latest per key" value, grouped by `event_id`. The result is one row per `event_id`, always the most recent by `event_time`.

Fill the Add Computed File modal with:

- **Source File**: the base Parquet file container that scans the raw events path.
- **Select Expression**:

  ```text
  event_id,
  MAX_BY(user_id, event_time) AS user_id,
  MAX_BY(event_type, event_time) AS event_type,
  MAX(event_time) AS event_time,
  MAX_BY(session_id, event_time) AS session_id
  ```
- **Group By Clause**:

  ```text
  event_id
  ```

**Why it works.** Splitting "raw" and "clean" projections of the same source as two containers keeps monitoring meaningful: the raw file location stays observable for pipeline-health checks (duplicate counts, ingest lag), and the clean Computed File is what business quality checks reference. `MAX_BY` is the CF-native alternative to the `ROW_NUMBER + WHERE rn = 1` idiom, which cannot be expressed in a clause-only form.

*Last updated: August 24, 2026.*