Examples

Four production patterns for Computed Files across JSON log parsing, IoT time-series rollup, array explosion, and event deduplication. Each scenario shows the clause fragments to paste into the Add Computed File modal. Every fragment uses Spark SQL syntax.

JSON Log ParsingIoT Daily RollupArray ExplosionDeduplicated Events

Context. A platform team ingests raw application logs as JSON files into their data lake. Each line carries a nested payload (level, service, trace_id, latency_ms, error.message). They want Qualytics to monitor error rates and latency percentiles without first running an ETL job to flatten the logs into a warehouse table.

What happens. The team creates a Computed File on top of the base JSON file container that already scans the log path, then projects the nested payload into typed top-level columns and filters to the events they care about. Every scan re-reads the underlying files, so newly landed logs are covered automatically. Quality checks land on latency_ms and error_message; a spike in error volume or p99 latency flags the incident directly on the container.

Fill the Add Computed File modal with:

Source File: the base JSON file container that scans the log path (for example, s3://logs-raw/app/).

Select Expression:

event_time,
payload.level AS level,
payload.service AS service,
payload.trace_id AS trace_id,
payload.latency_ms AS latency_ms,
payload.error.message AS error_message

Filter Clause:
```
payload.level IN ('ERROR', 'WARN')
```

Why it works. Parsing structured fields inside the Computed File keeps the raw JSON immutable while still giving business quality checks a typed surface to target. If a new deploy renames payload.error.message to payload.err.msg, the next profile surfaces the loss as a Missing field on the Computed File and the team sees the schema drift before a downstream dashboard does.

Context. A manufacturing team lands sensor readings as Parquet files partitioned by date (dt=YYYY-MM-DD/). They monitor daily min, max, and average readings per device and want an anomaly the moment a day is missing or a device goes silent.

What happens. The Computed File aggregates by day and device across the partitioned Parquet source. A partition-column predicate in the Filter Clause lets the analytical engine prune whole partitions instead of listing every file. Freshness tracking and an automated freshness check turn a delayed ingest into an anomaly the same day.

Fill the Add Computed File modal with:

Source File: the base Parquet file container that scans s3://iot-sensors/dt=*/.

Select Expression:

date_trunc('day', reading_time) AS reading_date,
device_id,
site_code,
MIN(reading_value) AS min_reading,
MAX(reading_value) AS max_reading,
AVG(reading_value) AS avg_reading,
COUNT(*) AS reading_count

Filter Clause:

dt >= '2026-01-01' AND reading_value IS NOT NULL

Group By Clause:

date_trunc('day', reading_time), device_id, site_code

Why it works. A rolling metric that gets stale defeats its own purpose. Because a Computed File is not materialized, every scan re-reads the object store; there is no refresh job to forget. The partition predicate on dt lets the analytical engine skip whole directories instead of listing every file, so the aggregation cost scales with the window you monitor rather than the full history.

Context. A product-analytics team lands events as JSON files. Each event carries a tags array (["premium", "beta", "us-west"]) and they want one row per (user, tag) pair for downstream tag-level quality checks.

What happens. A Lateral View with EXPLODE(tags) unfolds the array into rows. The Computed File exposes a flat user_id, tag shape that quality checks can target without dealing with array types.

Fill the Add Computed File modal with:

Source File: the base JSON file container that scans the events path.
Select Expression:
```
user_id,
tag
```
Filter Clause:
```
tag IS NOT NULL
```
Lateral Views (Advanced Options):
```
EXPLODE(tags) t AS tag
```

Why it works. Lateral views are the CF-native pattern for flattening nested arrays. Qualytics prepends LATERAL VIEW automatically, so the user only supplies the body. Quality checks like "at most one row per user per tag" or "no null tags" can then target the flat tag column directly.

Context. A data-engineering team lands click-stream events as Parquet files, but the upstream ingest occasionally replays batches, producing duplicate rows for the same event_id. They want to run uniqueness and reconciliation checks against a clean projection without changing the raw file layout.

What happens. Because the CF form does not accept a full subquery with a ROW_NUMBER() filter, deduplication is expressed with MAX_BY on the columns that carry the "latest per key" value, grouped by event_id. The result is one row per event_id, always the most recent by event_time.

Fill the Add Computed File modal with:

Source File: the base Parquet file container that scans the raw events path.

Select Expression:

event_id,
MAX_BY(user_id, event_time) AS user_id,
MAX_BY(event_type, event_time) AS event_type,
MAX(event_time) AS event_time,
MAX_BY(session_id, event_time) AS session_id

Group By Clause:
```
event_id
```

Why it works. Splitting "raw" and "clean" projections of the same source as two containers keeps monitoring meaningful: the raw file location stays observable for pipeline-health checks (duplicate counts, ingest lag), and the clean Computed File is what business quality checks reference. MAX_BY is the CF-native alternative to the ROW_NUMBER + WHERE rn = 1 idiom, which cannot be expressed in a clause-only form.