Freshness Best Practices¶

Guidelines for getting reliable signal from Freshness checks while keeping the noise (and the maintenance) low.

Schedule the Scan, not just the check¶

A Freshness check raises anomalies only when a Scan evaluates it. The Observability chart keeps updating every day without one, which makes it easy to assume the check is watching. Give every container with a Freshness check a scheduled Scan that includes the Metadata category, and route the anomalies with a Flow. A Metadata-only Scan does not load records into Qualytics, but on a view, a computed container, or a table measured by its incremental field, the source still computes the newest value across the container. Check what that costs on large containers before scheduling the Scan every few minutes.

Match the Scan frequency to how fast you need to know¶

The check can only notice staleness when a Scan runs. With a Maximum Age of 6 hours and a daily Scan, a table can be stale for most of a day before anyone is told. Scan at least as often as the delay you can tolerate: hourly for an SLA counted in hours, once a day, timed before the data is used, for a daily load.

Set the Maximum Age with some slack¶

Take the normal gap between updates and add the lateness you are willing to accept: a 4 hour extract with a 6 hour Maximum Age allows about two hours of lateness. If an update is missed entirely, the check fails about two hours after it was due, before the next scheduled update. A Maximum Age equal to the update interval fires every time a run is a few minutes late.

Give views and computed containers an incremental field¶

Outside Snowflake, BigQuery, and Databricks tables, the last modification time is the newest value of the container's incremental field. Without a Last Modified incremental field, the check is Not Asserted on every Scan. Set one before creating the check, and pick the field your pipeline actually updates on every load. See Where the Last Modification Time Comes From.

Let AI Managed checks set the baseline, author the exceptions¶

Create AI Freshness checks gives every tracked container a check whose Maximum Age follows its real update rhythm. Author the Maximum Age yourself when it comes from a commitment rather than past behavior, such as a delivery SLA. A container takes only one Active Freshness check, so to do that, edit the AI Managed check. The edit converts it to Authored and Qualytics stops adjusting it.

Pair Freshness with Volumetric¶

A load can run on time and write nothing new, and a table can grow without its incremental field moving. Volumetric watches how much the container holds, Freshness watches when it last changed. Running both on critical containers covers both failure modes.