How Freshness Checks Work¶

Definition¶

Asserts that a table or file was modified recently: the time since its last modification must not exceed the Maximum Age.

Overview¶

The Freshness rule watches when a container last changed. You set a Maximum Age, such as 6 Hours or 1 Day. Each time a Scan evaluates the check, it reads the container's last modification time and measures how long ago that was. When the age exceeds the Maximum Age, the check reports a Shape Anomaly.

The rule reads the container's metadata rather than its values, so it offers neither a filter clause nor a coverage setting.

Typical use cases:

• Alert when a pipeline stops updating a table, even though the table still holds all its old rows.
• Enforce a data delivery SLA, such as "the sales extract is never more than 4 hours old".
• Catch a scheduled export that silently stopped writing new files.

Field Scope¶

None: The rule evaluates the whole container, so there is no field to select on the form. Because nothing is evaluated per row, the check offers neither a filter clause nor a coverage setting.

Anomaly Types¶

When the Check Is Evaluated¶

A Freshness check is evaluated only by a Scan. Freshness and Volumetric checks form the Metadata check category of a Scan, so a Scan includes them when the Metadata category is selected. See Select Check Categories.

When the Scan reaches the check, it reads the container's last modification time at that moment and compares the time elapsed since then with the Maximum Age. It does not load the records into Qualytics, so a Scan that runs only Metadata checks skips the read settings. The source still has to produce the time: Snowflake, BigQuery, and Databricks tables answer from their own metadata, but elsewhere the source computes the newest value of the incremental field across the container, which on a large view or computed container is real work for the database. The Scan also records the time it read as a measurement, next to the daily Observability measurements.

Where the Last Modification Time Comes From¶

What "last modified" means depends on the container and the platform:

A container with no way to provide the time, such as a view without an incremental field or one using the Batch Value strategy, or a partitioned Parquet, ORC, or Avro folder, cannot be measured. The Scan then records the check as Not Asserted, with a message that the container could not provide a freshness reading, and creates no anomaly.

When the time comes from an incremental field, it reflects the newest value in that field, not when the platform last wrote to the table. A load that writes rows with old timestamps does not make the container look fresh.

Maximum Age¶

The Maximum Age is the longest time allowed since the last modification. In the form, you pick a Unit (Millis, Seconds, Minutes, Hours, Days, or Weeks) and enter a value, which can include decimals (1.5 Days). The check stores it as a duration, so the unit only affects how it is entered and shown: the app displays it in words, such as 1 Day and 6 Hours.

The comparison allows a small margin of up to one minute, so a Maximum Age of exactly 6 hours does not fail by a few seconds of clock difference. For a Maximum Age shorter than two minutes, the margin is half the Maximum Age.

One Active Check per Container¶

A container can hold only one Active Freshness check. Creating or activating a second one is rejected with a message that an existing check conflicts, whatever its Maximum Age. AI Managed checks count as well, so when Create AI Freshness checks is on, edit the existing check instead of adding one. Draft and archived checks never conflict.

AI Managed Freshness Checks¶

When Create AI Freshness checks is on in the container's Observability Settings, Qualytics creates one Freshness check per container from its measurement history.

• When it appears. After at least 8 days with a recorded modification time in the last 90 days, and at least three distinct modification times, so that two intervals between updates can be compared.
• How the Maximum Age is set. Qualytics looks at the intervals between successive updates and estimates the longest normal gap three ways (from the average and spread, from the 95th percentile, and from the median with a margin for outliers). The Maximum Age is the middle of the three, which keeps a single unusual gap from stretching it.
• How it is maintained. The Maximum Age is recalculated as new measurements arrive, at most twice a day. A container whose updates have no regular rhythm may never receive an AI Managed check.

Changing the Maximum Age of an AI Managed check converts it to Authored. From then on Qualytics stops updating it. See AI Managed Checks. Qualytics also does not create an AI Managed check on a container that already has a Freshness check of your own, Active or Draft, or whose earlier AI Managed check was archived as Invalid. Checks owned by a locked check template keep the Maximum Age the template sets.

Freshness or Volumetric?¶

Both checks read the container's metadata instead of its records, and both run in the Metadata category of a Scan. They answer different questions:

A table can be modified without gaining rows (an update in place), and a load can run on time but write nothing new. Pair both checks on critical containers. See Volumetric.

No Filter, No Coverage¶

A filter selects rows and coverage tolerates failing rows. Neither concept applies to a rule that reads the container's last modification time. Both inputs are absent from the form, and the API rejects them: a non-empty filter, or a coverage other than 1, returns 422.

See Also¶