Skip to content

How Freshness Works

Freshness tracking watches when a table or file was last modified. Once tracking is enabled on a container, Qualytics records the container's last-modified timestamp on each sweep, and Freshness checks compare the age of the newest data against a configured Maximum Age. When data is older than the threshold, an anomaly is raised.

Enabling freshness tracking

Freshness tracking is a per-container toggle in the container's Observability Settings, independent from Volume Tracking.

  • Freshness Tracking is the base toggle. It has to be on for any freshness measurement to happen.
  • Create AI Freshness checks is a separate toggle. When on, Qualytics automatically authors freshness checks once enough measurements have accumulated (the check pill carries an AI badge).
  • Turning Freshness Tracking off stops new measurements from being recorded. Existing authored checks are preserved but stop receiving new data.

You can also author a Freshness check manually from the Quality Checks flow. Manually authored checks do not carry the AI badge.

The measurement sweep

Qualytics runs a scheduler-driven sweep every hour at minute 30 (UTC). Each container gets one Freshness measurement per UTC day: the first successful attempt of the day sticks, and later hourly ticks skip containers that already have a measurement for that UTC day. Large datastores can take a few hourly ticks to fully sweep before every container has recorded its measurement; a container that missed an earlier tick is picked up automatically on the next one.

Freshness measurements come from this sweep only. Running a Profile or Scan on a container does not record a new last_modification_time for the Freshness chart, so a check that has been idle since the previous UTC day will not update just because you triggered another operation.

Containers marked Inaccessible are excluded from the sweep, and three consecutive failed measurements will flip a container's status to Inaccessible. Only a subsequent successful measurement restores the container to Available: aborted or timed-out attempts do not clear the status, even if the underlying source has recovered.

Maximum Age

Maximum Age is the threshold that separates "fresh" from "stale". Its value is stored in milliseconds internally, but the UI shows and edits it as a human-readable duration with two segments joined by and:

  • 1 Day and 38 Seconds
  • 2 Hours and 15 Minutes
  • 3 Weeks

When editing, the value expands into an input where the current unit is shown as a prefix. Supported units are:

Millis, Seconds, Minutes, Hours, Days, Weeks.

The unit only affects display and editing; the underlying value is always converted to milliseconds for storage. A stored value of 0 is treated as unset: the read-only view shows nothing next to the pencil (no 0 Milliseconds label). Open the editor and enter a positive value to bring the check back into range.

What the chart shows

  • Bars. Each bar is one measured period. The height represents the age of the data at the time of the measurement.
  • Threshold line. A solid yellow line marks the configured Maximum Age. When Maximum Age is unset (stored value 0), the line is hidden.
  • Salmon-tinted bars. Measurements that exceeded the threshold render in salmon, making stale-data periods easy to spot.
  • In-progress indicator. The bucket for the current report date shows a pulsing orange marker while its measurement is still being recorded. During this window, hovering that bucket displays the note Measurements are still being recorded for the selected report date and both timestamp rows in the tooltip show —. The indicator disappears once the measurement lands and the bar switches to its final color.

Tooltip

Hovering a bar opens a two-part tooltip.

Thresholds and staleness (top section)

  • Max age. The currently configured threshold in compact form (for example, 2h 30m).
  • Staleness. The age of the data at the time of the measurement (the same value the bar height represents). Rendered in red when it exceeds Max age.

When Maximum Age is unset (stored value 0), this entire top section is omitted and the tooltip shows only the Timestamps section below.

Timestamps (bottom section)

  • Observed. When the measurement was taken.
  • Last modified. The container's last-modification timestamp for that period.

Both timestamps use your browser's local timezone and include a DST-aware abbreviation in parentheses. Day and hour buckets show a shorter format like 2:45 PM (EDT); week and month buckets include the date, like Jul 15, 2:45 PM (EDT). An empty timestamp renders as —.

If the period accumulated anomalies, an orange badge with an alert icon and the anomaly count appears at the top right of the tooltip.

Filters, sort, and search on the listing

The Measures tab supports these controls:

  • Filter: by Tags, by Rule Type (Volumetric or Freshness), and, on the Explore scope, by Datastore and Container as well.
  • Sort by: Name, Type, Created Date, Active Anomalies, Active Checks, Last Scanned.
  • Search: matches container name or identifier.

Marking a card as a favorite pins it to the top of the listing; favorites are per-user.

Inferred vs authored checks

Every Freshness check has an inferred flag. Inferred checks are AI-authored from accumulated measurements and display an AI badge next to the check id. Authored checks (created manually or via the API) show no badge. Both flavors use the same Maximum Age semantics and appear on the same card as Volumetric.